A multi-modal identity cross-verification method and system

Abstract

The present invention relates to the field of deep learning technology, and discloses a multimodal identity cross-verification method and system, wherein a multimodal identity cross-verification method includes the following steps: obtaining vehicle owner identity association information and vehicle owner identity authentication history record data; constructing first structured data based on the vehicle owner identity association information, and constructing second structured data including a structural dimension based on the vehicle owner identity authentication history record data; inputting the first structured data and the second structured data into an identity authentication model; and outputting the result of the vehicle owner identity cross-verification and the probability distribution of identity authentication anomalies. The present invention can comprehensively utilize multimodal data associated with individual identities, can improve the reliability and security of identity authentication, and expand its scope of application.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and more specifically, to a multimodal identity cross-validation method and system. Background Art

[0002] With the rapid development of mobile internet and artificial intelligence technologies, identity authentication technologies based on multimodal data have become an important means of addressing identity recognition and verification issues. Traditional identity authentication methods rely primarily on single verification factors such as passwords and ID documents, which are inherently insecure and vulnerable to attack. Multimodal data-based identity cross-verification methods, on the other hand, can comprehensively utilize multiple aspects of information, such as the vehicle owner's biometrics, behavioral habits, and vehicle environment, to confirm their identity, offering greater security and reliability. However, existing multimodal authentication technologies still have some shortcomings. For example, most methods simply concatenate data from various modalities without considering the inherent connections and temporal characteristics between these modalities. Summary of the Invention

[0003] The present invention provides a multimodal identity cross-verification method and system. The method first obtains the identity-related information and historical verification records of the car owner, and constructs them into two types of structured data: one is two-dimensional structured data containing objects such as the car owner, biometrics, behavioral characteristics, vehicle environment, and time series units; the other is historical identity verification record data with a pure time series unit structure. Then, these two types of structured data are input into a special identity verification model for processing. The model adopts a multi-layer neural network structure, including multiple data structure recognition layers, fusion layers, and fully connected layers. It can extract the features of each type of data separately, fuse them, and finally give the identity verification results and anomaly probabilities. The present invention solves the technical problems of insufficient security of traditional single-modal identity verification and the lack of spatiotemporal modeling and anomaly detection capabilities of existing multimodal identity verification methods.

[0004] The present invention provides a multi-modal identity cross-verification method, comprising the following steps:

[0005] Step 100: Obtain the vehicle owner's identity association information and the vehicle owner's identity verification history record data;

[0006] Step 200: constructing first structured data based on the vehicle owner identity association information; the first structured data is represented as ordered information in two structural dimensions, where the basic structural elements included in the first structural dimension are objects; and the basic structural elements included in the second structural dimension are time series units.

[0007] Step 300 , constructing second structured data including a structural dimension based on the vehicle owner identity verification history data, wherein the basic structural elements included in this structural dimension are time series units;

[0008] Step 400: Input the first structured data and the second structured data into the identity authentication model. The identity authentication model includes a backbone network, which includes a first data structure recognition layer, a second data structure recognition layer, a third data structure recognition layer, a first data structure recognition fusion layer, a first fully connected layer, and a second fully connected layer. The first structured data is input into the first data structure recognition layer, which outputs the first hidden feature to the second data structure recognition layer; the second data structure recognition layer outputs the second hidden feature to the first data structure recognition fusion layer; the first data structure recognition fusion layer outputs the first fused hidden feature to the first fully connected layer; the first fully connected layer outputs the result of cross-verification of the vehicle owner's identity; the second structured data is input into the third data structure recognition layer, which outputs the third hidden feature to the first data structure recognition fusion layer and the second fully connected layer; the second fully connected layer outputs the probability distribution of identity authentication anomalies.

[0009] The calculation formula of the first data structure identification layer is as follows:

[0010]

[0011] represents the object hidden features of the v-th object in the L-th layer in the first structured data, represents the object hidden features of the u-th object in the L-1-th layer, and Respectively represent the set of objects that have object relationships with object v and object u, represents the cardinality of the set, represents the plane recognition weight matrix of the Lth layer of the first data structure recognition layer, , E represents the total number of layers, when L=1 , Represents the features associated with the u-th object, when L=E is equal to the first hidden feature of object v, is the sigmoid activation function;

[0012] The calculation formula of the second data structure identification layer is as follows:

[0013]

[0014] in, and Represents the activation vector of the reset gate and update gate at step t; represents the candidate state generated in step t; represents the second hidden feature of the t-th step, represents the second hidden feature of step t-1; 、 、 、 、 、 Represent the transformation matrices of the 1st, 2nd, 3rd, 4th, 5th, and 6th trainable parameters respectively; 、 、 Indicates the 1st, 2nd, and 3rd trainable bias parameters; , It represents the first hidden feature of the object v output when the t-th time sequence unit inputs the first data structure recognition layer, , Indicates the number of time series units of the first structured data, hour, ,at this time represents the tth time sequence unit in the first structured data, Represents the collection of all objects; Represents the Sigmod activation function; represents the tanh activation function;

[0015] The calculation formula of the third data structure identification layer is as follows:

[0016]

[0017] in Indicates the A third hidden feature, Indicates the A third hidden feature, 1, Indicates the number of time series units of the second structured data, Indicates the first Timing units, and is the transformation matrix of the 7th and 8th trainable parameters, is the fourth trainable bias parameter, ReLU is the ReLU activation function;

[0018] The calculation formula of the first data structure identification fusion layer is as follows:

[0019]

[0020] in, represents the combined features, represents the feature combination function (splicing function or summation function), Indicates the A third hidden feature, Indicates the A second hidden feature.

[0021] In a preferred embodiment, the owner's identity association information includes the owner's basic information, the owner's biometric information, the owner's behavioral information, and the vehicle environment information; the owner's identity authentication history record data includes: the identity authentication history record result, historical biometric similarity, historical behavioral similarity, and the identity authentication history record description text composed of historical vehicle environment information.

[0022] In a preferred embodiment, the method for constructing the first structured data includes:

[0023] Objects for extracting orderliness information;

[0024] Extract the features associated with the object and divide the extracted features into multiple parts according to the collection period. Each part is mapped to a time sequence unit. The order of the time sequence units is consistent with the time sequence of the feature collection period.

[0025] Establish a mapping between the extracted objects and their associated features;

[0026] For any two objects, if there is a correlation between the associated features related to the authentication task or there is a correlation between the objects related to the authentication task, an object relationship is established between the two objects.

[0027] In a preferred embodiment, the objects in the first structured data include: a vehicle owner, a vehicle owner's biological representation, a vehicle owner's behavioral representation, and a vehicle environment.

[0028] In a preferred embodiment, the characteristics associated with the vehicle owner include: vehicle owner's name, contact information, and emergency contact;

[0029] The features associated with the vehicle owner’s biometric representation include: facial image, fingerprint image, iris image, and voice sample;

[0030] Features associated with vehicle owner behavior representation include: key usage data, key usage sequence, vehicle system operation log, and seat adjustment data;

[0031] The characteristics of vehicle-environment association include: vehicle status, external environment;

[0032] Vehicle status includes: vehicle speed, steering wheel angle, accelerator and brake pedal data, and vehicle position; external environment includes: external temperature, humidity, lighting, weather conditions, visibility, and ground conditions; weather conditions include: clear, cloudy, overcast, rain and snow, fog and haze, thunderstorms, and typhoons; ground conditions include: dry, wet, waterlogged, iced, snowed, muddy, sandy, oily, and under construction.

[0033] In a preferred embodiment, the associations related to the identity verification task include:

[0034] The correlation between car owners means that: the two car owners drive the same vehicle model;

[0035] The correlation between the vehicle owner's biological representation and the vehicle owner means that: the vehicle owner has his or her own biological representation;

[0036] The correlation between the owner's behavioral representation and the vehicle owner's behavioral representation means that: the two behavioral representations are similar;

[0037] The correlation between the owner’s behavioral representation and the owner means that: the owner has his or her own behavioral characteristics;

[0038] The correlation between the vehicle environment and the driver's behavioral characteristics means that the vehicle environment will affect the driver's behavioral characteristics.

[0039] In a preferred embodiment, the method for constructing the second structured data includes:

[0040] Extract the vehicle owner identity authentication history record data, divide the extracted features of the vehicle owner identity authentication history record data into multiple parts according to the data collection period, map one part to a time series unit, and the order of the time series units is consistent with the time order of the collection period of the vehicle owner identity authentication history record data.

[0041] In a preferred embodiment, the first data structure recognition fusion layer is combined with the first fully connected layer for pre-training, and the identity authentication model is trained to have the ability to accurately determine the identity of the vehicle owner.

[0042] In a preferred embodiment, the third data structure recognition layer is pre-trained in combination with the second fully connected layer, and the trained identity authentication model has the ability to accurately determine whether the owner's identity is abnormal and the abnormal category.

[0043] In a preferred embodiment, a multimodal identity cross-verification system includes the following modules:

[0044] A data collection module is used to obtain the vehicle owner's identity association information and the vehicle owner's identity verification history data;

[0045] a structured data construction module, configured to construct the vehicle owner identity association information into first structured data, and the vehicle owner identity verification history record data into second structured data including a structure dimension;

[0046] The structured data processing module is used to input the first structured data and the second structured data into the identity authentication model, and output the result of the cross-verification of the vehicle owner's identity and the probability distribution of the identity authentication anomaly.

[0047] The beneficial effects of the present invention are:

[0048] Improving identity verification accuracy: This invention introduces multimodal information related to the vehicle owner's identity and considers the spatiotemporal structure characteristics of the data to construct a multi-layer neural network model, which can fully explore the correlation and temporal evolution laws between different modal data, thereby greatly improving the accuracy of identity verification.

[0049] Enhanced identity authentication security: This invention introduces an anomaly detection mechanism into the identity authentication model. By modeling and analyzing historical authentication records, it can promptly detect potential identity theft, illegal access and other anomalies, and take corresponding preventive measures, thereby enhancing the security of identity authentication and effectively protecting the legitimate rights and interests of car owners. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flowchart of a multimodal identity cross-validation method of the present invention. DETAILED DESCRIPTION

[0051] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that these embodiments are discussed solely to enable those skilled in the art to better understand and implement the subject matter described herein, and that the functions and arrangements of the elements discussed may be varied without departing from the scope of this specification. Various examples may omit, substitute, or add various processes or components as needed. Furthermore, features described in some examples may be combined in other examples.

[0052] At least one embodiment of the present invention discloses a multimodal identity cross-verification method, such as Figure 1 As shown, the following steps are included:

[0053] Step 100: Obtain vehicle owner identity association information and vehicle owner authentication history data. The vehicle owner identity association information includes basic vehicle owner information, vehicle owner biometric information, vehicle owner behavioral information, and vehicle environment information. The vehicle owner authentication history data includes authentication history record results, historical biometric similarity, historical behavioral similarity, and historical vehicle environment information.

[0054] In one embodiment of the present invention, the vehicle owner identity association information and the vehicle owner identity verification history record data are collected once every fixed time, and the fixed time is 60 minutes.

[0055] Step 200: Construct first structured data based on the vehicle owner identity association information. The first structured data is represented as ordered information in two structural dimensions. The basic structural elements included in the first structural dimension are objects, including: vehicle owner, vehicle owner biometric representation, vehicle owner behavioral representation, and vehicle environment. The basic structural elements included in the second structural dimension are time series units.

[0056] In some embodiments of the present invention, the method of constructing the first structured data includes:

[0057] Objects for extracting orderliness information;

[0058] Extract the features associated with the object and divide the extracted features into multiple parts according to the collection period. Each part is mapped to a time sequence unit. The order of the time sequence units is consistent with the time sequence of the feature collection period.

[0059] Establish a mapping between the extracted objects and their associated features;

[0060] For any two objects, if there is a correlation between the associated features related to the authentication task or there is a correlation between the objects related to the authentication task, an object relationship is established between the two objects.

[0061] In some embodiments of the present invention, the characteristics associated with the vehicle owner include: the vehicle owner's name, contact information, and emergency contact;

[0062] The features associated with the vehicle owner’s biometric representation include: facial image, fingerprint image, iris image, and voice sample;

[0063] Features associated with vehicle owner behavior representation include: key usage data, key usage sequence, vehicle system operation log, and seat adjustment data;

[0064] The characteristics of vehicle-environment association include: vehicle status, external environment;

[0065] Vehicle status includes: vehicle speed, steering wheel angle, accelerator and brake pedal data, and vehicle position; external environment includes: external temperature, humidity, lighting, weather conditions, visibility, and ground conditions; weather conditions include: clear, cloudy, overcast, rain and snow, fog and haze, thunderstorms, and typhoons; ground conditions include: dry, wet, waterlogged, iced, snowed, muddy, sandy, oily, and under construction.

[0066] In some embodiments of the present invention, the associations related to the identity authentication task include:

[0067] The correlation between car owners means that: the two car owners drive the same vehicle model;

[0068] The correlation between the vehicle owner's biological representation and the vehicle owner means that: the vehicle owner has his or her own biological representation;

[0069] The correlation between the owner's behavioral representation and the vehicle owner's behavioral representation means that: the two behavioral representations are similar;

[0070] The correlation between the owner’s behavioral representation and the owner means that: the owner has his or her own behavioral characteristics;

[0071] The correlation between the vehicle environment and the driver's behavioral characteristics means that the vehicle environment will affect the driver's behavioral characteristics.

[0072] Step 300: construct second structured data including a structural dimension based on the vehicle owner identity verification history data. The basic structural elements included in this structural dimension are time series units. The structuring method includes:

[0073] Extract the vehicle owner identity authentication history record data, divide the extracted features of the vehicle owner identity authentication history record data into multiple parts according to the data collection period, map one part to a time series unit, and the order of the time series units is consistent with the time order of the collection period of the vehicle owner identity authentication history record data.

[0074] In one embodiment of the present invention, the identity authentication history result includes: the time, result and authentication method of each identity authentication recorded in history, and the result includes pass or fail.

[0075] In one embodiment of the present invention, the vehicle owner's identity association information and the vehicle owner's identity verification history data are input into the identity verification model after undergoing feature engineering;

[0076] The vehicle owner identity-related information includes a description of the vehicle owner's basic information, the vehicle owner's biological information, the vehicle owner's behavior, and the vehicle environment.

[0077] The WordEmbedding algorithm is used as a feature engineering method to describe the owner's basic information, the owner's behavior, and the vehicle environment.

[0078] The vehicle owner's biometric information includes facial images, fingerprint images, iris images, and voice samples. The facial images, fingerprint images, and iris images are processed using the Convolutional Neural Network (CNN) algorithm as feature engineering. The voice samples are processed using the Mel-Frequency Cepstral Coefficient (MFCC) algorithm as feature engineering.

[0079] The owner authentication history data includes authentication history description text, which is processed using the WordEmbedding algorithm as a feature engineering method.

[0080] Step 400: Input the first structured data and the second structured data into the identity verification model. The identity verification model includes a backbone network, which includes a first data structure recognition layer, a second data structure recognition layer, a third data structure recognition layer, a first data structure recognition fusion layer, a first fully connected layer, and a second fully connected layer. The first structured data is input into the first data structure recognition layer, which outputs a first hidden feature to the second data structure recognition layer; the second data structure recognition layer outputs a second hidden feature to the first data structure recognition fusion layer; the first data structure recognition fusion layer outputs the first fused hidden feature to the first fully connected layer; and the first fully connected layer outputs the result of cross-verification of the vehicle owner's identity.

[0081] The second structured data is input into the third data structure recognition layer, which outputs the third hidden feature to the first data structure recognition fusion layer and the second fully connected layer; the second fully connected layer outputs the probability distribution of identity authentication anomalies.

[0082] In one embodiment of the present invention, the calculation formula of the first data structure identification layer is as follows:

[0083]

[0084] represents the object hidden features of the v-th object in the L-th layer in the first structured data, represents the object hidden features of the u-th object in the L-1-th layer, and Respectively represent the set of objects that have object relationships with object v and object u, represents the cardinality of the set, represents the plane recognition weight matrix of the Lth layer of the first data structure recognition layer, , E represents the total number of layers, when L=1 , Represents the features associated with the u-th object, when L=E is equal to the first hidden feature of object v, is the sigmoid activation function.

[0085] In one embodiment of the present invention, the calculation formula of the second data structure identification layer is as follows:

[0086]

[0087] in, and Represents the activation vector of the reset gate and update gate at step t; represents the candidate state generated in step t; represents the second hidden feature of the t-th step, represents the second hidden feature of step t-1; 、 、 、 、 、 Represent the transformation matrices of the 1st, 2nd, 3rd, 4th, 5th, and 6th trainable parameters respectively; 、 、 Indicates the 1st, 2nd, and 3rd trainable bias parameters; , It represents the first hidden feature of the object v output when the t-th time sequence unit inputs the first data structure recognition layer, , Indicates the number of time series units of the first structured data, hour, ,at this time represents the tth time sequence unit in the first structured data, Represents the collection of all objects; Represents the Sigmod activation function; Represents the tanh activation function.

[0088] In one embodiment of the present invention, the calculation formula of the third data structure identification layer is as follows:

[0089]

[0090] in Indicates the A third hidden feature, Indicates the A third hidden feature, 1, Indicates the number of time series units of the second structured data, Indicates the first Timing units, and is the transformation matrix of the 7th and 8th trainable parameters, is the fourth trainable bias parameter and ReLU is the ReLU activation function.

[0091] In one embodiment of the present invention, the calculation formula for the first data structure identification fusion layer is as follows:

[0092]

[0093] in, represents the combined features, represents the feature combination function (splicing function or summation function), Indicates the A third hidden feature, Indicates the A second hidden feature.

[0094] In one embodiment of the present invention, the calculation formula of the first fully connected layer is as follows:

[0095]

[0096] in represents the first output matrix, where the first row and the ath column of the matrix represent the biometric similarity of the ath owner; the second row and the bth column of the matrix represent the behavioral similarity of the bth owner; the third row and the cth column of the matrix represent the identity verification result of the cth owner; the fourth row and the dth column of the matrix represent the safety measures of the dth owner; the fifth row and the eth column of the matrix represent the abnormal handling measures during the driving process of the eth owner; represents the first fusion hidden feature, Indicates full connection.

[0097] In one embodiment of the present invention, the biometric similarity includes: similarity between biometric representations such as face, fingerprint, and iris and biometric representations pre-stored in the system;

[0098] Behavioral similarity includes the similarity between operating modes and driving habits and pre-stored behaviors. Operating modes include key usage, button usage sequence, and vehicle system usage habits. Driving habits include seat adjustment, ventilation, heating, and other habits before driving.

[0099] The verification result is obtained by weighting the similarity between biological representation and behavioral representation. If the weighted result exceeds the set threshold, it means that the identity verification is passed, otherwise it fails.

[0100] Safety measures include: driving authority, additional verification, automatic alarm, remote control;

[0101] Driving authority is expressed as driving authority given according to the authentication result, including: full authorization, denied authorization;

[0102] Additional verification is used when the owner's identity verification fails or someone else is driving the vehicle, including: SMS verification code, voice verification, security questions, spare key, emergency contact verification, and authorized device verification. Authorized device verification refers to verification on another person's device authorized in advance by the owner;

[0103] Automatic alert means it is issued when the identity verification fails three times in a row;

[0104] Remote control means that when an automatic alarm is sounded, the owner can remotely control the vehicle to lock, slow down, or pull over;

[0105] Exception handling measures are those taken when the driver's identity verification fails continuously due to a sudden illness or traffic accident while driving. These measures include: slowing down and pulling over (this measure is performed on roads where pulling over is allowed) and making an emergency call.

[0106] In one embodiment of the present invention, the calculation formula of the second fully connected layer is as follows:

[0107]

[0108] in represents the first output vector, whose fth component represents the probability distribution of the fth owner identity authentication anomaly, Indicates the A third hidden feature, Indicates full connection.

[0109] In one embodiment of the present invention, identity authentication anomalies include: system failure, environmental interference, changes in biometrics, changes in driving behavior, and malicious attacks;

[0110] System failures include: failure of biological characterization acquisition equipment, failure of acquisition data transmission, and algorithm failure;

[0111] Environmental interference includes: light changes, noise interference;

[0112] Changes in biometrics: changes in the owner's facial expression, fingerprint status, iris features, etc., resulting in a mismatch with the registration template;

[0113] Changes in driving behavior: The driver's driving habits, operating modes, etc. change and deviate from the normal behavior model.

[0114] Changes in physical condition: The owner may exhibit behavioral characteristics different from usual due to illness, fatigue, or mood swings.

[0115] Malicious attacks include: biological representation forgery, behavior imitation, and data tampering;

[0116] Data tampering occurs when an attacker invades the authentication system, tampers with historical verification data or biometric templates, and undermines the credibility of the verification results.

[0117] In one embodiment of the present invention, the first data structure recognition fusion layer is combined with the first fully connected layer for pre-training, and the identity verification model is trained to have the ability to accurately determine the identity of the vehicle owner; the loss function calculation formula for training is as follows:

[0118]

[0119]

[0120] in, represents the total loss, represents the first loss, Indicates the number of car owners whose identities are to be verified. represents the true biological similarity of the qth car owner, represents the biometric similarity of the qth vehicle owner output in the first output matrix.

[0121]

[0122] in represents the second loss, Indicates the number of car owners whose identities are to be verified. Indicates the The real behavior similarity of each car owner, Represents the output of the first output matrix The behavioral similarity of the car owners.

[0123]

[0124] in represents the third loss, Indicates the number of car owners whose identities are to be verified. It is The real identity verification tag of each car owner, with a value of 0 or 1, 1 means the verification is passed, and 0 means the verification is not passed. is the first output vector The owner's authentication tag, Represents the logarithmic function with the natural constant e as the base.

[0125] In one embodiment of the present invention, the third data structure recognition layer is pre-trained in conjunction with the second fully connected layer to train the identity verification model to accurately determine whether the vehicle owner's identity is abnormal and the abnormal category; the loss function calculation formula for training is as follows:

[0126]

[0127] in represents the fourth loss, M represents the total number of abnormal categories for authentication, Indicates the number of car owners whose identities are to be verified. It is The symbol value of the owner of the car, if The abnormal category of the real identity authentication of the car owner is the cth category and its value is 1, otherwise it is 0. is the first output vector output The probability that the abnormal category of the owner's identity verification belongs to the cth category, Represents the logarithmic function with the natural constant e as the base.

[0128] In one embodiment of the present invention, a multimodal identity cross-verification system includes the following modules:

[0129] A data collection module is used to obtain the vehicle owner's identity association information and the vehicle owner's identity verification history data;

[0130] a structured data construction module, configured to construct the vehicle owner identity association information into first structured data, and the vehicle owner identity verification history record data into second structured data including a structure dimension;

[0131] The structured data processing module is used to input the first structured data and the second structured data into the identity authentication model, and output the result of the cross-verification of the vehicle owner's identity and the probability distribution of the identity authentication anomaly.

[0132] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Ordinary technicians in this field can also make more forms of equivalent embodiments based on the inspiration of this embodiment, all of which are protected by this embodiment.

Claims

1. A multimodal identity cross-validation method, characterized in that: The following steps are involved: Step 100, obtaining the vehicle owner's identity association information and the vehicle owner's identity verification history record data; Step 200, constructing first structured data based on the vehicle owner identity association information; The first structured data is represented as ordered information of two structural dimensions, the basic structural elements included in the first structural dimension are objects; the basic structural elements included in the second structural dimension are time series units; Step 300, constructing second structured data including a structural dimension based on the vehicle owner identity authentication history record data, wherein the basic structural element included in this structural dimension is a time series unit; Step 400, inputting the first structured data and the second structured data into an identity authentication model, the identity authentication model comprising a backbone network, the backbone network comprising a first data structure identification layer, a second data structure identification layer, a third data structure identification layer, a first data structure identification fusion layer, a first fully connected layer, and a second fully connected layer; The first structured data is input into the first data structure recognition layer, and the first hidden feature is output to the second data structure recognition layer; the second hidden feature output by the second data structure recognition layer is input to the first data structure recognition fusion layer; the first data structure recognition fusion layer outputs the first fused hidden feature to the first fully connected layer; The first fully connected layer outputs the result of cross-verification of the car owner’s identity; The second structured data is input into the third data structure recognition layer, and the third hidden feature is output to the first data structure recognition fusion layer and the second fully connected layer; the second fully connected layer outputs the probability distribution of identity authentication anomalies; The calculation formula of the first data structure identification layer is as follows: represents the object hidden features of the v-th object in the L-th layer in the first structured data, represents the object hidden features of the u-th object in the L-1-th layer, and Respectively represent the set of objects that have object relationships with object v and object u, represents the cardinality of a set, represents the plane recognition weight matrix of the Lth layer of the first data structure recognition layer, , E represents the total number of layers, when L=1 , Represents the features associated with the u-th object, when L=E is equal to the first hidden feature of object v, is the sigmoid activation function; The calculation formula of the second data structure identification layer is as follows: in, and represents the activation vector of the reset gate and update gate at step t; represents the candidate state generated in step t; represents the second hidden feature of the t-th step, represents the second hidden feature of the t-1th step; , , , , , Represent the transformation matrices of the 1st, 2nd, 3rd, 4th, 5th, and 6th trainable parameters respectively; , , Represents the 1st, 2nd, and 3rd trainable bias parameters; , represents the first hidden feature of the object v output when the t-th time sequence unit inputs the first data structure recognition layer, , represents the number of sequential units of the first structured data, hour, ,at this time represents the tth time sequence unit in the first structured data, Represents the collection of all objects; Represents the Sigmod activation function; represents the tanh activation function; The calculation formula of the third data structure identification layer is as follows: in Indicates A third hidden feature, Indicates A third hidden feature, 1, represents the number of time series units of the second structured data, Indicates the first Timing units, and is the transformation matrix of the 7th and 8th trainable parameters, is the fourth trainable bias parameter, ReLU is the ReLU activation function; The calculation formula of the first data structure identification fusion layer is as follows: in, represents the combined features, represents the feature combination function, Indicates A third hidden feature, Indicates A second hidden feature.

2. The multi-modal identity cross-verification method according to claim 1, characterized in that: The vehicle owner's identity-related information includes the vehicle owner's basic information, the vehicle owner's biological representation information, the vehicle owner's behavioral representation information, and the vehicle's environmental information; The vehicle owner's identity authentication history record data includes: identity authentication history record results, historical biological representation similarity, historical behavioral representation similarity, and identity authentication history record description text composed of historical vehicle environment information.

3. The multi-modal identity cross-verification method according to claim 1, characterized in that: The method of constructing the first structured data includes: Objects for extracting orderliness information; Extract the features associated with the object, and divide the extracted features into multiple parts according to the collection period. One part is mapped to a time sequence unit. The order of the time sequence units is consistent with the time order of the feature collection period. Establishing a mapping between the extracted objects and their associated features; For any two objects, if there is a correlation between the associated features related to the authentication task or there is a correlation between the objects related to the authentication task, an object relationship is established between the two objects.

4. The multi-modal identity cross-verification method according to claim 3, characterized in that: The objects in the first structured data include: the vehicle owner, the vehicle owner's biological representation, the vehicle owner's behavioral representation, and the vehicle environment.

5. The multi-modal identity cross-verification method according to claim 4, characterized in that: The characteristics associated with the owner include: owner name, contact information, and emergency contact; The features associated with the owner’s biometric representation include: facial image, fingerprint image, iris image, and voice sample; Features associated with the owner's behavior representation include: key usage data, key usage sequence, vehicle system operation log, and seat adjustment data; The characteristics of vehicle-environment association include: vehicle status, external environment; Vehicle status includes: vehicle speed, steering wheel angle, accelerator and brake pedal data, and vehicle position; the external environment includes: external temperature, humidity, light, weather conditions, visibility, and ground conditions; weather conditions include: clear, cloudy, overcast, rain and snow, haze, thunderstorm, and typhoon; ground conditions include: dry, wet, waterlogged, iced, snowed, muddy, sandy, oily, and under construction.

6. The multi-modal identity cross-verification method according to claim 3, characterized in that: Dependencies related to the identity verification task include: The correlation between car owners means that: the two car owners drive the same vehicle model; The correlation between the biological representation of the car owner and the car owner means that: the car owner has his own biological representation; The correlation between the behavior representation of the owner and the behavior representation of the vehicle owner means that: the two behavior representations are similar; The correlation between the behavioral representation of the car owner and the car owner means that the car owner has his own behavioral characteristics; The correlation between the vehicle environment and the behavior representation of the owner means that the vehicle environment will affect the behavioral characteristics of the owner.

7. The multi-modal identity cross-verification method according to claim 1, characterized in that: The method of constructing the second structured data includes: The vehicle owner identity authentication history record data is extracted, and the features of the extracted vehicle owner identity authentication history record data are divided into multiple parts according to the data collection cycle, and one part is mapped to a time series unit, and the order of the time series units is consistent with the time order of the collection cycle of the vehicle owner identity authentication history record data.

8. The multi-modal identity cross-verification method according to claim 1, characterized in that: The first data structure recognition fusion layer is combined with the first fully connected layer for pre-training, and the identity authentication model is trained to have the ability to accurately determine the identity of the car owner.

9. The multi-modal identity cross-verification method according to claim 1, characterized in that: The third data structure recognition layer is combined with the second fully connected layer for pre-training, and the identity authentication model is trained to have the ability to accurately determine whether the owner's identity is abnormal and the abnormal category.

10. A multi-modal identity cross-verification system, characterized in that: It is used to execute a multi-modal identity cross-verification method as described in any one of claims 1 to 9. A multi-modal identity cross-verification system includes the following modules: A data collection module, used to obtain the vehicle owner's identity association information and the vehicle owner's identity verification history record data; A structured data construction module, used to construct the vehicle owner identity association information into first structured data, and the vehicle owner identity authentication history record data into second structured data including a structure dimension; The structured data processing module is used to input the first structured data and the second structured data into the identity authentication model, and output the result of the cross-verification of the vehicle owner's identity and the probability distribution of the identity authentication anomaly.

Images