Identity cross verification method and system based on multiple modes

Abstract

The invention relates to the technical field of deep learning, and discloses a multi-modal-based identity cross verification method and system, and the method comprises the following steps: obtaining vehicle owner identity association information and vehicle owner identity verification historical record data; constructing first structured data based on the vehicle owner identity association information, and constructing second structured data including a structural dimension based on the vehicle owner identity verification historical record data; inputting the first structured data and the second structured data into an identity verification model; and outputting a vehicle owner identity cross verification result and the probability distribution of identity verification abnormity. According to the method, the multi-modal data associated with the individual identity can be comprehensively utilized, the reliability and safety of identity verification can be improved, and the application range of the method is expanded.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and more specifically, to a multimodal identity cross-verification method and system. Background Art

[0002] With the rapid development of mobile Internet and artificial intelligence technology, identity authentication technology based on multimodal data has become an important means to solve identity recognition and verification problems. Traditional identity authentication methods mainly rely on single verification factors such as passwords and identity documents, which have the problems of low security and easy to be hacked. The identity cross-verification method based on multimodal data can comprehensively utilize the owner's biometrics, behavioral habits, vehicle environment and other information to confirm the identity, which has higher security and reliability. However, the existing multimodal identity authentication technology still has some shortcomings. For example, most methods simply splice data of various modalities without considering the intrinsic connection and time series characteristics between different modal data. Summary of the invention

[0003] The present invention provides a multi-modal identity cross-verification method and system, which first obtains the identity-related information and historical verification records of the car owner, and constructs them into two types of structured data: one is two-dimensional structured data containing objects such as the car owner, biometrics, behavioral characteristics, vehicle environment, and time series units; the other is historical identity verification record data of a pure time series unit structure. Then, these two types of structured data are input into a special identity verification model for processing. The model adopts a multi-layer neural network structure, including multiple data structure recognition layers, fusion layers, and fully connected layers, which can extract the features of each type of data respectively, and fuse them, and finally give the identity verification results and abnormality probabilities. The present invention solves the technical problems of insufficient security of traditional single-modal identity verification and the lack of spatiotemporal modeling and abnormality detection capabilities of existing multi-modal identity verification methods.

[0004] The present invention provides a multi-modal identity cross-verification method, comprising the following steps: Step 100, obtaining the vehicle owner's identity association information and the vehicle owner's identity verification history record data; Step 200, constructing first structured data based on the vehicle owner identity association information; the first structured data is represented as ordered information of two structural dimensions, the basic structural elements included in the first structural dimension are objects; the basic structural elements included in the second structural dimension are time series units; Step 300, constructing second structured data including a structural dimension based on the vehicle owner identity authentication history record data, wherein the basic structural element included in this structural dimension is a time series unit; Step 400, input the first structured data and the second structured data into the identity authentication model, the identity authentication model includes a backbone network, the backbone network includes a first data structure recognition layer, a second data structure recognition layer, a third data structure recognition layer, a first data structure recognition fusion layer, a first fully connected layer, and a second fully connected layer; wherein the first structured data is input into the first data structure recognition layer, and the first hidden feature is output to the second data structure recognition layer; the second hidden feature output by the second data structure recognition layer is output to the first data structure recognition fusion layer; the first data structure recognition fusion layer outputs the first fused hidden feature to the first fully connected layer; the first fully connected layer outputs the result of cross-verification of the vehicle owner's identity; the second structured data is input into the third data structure recognition layer, and the third hidden feature is output to the first data structure recognition fusion layer and the second fully connected layer; the second fully connected layer outputs the probability distribution of identity authentication anomalies.

[0005] In a preferred embodiment, the owner's identity association information includes the owner's basic information, the owner's biometric representation information, the owner's behavioral representation information, and the vehicle environment information; the owner's identity authentication history record data includes: an identity authentication history record description text composed of the identity authentication history record results, historical biometric representation similarity, historical behavioral representation similarity, and historical vehicle environment information.

[0006] In a preferred embodiment, the method for constructing the first structured data includes: Objects for extracting orderliness information; Extract the features associated with the object, and divide the extracted features into multiple parts according to the collection period. One part is mapped to a time sequence unit. The order of the time sequence units is consistent with the time order of the feature collection period. Establishing a mapping between the extracted objects and their associated features; For any two objects, if there is a correlation between the associated features related to the authentication task or there is a correlation between the objects related to the authentication task, an object relationship is established between the two objects.

[0007] In a preferred embodiment, the objects in the first structured data include: a vehicle owner, a vehicle owner's biological representation, a vehicle owner's behavioral representation, and a vehicle environment.

[0008] In a preferred embodiment, the characteristics associated with the vehicle owner include: the vehicle owner's name, contact information, and emergency contact; The features associated with the owner’s biometric representation include: facial image, fingerprint image, iris image, and voice sample; Features associated with the owner's behavior representation include: key usage data, key usage sequence, vehicle system operation log, and seat adjustment data; The characteristics of vehicle-environment association include: vehicle status, external environment; Vehicle status includes: vehicle speed, steering wheel angle, accelerator and brake pedal data, and vehicle position; the external environment includes: external temperature, humidity, light, weather conditions, visibility, and ground conditions; weather conditions include: clear, cloudy, overcast, rain and snow, haze, thunderstorm, and typhoon; ground conditions include: dry, wet, waterlogged, iced, snowed, muddy, sandy, oily, and under construction.

[0009] In a preferred embodiment, the associations related to the identity verification task include: The correlation between car owners means that: the two car owners drive the same vehicle model; The correlation between the biological representation of the car owner and the car owner means that: the car owner has his own biological representation; The correlation between the behavior representation of the owner and the behavior representation of the vehicle owner means that: the two behavior representations are similar; The correlation between the behavioral representation of the car owner and the car owner means that the car owner has his own behavioral characteristics; The correlation between the vehicle environment and the behavior representation of the owner means that the vehicle environment will affect the behavioral characteristics of the owner.

[0010] In a preferred embodiment, the method for constructing the second structured data includes: The vehicle owner identity authentication history record data is extracted, and the features of the extracted vehicle owner identity authentication history record data are divided into multiple parts according to the data collection cycle, and one part is mapped to a time series unit, and the order of the time series units is consistent with the time order of the collection cycle of the vehicle owner identity authentication history record data.

[0011] In a preferred embodiment, the first data structure recognition fusion layer is combined with the first fully connected layer for pre-training, and the identity authentication model is trained to have the ability to accurately determine the identity of the vehicle owner.

[0012] In a preferred embodiment, the third data structure recognition layer is combined with the second fully connected layer for pre-training, and the identity authentication model is trained to have the ability to accurately determine whether the owner's identity is abnormal and the abnormal category.

[0013] In a preferred embodiment, a multi-modal identity cross-verification system includes the following modules: A data collection module, used to obtain the vehicle owner's identity association information and the vehicle owner's identity verification history record data; A structured data construction module, used to construct the vehicle owner identity association information into first structured data, and the vehicle owner identity authentication history record data into second structured data including a structure dimension; The structured data processing module is used to input the first structured data and the second structured data into the identity authentication model, and output the result of the cross-verification of the vehicle owner's identity and the probability distribution of the identity authentication anomaly.

[0014] The beneficial effects of the present invention are: Improving the accuracy of identity authentication: The present invention introduces multimodal information related to the identity of the vehicle owner and considers the spatiotemporal structure characteristics of the data to construct a multi-layer neural network model, which can fully explore the correlation and temporal evolution laws between different modal data, thereby greatly improving the accuracy of identity authentication.

[0015] Enhanced identity authentication security: The present invention introduces an anomaly detection mechanism into the identity authentication model. By modeling and analyzing historical authentication records, it can timely discover potential identity theft, illegal access and other anomalies, and take corresponding preventive measures, thereby enhancing the security of identity authentication and effectively protecting the legitimate rights and interests of car owners. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a multi-modal identity cross-verification method of the present invention. DETAILED DESCRIPTION

[0017] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the present specification. Various examples may omit, replace, or add various processes or components as needed. In addition, the features described in some examples may also be combined in other examples.

[0018] At least one embodiment of the present invention discloses a multi-modal identity cross-verification method, such as Figure 1 As shown, the following steps are included: Step 100, obtaining the vehicle owner's identity association information and the vehicle owner's identity authentication history record data, wherein the vehicle owner's identity association information includes the vehicle owner's basic information, the vehicle owner's biological representation information, the vehicle owner's behavioral representation information, and the vehicle environment information; the vehicle owner's identity authentication history record data includes: an identity authentication history record result, a historical biological representation similarity, a historical behavioral representation similarity, and an identity authentication history record description text composed of historical vehicle environment information; In one embodiment of the present invention, the vehicle owner identity association information and the vehicle owner identity verification history record data are collected once every fixed time, and the fixed time is 60 minutes.

[0019] Step 200, constructing first structured data based on the vehicle owner identity association information; the first structured data is represented as ordered information of two structural dimensions, the basic structural elements included in the first structural dimension are objects, and the objects include: vehicle owner, vehicle owner biological representation, vehicle owner behavior representation, and vehicle environment; the basic structural elements included in the second structural dimension are time sequence units; In some embodiments of the present invention, the method of constructing the first structured data includes: Objects for extracting orderliness information; Extract the features associated with the object, and divide the extracted features into multiple parts according to the collection period. One part is mapped to a time sequence unit. The order of the time sequence units is consistent with the time order of the feature collection period. Establishing a mapping between the extracted objects and their associated features; For any two objects, if there is a correlation between the associated features related to the authentication task or there is a correlation between the objects related to the authentication task, an object relationship is established between the two objects.

[0020] In some embodiments of the present invention, the characteristics associated with the vehicle owner include: the vehicle owner's name, contact information, and emergency contact; The features associated with the owner’s biometric representation include: facial image, fingerprint image, iris image, and voice sample; Features associated with the owner's behavior representation include: key usage data, key usage sequence, vehicle system operation log, and seat adjustment data; The characteristics of vehicle-environment association include: vehicle status, external environment; Vehicle status includes: vehicle speed, steering wheel angle, accelerator and brake pedal data, and vehicle position; the external environment includes: external temperature, humidity, light, weather conditions, visibility, and ground conditions; weather conditions include: clear, cloudy, overcast, rain and snow, haze, thunderstorm, and typhoon; ground conditions include: dry, wet, waterlogged, iced, snowed, muddy, sandy, oily, and under construction.

[0021] In some embodiments of the present invention, the associations related to the identity authentication task include: The correlation between car owners means that: the two car owners drive the same vehicle model; The correlation between the biological representation of the car owner and the car owner means that: the car owner has his own biological representation; The correlation between the behavior representation of the owner and the behavior representation of the vehicle owner means that: the two behavior representations are similar; The correlation between the behavioral representation of the car owner and the car owner means that the car owner has his own behavioral characteristics; The correlation between the vehicle environment and the behavior representation of the owner means that the vehicle environment will affect the behavioral characteristics of the owner.

[0022] Step 300, constructing second structured data including a structural dimension based on the vehicle owner identity authentication history record data, wherein the basic structural element included in this structural dimension is a time sequence unit, and the structuring method includes: The vehicle owner identity authentication history record data is extracted, and the features of the extracted vehicle owner identity authentication history record data are divided into multiple parts according to the data collection cycle, and one part is mapped to a time series unit, and the order of the time series units is consistent with the time order of the collection cycle of the vehicle owner identity authentication history record data.

[0023] In one embodiment of the present invention, the identity authentication history record result includes: the time, result and authentication method of each identity authentication recorded in history, and the result includes pass and fail.

[0024] In one embodiment of the present invention, the vehicle owner's identity association information and the vehicle owner's identity verification history record data are input into the identity verification model after feature engineering; The vehicle owner identity-related information includes a description text of the vehicle owner's basic information, the vehicle owner's biological representation information, the vehicle owner's behavior representation description text, and the vehicle environment description text; The WordEmbedding algorithm is used as a feature engineering method to describe the basic information of the owner, the description of the owner's behavior, and the description of the vehicle environment. The vehicle owner's biometric information includes facial images, fingerprint images, iris images, and voice samples. The facial images, fingerprint images, and iris images are used as feature engineering methods through the convolutional neural network (CNN) algorithm. The voice samples are used as feature engineering methods through the Mel frequency cepstral coefficient (MFCC) algorithm. The owner identity verification history data includes the identity verification history description text, which is processed by the WordEmbedding algorithm as a feature engineering method.

[0025] Step 400, input the first structured data and the second structured data into the identity authentication model, the identity authentication model includes a backbone network, the backbone network includes a first data structure recognition layer, a second data structure recognition layer, a third data structure recognition layer, a first data structure recognition fusion layer, a first fully connected layer, and a second fully connected layer; wherein the first structured data is input into the first data structure recognition layer, and the first hidden feature is output to the second data structure recognition layer; the second hidden feature output by the second data structure recognition layer is output to the first data structure recognition fusion layer; the first data structure recognition fusion layer outputs the first fused hidden feature to the first fully connected layer; the first fully connected layer outputs the result of the cross-verification of the identity of the vehicle owner; The second structured data is input into the third data structure recognition layer, and the third hidden feature is output to the first data structure recognition fusion layer and the second fully connected layer; the second fully connected layer outputs the probability distribution of identity authentication anomalies.

[0026] In one embodiment of the present invention, the calculation formula of the first data structure identification layer is as follows:

[0027] represents the object hidden features of the v-th object in the L-th layer in the first structured data, represents the object hidden features of the u-th object in the L-1-th layer, and Respectively represent the set of objects that have object relationships with object v and object u, represents the cardinality of a set, represents the plane recognition weight matrix of the Lth layer of the first data structure recognition layer, , E represents the total number of layers, when L=1 , Represents the features associated with the u-th object, when L=E is equal to the first hidden feature of object v, is the sigmoid function.

[0028] In one embodiment of the present invention, the calculation formula of the second data structure identification layer is as follows:

[0029] in, and represents the activation vector of the reset gate and update gate at step t; represents the candidate state generated in step t; represents the second hidden feature of the t-th step, represents the second hidden feature of the t-1th step; , , , , , Represent the transformation matrices of the 1st, 2nd, 3rd, 4th, 5th, and 6th trainable parameters respectively; , , Represents the 1st, 2nd, and 3rd trainable bias parameters; , represents the first hidden feature of the object v output when the t-th time sequence unit inputs the first data structure recognition layer, , represents the number of sequential units of the first structured data, hour, ,at this time represents the tth time sequence unit in the first structured data, Represents the collection of all objects; Represents the Sigmod activation function; Represents the tanh activation function.

[0030] In one embodiment of the present invention, the calculation formula of the third data structure identification layer is as follows:

[0031] in Indicates A third hidden feature, Indicates A third hidden feature, 1, represents the number of time series units of the second structured data, Indicates the first Timing units, and is the transformation matrix of the 7th and 8th trainable parameters, is the fourth trainable bias parameter and ReLU is the ReLU activation function.

[0032] In one embodiment of the present invention, the calculation formula for the first data structure to identify the fusion layer is as follows:

[0033] in, represents the combined features, represents the feature combination function (concatenation function or summation function), Indicates A third hidden feature, Indicates A second hidden feature.

[0034] In one embodiment of the present invention, the calculation formula of the first fully connected layer is as follows:

[0035] in represents the first output matrix, the first row and the ath column of the matrix represent the biological similarity of the ath owner; the second row and the bth column of the matrix represent the behavioral similarity of the bth owner; the third row and the cth column of the matrix represent the identity verification result of the cth owner; the fourth row and the dth column of the matrix represent the safety measures of the dth owner; the fifth row and the eth column of the matrix represent the abnormal handling measures during the driving process of the eth owner; represents the first fusion hidden feature, Indicates full connection.

[0036] In one embodiment of the present invention, the biometric similarity includes: the similarity between biometric representations such as face, fingerprint, and iris and biometric representations pre-stored in the system; Behavior similarity includes: the similarity between the operation mode, driving habits and the system-prestored behaviors. The operation mode includes: key usage method, key usage sequence, and vehicle system usage habits. Driving habits include seat adjustment, ventilation, heating, etc. before driving. The verification result is obtained by weighting the similarity between biological representation and behavioral representation. If the weighted result exceeds the set threshold, it means that the identity verification is passed, otherwise it fails. Safety measures include: driving authority, additional verification, automatic alarm, remote control; Driving authority is represented by driving authority given according to identity verification results, including: full authorization, denied authorization; Additional verification is used when the owner's identity verification fails or someone else is driving the vehicle, including: SMS verification code, voice verification, security questions, spare keys, emergency contact verification, and authorized device verification. Authorized device verification means verification with another person's device authorized in advance by the owner. Automatic alert means that the identity verification fails three times in a row; Remote control means that when an automatic alarm is sounded, the owner can remotely control the vehicle to lock, slow down, or pull over; Abnormal handling measures are measures taken when the vehicle owner fails to pass identity authentication due to sudden illness or traffic accidents. The measures include: slowing down and pulling over (this measure is taken on roads where parking is allowed) and making an emergency call. In one embodiment of the present invention, the calculation formula of the second fully connected layer is as follows:

[0037] in represents the first output vector, whose fth component represents the probability distribution of the fth owner identity authentication anomaly, Indicates A third hidden feature, Indicates full connection.

[0038] In one embodiment of the present invention, identity authentication anomalies include: system failure, environmental interference, changes in biological representations, changes in driving behavior, and malicious attacks; System failures include: biological characterization acquisition equipment failure, acquisition data transmission failure, and algorithm failure; Environmental interference includes: light changes, noise interference; Changes in biometrics: changes in the owner's facial expression, fingerprint status, iris features, etc., resulting in a mismatch with the registration template; Driving behavior changes: The owner's driving habits, operating modes, etc. change and deviate from the normal behavior model.

[0039] Changes in physical condition: The owner may exhibit behavioral characteristics different from usual due to illness, fatigue, or mood swings.

[0040] Malicious attacks include: biological representation forgery, behavior imitation, and data tampering; Data tampering occurs when an attacker hacks into the authentication system and tampers with historical authentication data or biometric templates, thereby undermining the credibility of the authentication results.

[0041] In one embodiment of the present invention, the first data structure recognition fusion layer is combined with the first fully connected layer for pre-training, and the identity authentication model is trained to have the ability to accurately determine the identity of the vehicle owner; the loss function calculation formula for the training is as follows:

[0042]

[0043] in, represents the total loss, represents the first loss, Indicates the number of car owners whose identities are to be verified. represents the true biological similarity of the qth car owner, represents the biometric similarity of the qth car owner output in the first output matrix.

[0044]

[0045] in represents the second loss, Indicates the number of car owners whose identities are to be verified. Indicates The real behavior similarity of each car owner, Represents the output of the first output matrix The behavior similarity of the car owners.

[0046]

[0047] in represents the third loss, Indicates the number of car owners whose identities are to be verified. It is The real identity verification tag of the car owner, with a value of 0 or 1, 1 means the verification is passed, and 0 means the verification is not passed. is the first output vector The owner's identity verification tag, Represents the logarithmic function with the natural constant e as the base.

[0048] In one embodiment of the present invention, the third data structure recognition layer is combined with the second fully connected layer for pre-training, and the training identity authentication model has the ability to accurately determine whether the owner's identity is abnormal and the abnormal category; the training loss function calculation formula is as follows:

[0049] in represents the fourth loss, M represents the total number of abnormal categories of identity verification, Indicates the number of car owners whose identities are to be verified. It is The symbol value of the owner of the car, if The abnormal category of the real identity authentication of the car owner is the cth category and its value is 1, otherwise it is 0. is the first output vector output The probability that the abnormal category of the owner identity authentication belongs to the cth category, Represents the logarithmic function with the natural constant e as the base.

[0050] In one embodiment of the present invention, a multi-modal identity cross-verification system includes the following modules: A data collection module, used to obtain the vehicle owner's identity association information and the vehicle owner's identity verification history record data; A structured data construction module, used to construct the vehicle owner identity association information into first structured data, and the vehicle owner identity authentication history record data into second structured data including a structure dimension; The structured data processing module is used to input the first structured data and the second structured data into the identity authentication model, and output the result of the cross-verification of the vehicle owner's identity and the probability distribution of the identity authentication anomaly.

[0051] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation mode. The above-mentioned specific implementation mode is merely illustrative and not restrictive. Under the guidance of this embodiment, ordinary technicians in this field can also make more forms of equivalent embodiments, all of which are within the protection of this embodiment.

Claims

1. A multimodal identity cross-validation method, characterized in that: The following steps are involved: Step 100, obtaining the vehicle owner's identity association information and the vehicle owner's identity verification history record data; Step 200, constructing first structured data based on the vehicle owner identity association information; The first structured data is represented as ordered information of two structural dimensions, the basic structural elements included in the first structural dimension are objects; the basic structural elements included in the second structural dimension are time series units; Step 300, constructing second structured data including a structural dimension based on the vehicle owner identity authentication history record data, wherein the basic structural element included in this structural dimension is a time series unit; Step 400, inputting the first structured data and the second structured data into an identity authentication model, the identity authentication model comprising a backbone network, the backbone network comprising a first data structure identification layer, a second data structure identification layer, a third data structure identification layer, a first data structure identification fusion layer, a first fully connected layer, and a second fully connected layer; The first structured data is input into the first data structure recognition layer, and the first hidden feature is output to the second data structure recognition layer; the second hidden feature output by the second data structure recognition layer is input to the first data structure recognition fusion layer; the first data structure recognition fusion layer outputs the first fused hidden feature to the first fully connected layer; The first fully connected layer outputs the result of cross-verification of the car owner’s identity; The second structured data is input into the third data structure recognition layer, and the third hidden feature is output to the first data structure recognition fusion layer and the second fully connected layer; the second fully connected layer outputs the probability distribution of identity authentication anomalies.

2. The multi-modal identity cross-verification method according to claim 1, characterized in that: The vehicle owner's identity-related information includes the vehicle owner's basic information, the vehicle owner's biological representation information, the vehicle owner's behavioral representation information, and the vehicle's environmental information; The vehicle owner's identity authentication history record data includes: identity authentication history record results, historical biological representation similarity, historical behavioral representation similarity, and identity authentication history record description text composed of historical vehicle environment information.

3. The multi-modal identity cross-verification method according to claim 1, characterized in that: The method of constructing the first structured data includes: Objects for extracting orderliness information; Extract the features associated with the object, and divide the extracted features into multiple parts according to the collection period. One part is mapped to a time sequence unit. The order of the time sequence units is consistent with the time order of the feature collection period. Establishing a mapping between the extracted objects and their associated features; For any two objects, if there is a correlation between the associated features related to the authentication task or there is a correlation between the objects related to the authentication task, an object relationship is established between the two objects.

4. The multi-modal identity cross-verification method according to claim 3, characterized in that: The objects in the first structured data include: the vehicle owner, the vehicle owner's biological representation, the vehicle owner's behavioral representation, and the vehicle environment.

5. The multi-modal identity cross-verification method according to claim 4, characterized in that: The characteristics associated with the owner include: owner name, contact information, and emergency contact; The features associated with the vehicle owner’s biometric representation include: facial image, fingerprint image, iris image, and voice sample; Features associated with the owner's behavior representation include: key usage data, key usage sequence, vehicle system operation log, and seat adjustment data; The characteristics of vehicle-environment association include: vehicle status, external environment; Vehicle status includes: vehicle speed, steering wheel angle, accelerator and brake pedal data, and vehicle position; the external environment includes: external temperature, humidity, light, weather conditions, visibility, and ground conditions; weather conditions include: clear, cloudy, overcast, rain and snow, haze, thunderstorm, and typhoon; ground conditions include: dry, wet, waterlogged, iced, snowed, muddy, sandy, oily, and under construction.

6. The multi-modal identity cross-verification method according to claim 3, characterized in that: Dependencies related to the identity verification task include: The correlation between car owners means that: the two car owners drive the same vehicle model; The correlation between the biological representation of the car owner and the car owner means that: the car owner has his own biological representation; The correlation between the behavior representation of the owner and the behavior representation of the vehicle owner means that: the two behavior representations are similar; The correlation between the behavioral representation of the car owner and the car owner means that the car owner has his own behavioral characteristics; The correlation between the vehicle environment and the behavior representation of the owner means that the vehicle environment will affect the behavioral characteristics of the owner.

7. The multi-modal identity cross-verification method according to claim 1, characterized in that: The method of constructing the second structured data includes: The vehicle owner identity authentication history record data is extracted, and the features of the extracted vehicle owner identity authentication history record data are divided into multiple parts according to the data collection cycle, and one part is mapped to a time series unit, and the order of the time series units is consistent with the time order of the collection cycle of the vehicle owner identity authentication history record data.

8. The multi-modal identity cross-verification method according to claim 1, characterized in that: The first data structure recognition fusion layer is combined with the first fully connected layer for pre-training, and the identity authentication model is trained to have the ability to accurately determine the identity of the car owner.

9. The multi-modal identity cross-verification method according to claim 1, characterized in that: The third data structure recognition layer is combined with the second fully connected layer for pre-training, and the identity authentication model is trained to have the ability to accurately determine whether the owner's identity is abnormal and the abnormal category.

10. A multi-modal identity cross-verification system, characterized in that: It is used to execute a multi-modal identity cross-verification method as described in any one of claims 1 to 9. A multi-modal identity cross-verification system includes the following modules: A data collection module, used to obtain the vehicle owner's identity association information and the vehicle owner's identity verification history record data; A structured data construction module, used to construct the vehicle owner identity association information into first structured data, and the vehicle owner identity authentication history record data into second structured data including a structure dimension; The structured data processing module is used to input the first structured data and the second structured data into the identity authentication model, and output the result of the cross-verification of the vehicle owner's identity and the probability distribution of the identity authentication anomaly.

Images