Multimodal recognition method and system applied to Internet of Things card platform

Abstract

The present invention relates to the technical field of one-card platform management methods, and in particular to a multimodal recognition method and system applied to an Internet of Things one-card platform, wherein the method first obtains a multimodal data type set, then obtains the recognition data input by the target user, and performs content verification on the recognition data, and simultaneously performs behavior evaluation, and then deletes the recognition data types in the multimodal data type set that have not yet entered corresponding recognition data based on the content verification results and the behavior evaluation results. Compared with the prior art, in addition to performing content verification on the multimodal recognition data, the present invention also evaluates the behavior pattern of the target user when inputting the recognition data through behavior evaluation, and deletes the categories in the multimodal data type set based on the action evaluation results, thereby reducing the cumbersomeness of the user entering data, and solving the problem that the multimodal recognition method applied to the Internet of Things one-card platform in the prior art is relatively cumbersome.

Description

Technical Field

[0001] The present invention relates to the technical field of one-card platform management methods, and in particular to a multimodal recognition method and system applied to an Internet of Things one-card platform. Background Art

[0002] Multimodal recognition technology is an advanced biometric method that verifies personal identity by comprehensively analyzing multiple biometric features. The significance of this technology lies in its high accuracy and security. Compared with traditional single biometric methods, multimodal recognition can provide more reliable identity authentication services. For example, single fingerprint or facial recognition may be misjudged due to environmental factors or changes in individual characteristics, but multimodal recognition greatly improves the accuracy of recognition by combining multiple biometric features such as fingerprints, faces, and voices.

[0003] Combining multimodal recognition technology with the one-card platform can significantly improve the security of the one-card system. The one-card platform is usually used in campuses, enterprises or communities to provide users with convenient payment, identity authentication and access control functions. However, the traditional one-card system may face the risk of card loss or theft. The introduction of multimodal recognition technology makes the user identity verification no longer rely on physical cards, but confirms the identity by analyzing the user's biometrics. Multimodal recognition technology can effectively identify and prevent illegal card duplication or theft, thereby protecting the user's property safety and personal privacy.

[0004] However, an obvious drawback of multimodal recognition technology is that it requires users to input multimodal recognition data, which is cumbersome for users to input information and significantly reduces the user experience. Therefore, people need a multimodal recognition technology that can take into account both security and convenience and is applied to the IoT card platform. Summary of the invention

[0005] Therefore, the present invention provides a multimodal recognition method and system applied to an Internet of Things all-in-one card platform, so as to solve the problem that the multimodal recognition method applied to the Internet of Things all-in-one card platform in the prior art is relatively cumbersome.

[0006] The present invention provides a multimodal recognition method applied to an Internet of Things card platform, comprising:

[0007] Acquire a multimodal data type set corresponding to a target user, the multimodal data type set including multiple identification data types to be input verified by the target user;

[0008] Acquire at least one identification data sequentially input by the target user based on the multimodal data type set, and perform content verification on the identification data to obtain a content verification result;

[0009] Conducting a behavior evaluation on the target user's behavior when inputting identification data to obtain a behavior evaluation result;

[0010] According to the content verification result and the behavior evaluation result, delete the identification data type for which the corresponding identification data has not been input in the multimodal data type set;

[0011] When the identification data corresponding to all identification data types in the multimodal data type set pass the content verification, it is determined that the target user verification has passed.

[0012] The present invention also provides a preferred solution: the behavior evaluation result includes behavior similarity, and the behavior similarity represents the degree of similarity between the target user's behavior when inputting identification data this time and the behavior when inputting identification data in the past; according to the content verification result and the behavior evaluation result, the identification data types that have not yet been inputted in the multimodal data type set are deleted, including:

[0013] Calculate the sum of the behavior similarities of the identification data that has passed content verification to obtain a cumulative behavior credibility value;

[0014] According to the cumulative value of the behavior credibility, the target deletion quantity is obtained;

[0015] Based on the target deletion quantity, the identification data categories for which identification data has not yet been input in the multimodal data category set are deleted.

[0016] The present invention also provides a preferred solution: according to the cumulative value of the behavior credibility, the target deletion quantity is obtained, including:

[0017] The target deletion quantity is calculated by the following formula:

[0018] ;

[0019] in, Delete quantity for target, is the floor operator, is the preset linear function, is a natural constant, is the preset bias value, Indicates the behavioral similarity, Indicates the cumulative value of the behavior credibility. Represents the difference in behavior similarity between two recognition data that are continuously evaluated. It represents the sum of the differences in behavior similarity between multiple recognition data that are evaluated continuously.

[0020] The present invention also provides a preferred solution: the multiple identification data include one-card identification data; the behavior of the target user when inputting the identification data is evaluated to obtain a behavior evaluation result, including:

[0021] Get the actual card swiping moment when the target user enters the card identification data;

[0022] Obtain the predicted card swiping time data corresponding to the target user's card identification data;

[0023] Calculate the deviation between the actual card swiping time and the predicted card swiping time data to obtain the accurate card swiping value;

[0024] The card swiping accuracy value is used as the behavior similarity corresponding to the target user's one-card identification data.

[0025] The present invention also provides a preferred solution: predicting the card swiping time data includes predicting the card swiping reference time and predicting the card swiping floating time; calculating the deviation between the actual card swiping time and the predicted card swiping time data to obtain the card swiping accurate value, including:

[0026] Calculate the card swipe accuracy using the following formula:

[0027] ;

[0028] in, To swipe the card accurately, To predict the benchmark time for card swiping, To predict the floating time of card swiping, The actual card swiping moment. and They are different preset adjustment coefficients respectively.

[0029] The present invention also provides a preferred solution:

[0030] Obtain the historical card swiping time record of the target user inputting the card identification data;

[0031] Encode the historical card swiping moment records to obtain the historical card swiping vector sequence;

[0032] Input the historical card swiping vector sequence into the preset neural network model to obtain the predicted card swiping vector sequence output by the preset neural network model;

[0033] According to the predicted card swiping vector sequence, the predicted card swiping base time and the predicted card swiping floating time are obtained.

[0034] The present invention also provides a preferred solution: the historical card swiping vector sequence includes a plurality of historical card swiping vectors arranged in chronological order, the historical card swiping vector includes two elements, and the two elements are used to represent the date and time respectively; the predicted card swiping vector sequence includes a plurality of predicted card swiping vectors arranged in chronological order, the predicted card swiping vector includes two elements, and the two elements are used to represent the date and time respectively; according to the predicted card swiping vector sequence, the predicted card swiping reference time and the predicted card swiping floating time are obtained, including:

[0035] Get the time period and target date corresponding to the predicted card swiping vector sequence. The target date is a day in the time period corresponding to the predicted card swiping vector sequence.

[0036] The time corresponding to the predicted card swiping vector corresponding to the target date is used as the predicted card swiping reference time of the target date;

[0037] According to the statistical characteristics of the time corresponding to the multiple predicted card swiping vectors, the predicted card swiping floating time is obtained.

[0038] The present invention also provides a preferred solution: the behavior evaluation result includes behavior verification passing and behavior verification failing; according to the content verification result and the behavior evaluation result, deleting the identification data types in the multimodal data type set that have not yet been input with identification data, including:

[0039] Check the behavioral assessment results of the identification data verified by the content;

[0040] If the behavior evaluation result of the identification data is that the behavior verification is passed, then a type of identification data in the multimodal data type set that has not yet been input with identification data is deleted.

[0041] The present invention also provides a preferred solution: the behavior evaluation result includes behavior verification passing and behavior verification failing; according to the content verification result and the behavior evaluation result, deleting the identification data types in the multimodal data type set that have not yet been input with identification data, including:

[0042] Count the number of identification data that have continuously passed content verification and whose behavior evaluation results are behavior verification passed;

[0043] If the number exceeds a preset number threshold, all identification data types in the multimodal data type set for which identification data has not yet been input are deleted.

[0044] The present invention also provides a multimodal recognition system applied to an Internet of Things card platform, comprising:

[0045] A data preparation module is used to obtain a multimodal data type set corresponding to a target user, where the multimodal data type set includes multiple identification data types to be input and verified by the target user;

[0046] A content verification module, used to obtain at least one identification data sequentially input by the target user based on the multimodal data type set, and perform content verification on the identification data to obtain a content verification result;

[0047] A behavior evaluation module is used to evaluate the behavior of the target user when inputting identification data to obtain a behavior evaluation result;

[0048] A credit assessment module, used to delete the identification data types for which corresponding identification data has not been input in the multimodal data type set according to the content verification result and the behavior assessment result;

[0049] The verification determination module is used to determine whether the target user has passed the verification when the identification data corresponding to all identification data types in the multimodal data type set have passed the content verification.

[0050] The beneficial effects of adopting the above embodiment are:

[0051] The present invention provides a multimodal recognition method and system applied to an Internet of Things card platform, wherein the method first obtains a multimodal data type set corresponding to a target user, including a plurality of types of identification data to be input and verified by the target user, then obtains at least one identification data sequentially input by the target user based on the multimodal data type set, performs content verification on the identification data to obtain a content verification result, and simultaneously performs a behavior evaluation on the behavior of the target user when inputting the identification data to obtain a behavior evaluation result, then deletes the identification data types in the multimodal data type set for which corresponding identification data has not been input according to the content verification result and the behavior evaluation result, and finally determines that the target user has passed the verification when the identification data corresponding to all the identification data types in the multimodal data type set have passed the content verification. Compared with the prior art, in addition to verifying the content of multimodal recognition data, the present invention also evaluates the behavior pattern of the target user when inputting recognition data through behavior evaluation, and deletes the categories in the multimodal data type set based on the action evaluation result. For example, when the action evaluation result represents that the behavior pattern of the target user is highly credible, the categories in the multimodal data type set can be appropriately reduced, thereby reducing the number of recognition data types to be input by the target user, reducing the tediousness of user data entry, and improving ease of use. At the same time, security is guaranteed through the combination of content verification and behavior evaluation, thereby solving the problem that the multimodal recognition method applied to the Internet of Things card platform in the prior art is relatively cumbersome. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A method flow chart of an embodiment of a multimodal recognition method applied to an Internet of Things card platform provided by the present invention;

[0053] Figure 2 for Figure 1 A specific method flow chart of step S103;

[0054] Figure 3 for Figure 1 A specific method flow chart of step S104;

[0055] Figure 4This is a system structure diagram of an embodiment of a multimodal recognition system applied to an Internet of Things card platform provided by the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Combination Figure 1 As shown, a specific embodiment of the present invention discloses a multimodal recognition method applied to an Internet of Things card platform, including:

[0058] S101, obtaining a multimodal data type set corresponding to a target user, where the multimodal data type set includes multiple types of identification data to be input and verified by the target user;

[0059] S102, obtaining at least one identification data sequentially input by the target user based on the multimodal data type set, and performing content verification on the identification data to obtain a content verification result;

[0060] S103, performing a behavior evaluation on the target user's behavior when inputting identification data, and obtaining a behavior evaluation result;

[0061] S104, according to the content verification result and the behavior evaluation result, deleting the identification data type for which the corresponding identification data has not been input in the multimodal data type set;

[0062] S105 . When the identification data corresponding to all identification data types in the multimodal data type set pass the content verification, it is determined that the target user verification has passed.

[0063] Compared with the prior art, in addition to verifying the content of multimodal recognition data, the present invention also evaluates the behavior pattern of the target user when inputting recognition data through behavior evaluation, and deletes the categories in the multimodal data type set based on the action evaluation result. For example, when the action evaluation result represents that the behavior pattern of the target user is highly credible, the categories in the multimodal data type set can be appropriately reduced, thereby reducing the number of recognition data types to be input by the target user, reducing the tediousness of user data entry, and improving ease of use. At the same time, security is guaranteed through the combination of content verification and behavior evaluation, thereby solving the problem that the multimodal recognition method applied to the Internet of Things card platform in the prior art is relatively cumbersome.

[0064] In the above process, the multimodal data type set can be called according to the authority corresponding to the identity when the first identification data input by the user is identified. The computer can instruct the target user to input the corresponding identification data in sequence based on a random order according to the categories in the multimodal data type set to complete the identity authentication.

[0065] Identification data refers to identity authentication data of different modes input by the user, such as card swipe data, face recognition data, input password, iris recognition data, voice recognition data, etc. The content verification in the above process is to verify the content of these identification data itself, such as whether the card swipe data is correct, whether the password is correct, whether the features of face recognition, iris recognition, and voice recognition match the information entered in advance, etc. Obviously, the results of content verification only include two types: content verification passed and content verification failed. Only when all corresponding types of identification data in the multimodal data type set are verified, can the target user be determined to have passed the verification.

[0066] In practice, if users are required to enter a large number of identification data each time, it is obviously very cumbersome and will greatly affect the user experience. This leads to the improvement of the present invention, that is, using another evaluation dimension, namely action evaluation, to evaluate the credibility of the user when entering the identification data this time. It can also be understood as using action recognition to analyze whether the habitual characteristics of the user's input data this time are the same as usual. If the action evaluation can be used to determine that the user who enters the identification data this time is the same as the user who entered the data in the past, the number of types of identification data that the user needs to enter can be reduced as appropriate, thereby improving the convenience of identity verification and improving the user experience.

[0067] It should be emphasized that the action evaluation in the present invention is mainly used as a means to evaluate the credibility of user actions, rather than a standard for determining whether a user has passed identity authentication. In practice, whether action evaluation is used as a hard indicator for passing identity authentication can be flexibly determined based on specific circumstances.

[0068] Different from content verification, the specific method of determining the behavior evaluation results varies according to the evaluation criteria used when conducting the behavior evaluation, which in turn leads to different specific strategies for subsequently deleting the identified data types in the multimodal data type set.

[0069] Specifically, in a preferred embodiment, the behavior evaluation results include two types: behavior verification passed and behavior verification failed, that is, the behavior evaluation only has a hard evaluation standard of pass or fail. For example, when the user is performing face recognition, the user's gait recognition is performed to determine whether the user's gait recognition is passed; when the user enters the password through voice, in addition to verifying the content of the password itself, the user's voice speed, intonation and other characteristics are also verified to determine whether it is passed; when the user enters the password through a keyboard or other device, in addition to verifying the content of the password itself, the user's speed, strength or rhythm of pressing the button is also verified to determine whether it is passed.

[0070] Further, in a preferred embodiment, for the above behavior evaluation result, the above step S104, based on the content verification result and the behavior evaluation result, deletes the identification data types for which the corresponding identification data has not been input in the multimodal data type set, specifically including:

[0071] Check the behavioral assessment results of the identification data verified by the content;

[0072] If the behavior evaluation result of the identification data is that the behavior verification is passed, then a type of identification data in the multimodal data type set that has not yet been input with identification data is deleted.

[0073] The above method is that every time an identification data that passes the behavior evaluation appears, one type of identification data to be input is reduced until all the identification data to be input are input. The above method is simple and easy and has good feasibility, but its security is relatively general.

[0074] Furthermore, in another preferred embodiment, for the above behavior evaluation result, the above step S104, based on the content verification result and the behavior evaluation result, deletes the identification data type for which the corresponding identification data has not been input in the multimodal data type set, specifically including:

[0075] Count the number of identification data that have continuously passed content verification and whose behavior evaluation results are behavior verification passed;

[0076] If the number exceeds a preset number threshold, all identification data types in the multimodal data type set for which identification data has not yet been input are deleted.

[0077] Compared with the previous embodiment, this embodiment not only requires passing the behavior assessment, but also requires the target user to be able to pass the behavior assessment continuously, and the order in which the user inputs the behavior data can be randomly indicated by the system, which greatly increases the security of this embodiment.

[0078] Furthermore, in a preferred embodiment, the behavior evaluation result includes behavior similarity, which indicates the degree of similarity between the target user's behavior when inputting identification data this time and when inputting identification data in the past. For example, while performing face recognition on the user, gait recognition is performed on the user to evaluate the degree of similarity between the user's gait and the gait of the user previously detected; when the user inputs a password through voice, in addition to verifying the content of the password itself, the user's voice speed, intonation and other features are also verified to evaluate the degree of similarity between the user's voice features and the user's voice features previously detected; when the user inputs a password through a keyboard or other device, in addition to verifying the content of the password itself, the user's speed, strength or rhythm of pressing the button are also verified to evaluate the degree of similarity between the input action features and the input features previously detected.

[0079] It is understandable that the standard for measuring user behavior similarity can be obtained based on the user's advance input, or it can be obtained by analyzing the user's past behavior. For most types of recognition data (such as gait recognition, voice recognition, key action recognition, etc.), the specific determination of the measurement standard and the method of calculating the behavior similarity are existing technologies that can be understood by technical personnel in this field, so this article will not explain in detail.

[0080] The present invention provides an innovative behavior assessment method for the one-card technology, that is, in a preferred embodiment, combined with Figure 2 As shown, the multiple identification data include one-card identification data. The above step S103, the behavior of the target user when inputting the identification data is evaluated to obtain the behavior evaluation result, specifically including:

[0081] S201, obtaining the actual card swiping moment when the target user inputs the card identification data;

[0082] S202, obtaining predicted card swiping time data corresponding to the target user's card identification data;

[0083] S203, calculating the deviation between the actual card swiping time and the predicted card swiping time data to obtain the card swiping accuracy value;

[0084] S204: Use the card swiping accuracy value as the behavior similarity corresponding to the target user's one-card identification data.

[0085] In the above process, the rationality of the time when the user swipes the card is used as an indicator of behavior evaluation. The card swiping accuracy value is used to represent the degree of difference between the actual time when the user swipes the card and the time when the card should be swiped theoretically. The smaller the difference, the higher the card swiping accuracy value. It is used as the behavior similarity, so that the one-card recognition data also has a corresponding action evaluation method, so that the idea of ​​the present invention can be applied to the field of one-card technology.

[0086] The predicted card swipe time data can be a time period, and the deviation between the actual card swipe time and the time period is compared to obtain the accurate card swipe value. In practice, the time when users swipe their cards is obviously highly random, and this method has a high fault tolerance and can well accommodate the existence of such randomness in card swipe time, but it is obviously less secure.

[0087] The predicted card swiping time data can also be a specific moment. The actual card swiping time of the user is compared with this specific moment to obtain the precise card swiping value. This method is more accurate and safer, but when faced with actual conditions, its behavior evaluation criteria appear to be too strict.

[0088] The present invention has made improvements based on this. In a preferred embodiment, the predicted card swiping time data includes the predicted card swiping reference time and the predicted card swiping floating time. The above step S203, calculating the deviation between the actual card swiping time and the predicted card swiping time data to obtain the card swiping accurate value, specifically includes:

[0089] Calculate the card swipe accuracy using the following formula:

[0090] ;

[0091] in, To swipe the card accurately, To predict the benchmark time for card swiping, To predict the floating time of card swiping, The actual card swiping moment. and They are different preset adjustment coefficients respectively.

[0092] In this embodiment, the reasonable time period that is acceptable for users to swipe their cards is characterized by predicting the floating time of swiping the card (the time period obtained by floating based on the predicted floating time of swiping the card before and after the reference time of swiping the card). The significance of the above formula is that it measures the deviation between the actual swiping time and the predicted reference time of swiping the card, as well as the actual swiping time and the reasonable time period, so that the final calculated accurate value of swiping the card takes into account both security and fault tolerance. In the above formula, when the actual swiping time is outside the reasonable time period of swiping the card, the accurate value of swiping the card will change slowly, and when the actual swiping time is within the reasonable time period of swiping the card, the accurate value of swiping the card will increase significantly, and the closer the actual swiping time is to the predicted reference time of swiping the card, the greater the increase in the accurate value of swiping the card.

[0093] The card swiping accuracy value calculation method in this embodiment not only has good fault tolerance, but also can cooperate with the credibility accumulation value to be mentioned later to achieve the flexibility of the overall solution.

[0094] For determining the predicted card swiping time data, any method can be adopted in practice according to the specific circumstances. For the above-mentioned predicted card swiping time data, it is necessary not only to predict the predicted card swiping base moment, but also to make a reasonable prediction of the predicted card swiping floating time. For this, the present invention provides a preferred method.

[0095] Specifically, in a preferred embodiment, the multimodal recognition method applied to the Internet of Things card platform also includes:

[0096] Obtain the historical card swiping time record of the target user inputting the card identification data;

[0097] Encode the historical card swiping moment records to obtain the historical card swiping vector sequence;

[0098] Input the historical card swiping vector sequence into the preset neural network model to obtain the predicted card swiping vector sequence output by the preset neural network model;

[0099] According to the predicted card swiping vector sequence, the predicted card swiping base time and the predicted card swiping floating time are obtained.

[0100] This embodiment predicts the base moment of card swiping and the floating time of card swiping through a preset neural network model. The neural network model can handle complex nonlinear problems such as card swiping time prediction very well. The preset neural network model in this embodiment can be implemented by any existing technology such as recurrent neural network RNN, long short-term memory network LSTM, etc. It should be emphasized that this embodiment utilizes the characteristic of neural network that can predict a large amount of predicted card swiping time data for a period of time in the future, and realizes accurate prediction of the base moment of card swiping and the floating time of card swiping at the same time.

[0101] Specifically, in a preferred embodiment, the historical card swiping vector sequence includes a plurality of historical card swiping vectors arranged in chronological order, and the historical card swiping vector includes two elements, which are used to represent the date and time respectively; the predicted card swiping vector sequence includes a plurality of predicted card swiping vectors arranged in chronological order, and the predicted card swiping vector includes two elements, which are used to represent the date and time respectively. In the above process, the predicted card swiping reference moment and the predicted card swiping floating time are obtained according to the predicted card swiping vector sequence, which specifically includes:

[0102] Get the time period and target date corresponding to the predicted card swiping vector sequence. The target date is a day in the time period corresponding to the predicted card swiping vector sequence.

[0103] The time corresponding to the predicted card swiping vector corresponding to the target date is used as the predicted card swiping reference time of the target date;

[0104] According to the statistical characteristics of the time corresponding to the multiple predicted card swiping vectors, the predicted card swiping floating time is obtained.

[0105] The improvement of the above process mainly lies in the improvement of the vector format of the input preset neural network. In practice, the historical card swiping moment record can be encoded into a monotonically increasing numerical sequence, input into the preset neural network to obtain the input prediction sequence, and decoded to obtain the predicted time. In this embodiment, the date and the specific time of each day in the historical card swiping moment record are separated and encoded to obtain the historical card swiping vector and the predicted card swiping vector.

[0106] On the one hand, from the perspective of date-based scale, the card swiping time has a certain regularity (for example, staff will punch in during working hours but not on weekends), while from the perspective of a day, the card swiping time has a certain randomness (for example, the time to swipe the card for meals is different every day, and the time to go to and from get off work is different, etc.). Therefore, in this embodiment, the date and time are separated and used as two different dimensions to establish the historical card swiping vector, which can greatly improve the accuracy of the preset neural network.

[0107] On the other hand, the predicted card swiping vector sequence obtained by the above encoding method can also provide a data basis for obtaining a more accurate predicted card swiping reference moment and predicted card swiping floating duration. In this embodiment, the element corresponding to the date in the predicted card swiping vector sequence can be used as a reference, and the time corresponding to the predicted card swiping vector corresponding to the target date in the predicted card swiping vector sequence can be directly selected as the card swiping reference moment. At the same time, because the predicted card swiping vector sequence contains data for a period of time in the future, a reasonable predicted card swiping floating duration can be obtained based on the statistical characteristics of the changes in the time elements in different predicted card swiping vector sequences (such as variance, mean value, difference between maximum and minimum values, etc.).

[0108] Based on the types of the above behavior evaluation results, the present invention also provides a preferred method for deleting the types of identification data to be input, combined with Figure 3 As shown, in a preferred embodiment, the above step S104, based on the content verification result and the behavior evaluation result, deletes the identification data types in the multimodal data type set for which the corresponding identification data has not been input, specifically includes:

[0109] S301, calculating the sum of the behavior similarities of the identification data that has passed the content verification, and obtaining a cumulative behavior credibility value;

[0110] S302, obtaining a target deletion quantity according to the accumulated value of the behavior credibility;

[0111] S303: based on the target deletion quantity, delete the identification data types in the multimodal data type set for which identification data has not yet been input.

[0112] Compared with the previous embodiment, this embodiment determines the number of target deletions by calculating the sum of the accumulated similarities and then performs deletion. In practice, no matter how accurate the behavior assessment technology is, it cannot eliminate the impact of random events in daily life. For example, being late for work due to weather conditions will affect the clock-in time of staff, and facial recognition and voice recognition failures due to illness, accidents, etc. Therefore, compared with the hard indicators in the previous article, the cumulative credibility value method is used in this embodiment to more flexibly determine the number of target deletions, so as to improve the fault tolerance of the entire method.

[0113] The above process can be repeated in real time, and the target deletion quantity can be changed in real time. When deleting, it is only necessary to ensure that the total number of identification data types deleted from the multimodal data type set is equal to the target deletion number.

[0114] Specifically, in a preferred embodiment, the above step S302, obtaining the target deletion quantity according to the cumulative value of the behavior credibility, specifically includes:

[0115] The target deletion quantity is calculated by the following formula:

[0116] ;

[0117] in, Delete quantity for target, is the floor operator, is the preset linear function, is a natural constant, is the preset bias value, Indicates behavioral similarity, Indicates the cumulative value of the behavior credibility. Represents the difference in behavior similarity between two recognition data that are continuously evaluated. It represents the sum of the differences in behavior similarity between multiple recognition data that are evaluated continuously.

[0118] The significance of the above formula is that, in addition to directly calculating the number of target deletions linearly based on the cumulative credibility value, the number of target deletions is also calculated by combining the changes in the behavioral similarity of each calculation. In the above formula, the sum of the differences in the behavioral similarity between multiple identification data that are continuously evaluated is counted and used as the exponent of the natural constant. When the user continuously enters multiple compliant identification data and the behavioral similarity of these identification data gradually increases, it indicates that the user's identity credibility is high. At this time, the number of target deletions calculated by the above formula will increase sharply to improve the convenience of the user. On the contrary, when the behavioral similarity corresponding to the identification data entered by the user is not high, the number of target deletions calculated by the above formula will be small, forcing the user to enter multimodal data for strict authentication to ensure security. When the behavioral similarity obtained by the behavioral evaluation of individual identification data is not high due to random factors, the above formula can integrate the multi-dimensional behavioral similarity for an overall evaluation, reasonably tolerate this error, and obtain a reasonable number of target deletions, which takes into account security and convenience to a certain extent. By setting different preset bias values, the speed of change of the number of target deletions can be changed.

[0119] Combination Figure 4 As shown, the present invention also provides a multimodal recognition system applied to an Internet of Things card platform, comprising:

[0120] The data preparation module 410 is used to obtain a multimodal data type set corresponding to the target user, where the multimodal data type set includes multiple identification data types to be input and verified by the target user;

[0121] The content verification module 420 is used to obtain at least one identification data sequentially input by the target user based on the multimodal data type set, and perform content verification on the identification data to obtain a content verification result;

[0122] The behavior evaluation module 430 is used to evaluate the behavior of the target user when inputting identification data to obtain a behavior evaluation result;

[0123] The credit evaluation module 440 is used to delete the identification data types for which corresponding identification data has not been input in the multimodal data type set according to the content verification result and the behavior evaluation result;

[0124] The verification determination module 450 is used to determine whether the target user has passed the verification when the identification data corresponding to all identification data types in the multimodal data type set have passed the content verification.

[0125] It should be noted here that the corresponding system provided in the above embodiments can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can be found in the corresponding contents in the above method embodiments, which will not be repeated here.

[0126] The present invention provides a multimodal recognition method and system applied to an Internet of Things card platform, wherein the method first obtains a multimodal data type set corresponding to a target user, including a plurality of types of identification data to be input and verified by the target user, then obtains at least one identification data sequentially input by the target user based on the multimodal data type set, performs content verification on the identification data to obtain a content verification result, and simultaneously performs a behavior evaluation on the behavior of the target user when inputting the identification data to obtain a behavior evaluation result, then deletes the identification data types in the multimodal data type set for which corresponding identification data has not been input according to the content verification result and the behavior evaluation result, and finally determines that the target user has passed the verification when the identification data corresponding to all the identification data types in the multimodal data type set have passed the content verification. Compared with the prior art, in addition to verifying the content of multimodal recognition data, the present invention also evaluates the behavior pattern of the target user when inputting recognition data through behavior evaluation, and deletes the categories in the multimodal data type set based on the action evaluation result. For example, when the action evaluation result represents that the behavior pattern of the target user is highly credible, the categories in the multimodal data type set can be appropriately reduced, thereby reducing the number of recognition data types to be input by the target user, reducing the tediousness of user data entry, and improving ease of use. At the same time, security is guaranteed through the combination of content verification and behavior evaluation, thereby solving the problem that the multimodal recognition method applied to the Internet of Things card platform in the prior art is relatively cumbersome.

[0127] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0128] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal recognition method applied to an Internet of Things card platform, characterized in that: include: Acquire a multimodal data type set corresponding to a target user, the multimodal data type set including multiple identification data types to be input verified by the target user; Acquire at least one identification data sequentially input by the target user based on the multimodal data type set, and perform content verification on the identification data to obtain a content verification result; Conducting a behavior evaluation on the target user's behavior when inputting identification data to obtain a behavior evaluation result; According to the content verification result and the behavior evaluation result, delete the identification data type for which the corresponding identification data has not been input in the multimodal data type set; When the identification data corresponding to all identification data types in the multimodal data type set pass the content verification, it is determined that the target user has passed the verification; Among them, the behavior evaluation result includes behavior similarity, which represents the degree of similarity between the target user's behavior when inputting identification data this time and when he input identification data in the past; according to the content verification result and the behavior evaluation result, the identification data types that have not yet been input with identification data in the multimodal data type set are deleted, including: Calculate the sum of the behavior similarities of the identification data that has passed content verification to obtain a cumulative behavior credibility value; According to the cumulative value of the behavior credibility, the target deletion quantity is obtained; Based on the target deletion quantity, the identification data categories for which identification data has not yet been input in the multimodal data category set are deleted.

2. The multimodal recognition method applied to the Internet of Things Card Platform according to claim 1 is characterized in that: According to the cumulative value of the behavior credibility, the target deletion quantity is obtained, including: The target deletion quantity is calculated by the following formula: ; in, Delete quantity for target, is the floor operator, is the preset linear function, is a natural constant, is the preset bias value, Indicates the behavioral similarity, Indicates the cumulative value of the behavior credibility. represents the difference in behavior similarity between two recognition data that are continuously evaluated. It represents the sum of the differences in behavior similarity between multiple recognition data that are evaluated continuously.

3. The multimodal recognition method applied to the Internet of Things Card Platform according to claim 2 is characterized in that: Various identification data including card identification data; Perform a behavior evaluation on the target user's behavior when inputting identification data, and obtain a behavior evaluation result, including: Get the actual card swiping moment when the target user enters the card identification data; Obtain the predicted card swiping time data corresponding to the target user's card identification data; Calculate the deviation between the actual card swiping time and the predicted card swiping time data to obtain the accurate card swiping value; The card swiping accuracy value is used as the behavior similarity corresponding to the target user's one-card identification data.

4. The multimodal recognition method applied to the Internet of Things Card Platform according to claim 3 is characterized in that: The predicted card swiping time data includes the predicted card swiping base time and the predicted card swiping floating time; Calculate the deviation between the actual card swiping time and the predicted card swiping time data to obtain the accurate card swiping value, including: Calculate the card swipe accuracy using the following formula: ; in, To swipe the card accurately, To predict the benchmark time for card swiping, To predict the floating time of card swiping, The actual card swiping moment. and They are different preset adjustment coefficients respectively.

5. The multimodal recognition method applied to the Internet of Things Card Platform according to claim 4 is characterized in that: Also includes: Obtain the historical card swiping time record of the target user inputting the card identification data; Encode the historical card swiping moment records to obtain the historical card swiping vector sequence; Input the historical card swiping vector sequence into the preset neural network model to obtain the predicted card swiping vector sequence output by the preset neural network model; According to the predicted card swiping vector sequence, the predicted card swiping base time and the predicted card swiping floating time are obtained.

6. The multimodal recognition method applied to the Internet of Things Card Platform according to claim 5 is characterized in that: The historical card swipe vector sequence includes multiple historical card swipe vectors arranged in chronological order, and the historical card swipe vector includes two elements, which are used to represent the date and time respectively; the predicted card swipe vector sequence includes multiple predicted card swipe vectors arranged in chronological order, and the predicted card swipe vector includes two elements, which are used to represent the date and time respectively; according to the predicted card swipe vector sequence, the predicted card swipe base moment and the predicted card swipe floating time are obtained, including: Get the time period and target date corresponding to the predicted card swiping vector sequence. The target date is a day in the time period corresponding to the predicted card swiping vector sequence. The time corresponding to the predicted card swiping vector corresponding to the target date is used as the predicted card swiping reference time of the target date; According to the statistical characteristics of the time corresponding to the multiple predicted card swiping vectors, the predicted card swiping floating time is obtained.

7. The multimodal recognition method applied to the Internet of Things Card Platform according to claim 1 is characterized in that: Behavioral assessment results include behavioral verification pass and behavioral verification fail; Based on the content verification results and behavior assessment results, delete the identification data types that have not yet been entered into the multimodal data type set, including: Check the behavioral assessment results of the identification data verified by the content; If the behavior evaluation result of the identification data is that the behavior verification is passed, then a type of identification data in the multimodal data type set that has not yet been input with identification data is deleted.

8. The multimodal recognition method applied to the Internet of Things Card Platform according to claim 1 is characterized in that: Behavioral assessment results include behavioral verification pass and behavioral verification fail; Based on the content verification results and behavior assessment results, delete the identification data types that have not yet been entered into the multimodal data type set, including: Count the number of identification data that have continuously passed content verification and whose behavior evaluation results are behavior verification passed; If the number exceeds a preset number threshold, all identification data types in the multimodal data type set for which identification data has not yet been input are deleted.

9. A multimodal recognition system applied to an Internet of Things card platform, characterized in that: include: A data preparation module is used to obtain a multimodal data type set corresponding to a target user, where the multimodal data type set includes multiple identification data types to be input and verified by the target user; A content verification module, used to obtain at least one identification data sequentially input by the target user based on the multimodal data type set, and perform content verification on the identification data to obtain a content verification result; A behavior evaluation module is used to evaluate the behavior of the target user when inputting identification data to obtain a behavior evaluation result; A credit assessment module, used to delete the identification data types for which corresponding identification data has not been input in the multimodal data type set according to the content verification result and the behavior assessment result; A verification and determination module, used to determine that the target user has passed the verification when the identification data corresponding to all identification data types in the multimodal data type set have passed the content verification; Among them, the behavior evaluation result includes behavior similarity, which represents the degree of similarity between the target user's behavior when inputting identification data this time and when he input identification data in the past; according to the content verification result and the behavior evaluation result, the identification data types that have not yet been input with identification data in the multimodal data type set are deleted, including: Calculate the sum of the behavior similarities of the identification data that has passed content verification to obtain a cumulative behavior credibility value; According to the cumulative value of the behavior credibility, the target deletion quantity is obtained; Based on the target deletion quantity, the identification data categories for which identification data has not yet been input in the multimodal data category set are deleted.

Images