Identity authentication identification method and device based on multi-modal information, equipment and medium

Abstract

The invention provides an identity authentication identification method and device based on multi-modal information, equipment and a medium, and relates to the technical field of computers and big data, in particular to the technical field of data security. The method comprises the following steps: acquiring authorization of a user for acquiring multi-modal information; acquiring equipment environment information, physiological information of the user and behavioral habit information after acquiring authorization of the user for acquiring the multi-modal information; performing weighted fusion processing on the equipment environment information, the physiological information of the user and the behavior habit information to obtain fusion features; generating target time sequence data based on the fusion features; calculating a deviation degree between the target time sequence data and the reference time sequence data; and performing identity authentication on the user based on the deviation degree to obtain an authentication result.

Description

Technical Field

[0001] This disclosure relates to the fields of computer and big data technology, specifically to the field of data security technology, and more specifically to an identity authentication and recognition method, apparatus, device, and medium based on multimodal information. Background Technology

[0002] With the advent of the big data era, traditional authentication technologies have significant limitations in emerging scenarios. This is mainly reflected in the following aspects: Existing biometric authentication technologies mostly use single biometric information, such as fingerprints, faces, or irises, for identity verification. However, single biometric authentication has many limitations: on the one hand, single biometric information is easily forged; for example, forgery methods such as fingerprint films and 3D-printed face models are constantly emerging, leading to reduced authentication security; on the other hand, single biometric information is greatly affected by environmental factors; for example, the accuracy of facial recognition drops significantly in low light or when the user is wearing a mask. Summary of the Invention

[0003] In view of the above problems, this disclosure provides an identity authentication and recognition method, apparatus, device and medium based on multimodal information, which is used to at least partially solve the above technical problems.

[0004] According to a first aspect of the present disclosure, an identity authentication method based on multimodal information is provided. The method includes: obtaining user authorization for obtaining multimodal information; after obtaining user authorization for obtaining multimodal information, obtaining device environment information, user physiological information, and behavioral habit information; performing weighted fusion processing on the device environment information, user physiological information, and behavioral habit information to obtain fusion features; generating target time-series data based on the fusion features; calculating the deviation between the target time-series data and the reference time-series data; and authenticating the user's identity based on the deviation to obtain an authentication result.

[0005] According to embodiments of this disclosure, weighted fusion processing is performed on device environment information, user physiological information, and behavioral habit information to obtain fusion features. This includes: mapping numerical data in the device environment information to a preset interval; extracting features from fingerprint and facial images in the physiological information; and curve fitting of the sliding trajectory in the behavioral habit information; mapping the processed device environment information, processed physiological information, and processed behavioral habit information to a unified feature space using a deep neural network model; assigning weights to the processed device environment information, processed physiological information, and processed behavioral habit information respectively; and fusing the processed device environment information, processed physiological information, and processed behavioral habit information based on their respective weights to obtain fusion features.

[0006] According to embodiments of this disclosure, user authentication is performed based on deviation to obtain authentication results, including: in response to deviation being less than a preset threshold, user authentication is performed based on fused features; in response to deviation being not less than the preset threshold, a warning message is issued and multi-factor enhanced verification is triggered without interrupting user operations.

[0007] According to embodiments of this disclosure, multi-factor enhanced verification includes at least one of the following: repeatedly collecting the user's physiological information for verification; verifying based on a dynamic verification code; and verifying based on a preset security question.

[0008] According to embodiments of this disclosure, obtaining device environment information, user physiological information, and behavioral habit information includes: collecting device environment information through the sensors and system interfaces of the terminal device; collecting user physiological information through the biometric module of the terminal device; and collecting behavioral data of the user during the use of the terminal device as behavioral habit information.

[0009] According to embodiments of this disclosure, device environment information includes at least one of the hardware characteristics, network characteristics, and environmental sensor data of the terminal device; physiological information includes at least one of fingerprint, face, iris, and voiceprint; and behavioral habit information includes at least one of key press frequency, swipe trajectory, application usage habits, input speed, and screen touch pressure.

[0010] According to embodiments of this disclosure, the method further includes: using a long short-term memory network to process historical device environment information, user's historical physiological information, and historical behavioral habit information within a historical time period to generate baseline time series data.

[0011] According to a second aspect of the present disclosure, an identity authentication device based on multimodal information is provided. The device includes: a first acquisition module for acquiring user authorization to acquire multimodal information; a second acquisition module for acquiring device environment information, user physiological information, and behavioral habit information after acquiring user authorization to acquire multimodal information; a fusion module for performing weighted fusion processing on the device environment information, user physiological information, and behavioral habit information to obtain fusion features; a generation module for generating target time-series data based on the fusion features; a calculation module for calculating the deviation between the target time-series data and the reference time-series data; and an authentication module for authenticating the user's identity based on the deviation to obtain an authentication result.

[0012] A third aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0013] A fourth aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0014] A fifth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0015] The identity authentication method based on multimodal information disclosed in this embodiment focuses on the cross-dimensional data fusion of device environment, physiological characteristics and behavioral habits. By constructing a composite authentication model of "device environment information + biological information + behavioral sequence", it realizes real-time dynamic verification of terminal environment risk identification and user identity, thereby improving the security, accuracy and adaptability of authentication. Attached Figure Description

[0016] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0017] Figure 1 The illustration shows an application scenario of an identity authentication method, apparatus, device, medium, and program product based on multimodal information according to embodiments of the present disclosure.

[0018] Figure 2 A flowchart illustrating an identity authentication method based on multimodal information according to an embodiment of the present disclosure is shown schematically.

[0019] Figure 3 A schematic diagram illustrating the structure of a multimodal information-based identity authentication device according to an embodiment of the present disclosure is shown.

[0020] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing a multimodal information-based identity authentication method according to an embodiment of the present disclosure. Detailed Implementation

[0021] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0024] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0025] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0026] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this disclosure all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0027] Figure 1 The illustration shows an application scenario of an identity authentication method, apparatus, device, medium, and program product based on multimodal information according to embodiments of the present disclosure.

[0028] like Figure 1As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, a server 105, and a database 106. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, the server 105, and the database 106. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0029] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 and the database 106 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, financial consumer applications, etc. (for example only).

[0030] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0031] Server 105 can be a server that provides various services, such as a backend management server that supports authentication requests sent by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103, and responds to the submitted authentication requests (this is just an example). The backend management server can process information such as device environment information, user physiological information, and behavioral habit information, and store the processing results in database 106 and (e.g., web pages, information, or data obtained or generated according to user requests) feed them back to the terminal devices.

[0032] It should be noted that the multimodal information-based identity authentication method provided in this disclosure embodiment can generally be executed by server 105. Correspondingly, the multimodal information-based identity authentication device provided in this disclosure embodiment can generally be located in server 105. The multimodal information-based identity authentication method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the multimodal information-based identity authentication device provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0033] It should be understood that Figure 1 The number of terminal devices, networks, servers, and databases shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, servers, and databases can be included.

[0034] The following will be based on Figure 1 The described scene, through Figure 2 The identity authentication method based on multimodal information of the disclosed embodiments is described in detail.

[0035] Figure 2 A flowchart illustrating an identity authentication method based on multimodal information according to an embodiment of the present disclosure is shown.

[0036] like Figure 2 As shown, the identity authentication based on multimodal information in this embodiment includes operations S210 to S260, and the identity authentication method based on multimodal information can be executed by server 105.

[0037] In operation S210, obtain user authorization for obtaining multimodal information.

[0038] When operating S220, after obtaining the user's authorization to acquire multimodal information, the device environment information, the user's physiological information, and behavioral habit information are acquired.

[0039] In operation S230, the device environment information, user physiological information, and behavioral habit information are weighted and fused to obtain fused features.

[0040] In operation S240, target time series data is generated based on fusion features.

[0041] In operation S250, the deviation between the target time series data and the reference time series data is calculated.

[0042] When operating S260, user authentication is performed based on deviation, and the authentication result is obtained.

[0043] Before obtaining a user's multimodal information, the user's consent or authorization can be obtained. For example, before operating S220, a request to obtain the user's multimodal information can be sent to the user. If the user consents or authorizes the acquisition of the user's multimodal information, operation S220 is executed.

[0044] In the embodiments of this application, a corresponding operation entry point can be provided to the user, allowing the user to choose to agree to or reject the automated decision result. That is, before performing identity authentication processing / decision on the user information, the user can provide an instruction to agree to or reject the processing / decision through the corresponding operation entry point. If the user agrees to the processing / decision, then identity authentication processing / decision is performed on the user information, i.e., step S260 is executed. If the user rejects the processing / decision, then the expert decision-making process is initiated.

[0045] In some embodiments, acquiring device environment information, user physiological information, and behavioral habit information includes: collecting device environment information through the sensors and system interface of the terminal device; collecting user physiological information through the biometric module of the terminal device; and collecting behavioral data of the user during the use of the terminal device as behavioral habit information.

[0046] Furthermore, device environment information includes at least one of the following: hardware characteristics of the terminal device, network characteristics, and environmental sensor data. Physiological information includes at least one of the following: fingerprint, face, iris, and voiceprint. Behavioral habit information includes at least one of the following: key press frequency, swipe trajectory, application usage habits, input speed, and screen touch pressure.

[0047] For example, device environment information includes: hardware information (such as CPU model, memory capacity, and hardware serial number) collected through the terminal device system interface; network information (such as IP address, MAC address, and network connection status) collected through the network module; and environmental sensor data (such as accelerometer, gyroscope, light sensor, and temperature sensor data) collected through built-in accelerometer, gyroscope, light sensor, and temperature sensor.

[0048] Physiological characteristics: The biometric module of the terminal device collects the user's physiological information, such as fingerprints, face, iris, voiceprint, etc.; for example, the user's fingerprint image is collected through an intelligent fingerprint sensor, and the user's facial image is collected through a front-facing camera.

[0049] Behavioral habit information: Collect user behavior data during the use of terminal devices, such as recording user key press frequency, swipe trajectory, application usage habits, input speed, screen touch pressure, etc. through system logs.

[0050] In some embodiments, weighted fusion processing is performed on device environment information, user physiological information, and behavioral habit information to obtain fused features. This includes: mapping numerical data in the device environment information to a preset interval; extracting features from fingerprint and facial images in the physiological information; and curve fitting of the sliding trajectory in the behavioral habit information; mapping the processed device environment information, processed physiological information, and processed behavioral habit information to a unified feature space using a deep neural network model; assigning weights to the processed device environment information, processed physiological information, and processed behavioral habit information respectively; and fusing the processed device environment information, processed physiological information, and processed behavioral habit information based on their respective weights to obtain fused features.

[0051] For example, the collected equipment environment information, physiological characteristics, and behavioral habit information are preprocessed, including data cleaning, standardization, and feature extraction, to remove noise and redundant information. For instance, numerical data (such as acceleration and temperature) in the equipment environment information are standardized and mapped to the [0, 1] interval. Fingerprint and facial images in the physiological characteristics are used for feature extraction to obtain fingerprint feature points and facial feature vectors; the sliding trajectory in the behavioral habit information is subjected to curve fitting to extract features such as the curvature and length of the trajectory.

[0052] Machine learning algorithms (such as deep neural networks and support vector machines) are used to map the three types of preprocessed information onto a unified feature space, forming a unified feature vector. For example, a deep neural network can be used to map the preprocessed device environment information, physiological information, and behavioral habit information onto a unified feature space, forming a unified feature vector with a length of 256 dimensions.

[0053] A dynamic weight allocation strategy is designed to adjust the weights of the three types of features in the authentication process in real time based on the security status of the terminal environment and the reliability of the features. For example, when an anomaly is detected in the device environment (such as a sudden and significant change in device location or abnormal network connection), the weight of physiological feature verification is automatically increased; when physiological feature collection is greatly affected by the environment (such as poor facial recognition due to dim lighting), the weights of device features and behavioral habit information are increased.

[0054] For example, initial weights are set as follows: device environment information, physiological characteristics, and behavioral habits information are weighted at 0.3, 0.5, and 0.2, respectively. When an unfamiliar Wi-Fi connection is detected (abnormal device environment), the weight of physiological characteristics is adjusted to 0.7, and the weights of device environment information and behavioral habits information are adjusted to 0.1 and 0.2, respectively. In strong light environments (which affect face capture), the weight of physiological characteristics is adjusted to 0.3, and the weights of device environment information and behavioral habits information are adjusted to 0.4 and 0.3, respectively.

[0055] In some embodiments, a Long Short-Term Memory (LSTM) network can be used to analyze time-series data of device environment information, physiological feature matching degree, and behavioral habit information to establish a time-series model under normal conditions. The changes in each feature are monitored in real time, the deviation of the current feature sequence from the normal time-series model is calculated, and abnormal patterns are identified. For example, by analyzing the time-varying sequence of device position changes, it can be determined whether there are positional changes outside the user's normal movement range; by analyzing the time-series changes in user keystroke frequency, it can be identified whether there are input patterns inconsistent with user habits.

[0056] For example, an LSTM network can be used to train a normal time series model on time-series data of device environment information, physiological feature matching degree, and behavioral habit information from the past 30 days. Current time-series data is collected in real time and input into the LSTM network to calculate the deviation.

[0057] In some embodiments, user authentication is performed based on deviation to obtain authentication results, including: in response to deviation being less than a preset threshold, user authentication is performed based on fused features; in response to deviation being not less than the preset threshold, a warning message is issued and multi-factor enhanced verification is triggered without interrupting user operations.

[0058] Furthermore, multi-factor enhanced verification includes at least one of the following: repeatedly collecting users' physiological information for verification; verifying based on dynamic verification codes; and verifying based on preset security questions.

[0059] For example, a hierarchical protection system of "basic verification - anomaly warning - enhanced verification" can be constructed:

[0060] Basic verification: When the terminal environment is normal and all features meet the normal mode, routine authentication is performed based on the fused unified feature vector. If the authentication is successful, the user is allowed to access the device.

[0061] Anomaly warning: When the LSTM network detects a slight anomaly (such as feature deviation within the warning threshold range), it issues an anomaly warning without interrupting user operation, while strengthening the monitoring of subsequent features.

[0062] Enhanced Authentication: When a serious anomaly is detected (such as a feature deviation exceeding the warning threshold), multi-factor enhanced authentication is triggered, requiring the user to provide additional authentication information, such as secondary biometric information (fingerprint + iris), dynamic verification code, and answers to preset security questions. Only after passing the enhanced authentication can the user continue operating.

[0063] For example, a deviation warning threshold of 0.6 can be set. When the deviation is between 0.4 and 0.6, an anomaly warning is issued; when the deviation exceeds 0.6, it is judged as a serious anomaly. For example, when a user is using the device normally, they are usually in the office from 9 am to 6 pm (a relatively fixed location). If at some point the user's location is detected to suddenly appear in another location (not a location the user usually visits), and the deviation of this location change sequence from the normal model is 0.7, then it is judged as a serious anomaly.

[0064] According to embodiments of this disclosure, by integrating "device environment information" and "user behavior specificity," by focusing on cross-dimensional data fusion of device environment, physiological characteristics, and behavioral habits, and by constructing a composite authentication model of "device environment information + biometric information + behavioral sequence," the system achieves real-time dynamic verification of terminal environment risk identification and user identity, thereby improving the security and reliability of device login. This model can be widely applied to login authentication scenarios for various enterprises and organizations.

[0065] Based on the aforementioned identity authentication method based on multimodal information, this disclosure also provides an identity authentication device based on multimodal information. The following will be combined with... Figure 3 The device is described in detail.

[0066] Figure 3 A schematic block diagram of a multimodal information-based identity authentication device according to an embodiment of the present disclosure is shown.

[0067] like Figure 3 As shown, the identity authentication device 300 based on multimodal information in this embodiment includes a first acquisition module 310, a second acquisition module 320, a fusion module 330, a generation module 340, a calculation module 350, and an authentication module 360.

[0068] The first acquisition module 310 is used to obtain user authorization for acquiring multimodal information. In one embodiment, the first acquisition module 310 can be used to perform the operation S210 described above, which will not be repeated here. The second acquisition module 320 is used to acquire device environment information, user physiological information, and behavioral habit information after obtaining user authorization for acquiring multimodal information. In one embodiment, the second acquisition module 320 can be used to perform the operation S220 described above, which will not be repeated here.

[0069] The fusion module 330 is used to perform weighted fusion processing on device environmental information, user physiological information, and behavioral habit information to obtain fused features. In one embodiment, the fusion module 330 can be used to perform the operation S230 described above, which will not be repeated here.

[0070] The generation module 340 is used to generate target time-series data based on fusion features. In one embodiment, the generation module 340 can be used to perform the operation S240 described above, which will not be repeated here.

[0071] The calculation module 350 is used to calculate the deviation between the target time series data and the reference time series data. In one embodiment, the calculation module 350 can be used to perform the operation S250 described above, which will not be repeated here.

[0072] The authentication module 360 ​​is used to authenticate the user's identity based on the deviation degree and obtain the authentication result. In one embodiment, the authentication module 360 ​​can be used to perform the operation S260 described above, which will not be repeated here.

[0073] According to embodiments of this disclosure, weighted fusion processing is performed on device environment information, user physiological information, and behavioral habit information to obtain fusion features. This includes: mapping numerical data in the device environment information to a preset interval; extracting features from fingerprint and facial images in the physiological information; and curve fitting of the sliding trajectory in the behavioral habit information; mapping the processed device environment information, processed physiological information, and processed behavioral habit information to a unified feature space using a deep neural network model; assigning weights to the processed device environment information, processed physiological information, and processed behavioral habit information respectively; and fusing the processed device environment information, processed physiological information, and processed behavioral habit information based on their respective weights to obtain fusion features.

[0074] According to embodiments of this disclosure, user authentication is performed based on deviation to obtain authentication results, including: in response to deviation being less than a preset threshold, user authentication is performed based on fused features; in response to deviation being not less than the preset threshold, a warning message is issued and multi-factor enhanced verification is triggered without interrupting user operations.

[0075] According to embodiments of this disclosure, multi-factor enhanced verification includes at least one of the following: repeatedly collecting the user's physiological information for verification; verifying based on a dynamic verification code; and verifying based on a preset security question.

[0076] According to embodiments of this disclosure, obtaining device environment information, user physiological information, and behavioral habit information includes: collecting device environment information through the sensors and system interfaces of the terminal device; collecting user physiological information through the biometric module of the terminal device; and collecting behavioral data of the user during the use of the terminal device as behavioral habit information.

[0077] According to embodiments of this disclosure, device environment information includes at least one of the hardware characteristics, network characteristics, and environmental sensor data of the terminal device; physiological information includes at least one of fingerprint, face, iris, and voiceprint; and behavioral habit information includes at least one of key press frequency, swipe trajectory, application usage habits, input speed, and screen touch pressure.

[0078] According to embodiments of this disclosure, the device further includes: a reference time series data generation module, used to process historical device environment information, user's historical physiological information and historical behavioral habit information within a historical time period using a long short-term memory network to generate reference time series data.

[0079] According to embodiments of this disclosure, any plurality of modules among the first acquisition module 310, the second acquisition module 320, the fusion module 330, the generation module 340, the calculation module 350, and the authentication module 360 ​​can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the first acquisition module 310, the second acquisition module 320, the fusion module 330, the generation module 340, the calculation module 350, and the authentication module 360 ​​can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the first acquisition module 310, the second acquisition module 320, the fusion module 330, the generation module 340, the calculation module 350, and the authentication module 360 ​​may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0080] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing an object method according to an embodiment of the present disclosure.

[0081] like Figure 4As shown, an electronic device 400 according to an embodiment of the present disclosure includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0082] RAM 403 stores various programs and data required for the operation of electronic device 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Processor 401 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 402 and / or RAM 403. It should be noted that the programs may also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0083] According to embodiments of this disclosure, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to a bus 404. The electronic device 400 may also include one or more of the following components connected to the input / output (I / O) interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0084] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0085] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 402 and / or RAM 403 and / or one or more memories other than ROM 402 and RAM 403 described above.

[0086] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the multimodal information-based identity authentication method provided in the embodiments of this disclosure.

[0087] When the computer program is executed by the processor 401, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0088] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via communication section 409, and / or installed from removable medium 411. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0089] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0090] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0092] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0093] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An identity authentication method based on multimodal information, characterized in that, The method includes: Obtain user authorization for accessing multimodal information; After obtaining the user's authorization to acquire multimodal information, device environmental information, user physiological information, and behavioral habit information are acquired. The device environment information, the user's physiological information, and the behavioral habit information are weighted and fused to obtain fused features; Target time-series data is generated based on the fusion features; Calculate the deviation between the target time series data and the reference time series data; The user's identity is authenticated based on the deviation, and an authentication result is obtained.

2. The method according to claim 1, characterized in that, The weighted fusion processing of the device environment information, the user's physiological information, and the behavioral habit information to obtain fused features includes: The numerical data in the device environment information is mapped to a preset range, the fingerprint and facial images in the physiological information are extracted for features, and the sliding trajectory in the behavioral habit information is curve fitted. The processed device environment information, processed physiological information, and processed behavioral habit information are mapped to a unified feature space using a deep neural network model. Weights are assigned to the processed equipment environment information, processed physiological information, and processed behavioral habit information, respectively; Based on their respective weights, the processed device environment information, processed physiological information, and processed behavioral habit information are fused to obtain fused features.

3. The method according to claim 1, characterized in that, The process of authenticating the user's identity based on the deviation, and obtaining the authentication result, includes: In response to the deviation being less than a preset threshold, the user is authenticated based on the fusion features; In response to the deviation being no less than a preset threshold, a warning message is issued and multi-factor enhanced verification is triggered without interrupting user operation.

4. The method according to claim 3, characterized in that, The multi-factor enhanced validation includes at least one of the following: Repeatedly collect the user's physiological information for verification; Verification is performed using dynamic CAPTCHA. Verification is performed based on preset security questions.

5. The method according to claim 1, characterized in that, The acquisition of device environment information, user physiological information, and behavioral habit information includes: The device's environmental information is collected through the terminal device's sensors and system interface; The user's physiological information is collected through the biometric module of the terminal device; Collect user behavior data during the use of the terminal device as the behavior habit information.

6. The method according to claim 1 or 5, characterized in that, The device environment information includes at least one of the terminal device's hardware characteristics, network characteristics, and environmental sensor data; The physiological information includes at least one of fingerprints, face, iris, and voiceprint; The behavioral habit information includes at least one of the following: key press frequency, swipe trajectory, application usage habits, input speed, and screen touch pressure.

7. The method according to claim 1, characterized in that, The method further includes: The baseline time series data is generated by processing historical device environment information, user historical physiological information, and historical behavioral habit information within a historical time period using a long short-term memory network.

8. An identity authentication device based on multimodal information, characterized in that, The device includes: The first acquisition module is used to obtain the user's authorization to acquire multimodal information; The second acquisition module is used to acquire device environment information, user physiological information, and behavioral habit information after obtaining the user's authorization to acquire multimodal information; The fusion module is used to perform weighted fusion processing on the device environment information, the user's physiological information, and the behavioral habit information to obtain fused features; The generation module is used to generate target time-series data based on the fusion features; The calculation module is used to calculate the deviation between the target time series data and the reference time series data; The authentication module is used to authenticate the user's identity based on the deviation and obtain the authentication result.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images