Multi-modal fusion online account opening method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-modal fusion online account opening method and device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, and is suitable for the field of financial science and technology. The method comprises the steps that an account opening request sent by a client is responded, a connection success notification message is sent to the client to start a camera and a microphone, the camera is used for collecting a target face, and the microphone is used for collecting target voice; obtaining a target face sent by the client, and performing micro-expression recognition on the target face to obtain a target micro-expression feature; obtaining a target voice sent by the client, and performing voiceprint recognition on the target voice to obtain a target voiceprint feature; carrying out attention fusion on the target micro-expression feature and the target voiceprint feature to obtain a target fusion feature; performing identity verification on the target object according to the target fusion feature to obtain a legal object; and generating a digital signature for the legal object, and opening an account for the legal object based on the digital signature. According to the invention, the accuracy and reliability of online account opening can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology and is applicable to the field of financial technology, and in particular to a multimodal fusion online account opening method and device, electronic device, and storage medium. Background Art

[0002] Online account opening (also known as remote account opening) involves a user using a client to request a server to assign them an account. In the fintech sector, users can easily register new accounts (such as personal or corporate bank accounts) through a mobile app or web platform, eliminating the need to visit a bank in person. During the registration process, the server authenticates the user's face and voice, and only after successful authentication is the new account assigned.

[0003] In the identity verification process of existing online account opening methods, it is difficult to effectively perform multimodal fusion of face and voice because face and voice are data of different modalities. This makes identity verification prone to errors and may result in account opening for users with forged faces or voices, with low accuracy and reliability.

[0004] Therefore, the relevant technology has the problem of low accuracy and reliability of online account opening. Summary of the Invention

[0005] The main purpose of the embodiments of the present application is to propose a multimodal fusion online account opening method and device, electronic device, and storage medium, which can improve the accuracy and reliability of online account opening.

[0006] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a multimodal fusion online account opening method, which is applied to an account opening server. The method includes:

[0007] In response to the account opening request sent by the client, a connection success notification message is sent to the client, wherein the connection success notification message is used to turn on a camera and a microphone, wherein the camera is used to capture a target face of a target object, and the microphone is used to capture a target voice of the target object;

[0008] Acquire the target face sent by the client, and perform micro-expression recognition on the target face to obtain target micro-expression features;

[0009] Acquire the target voice sent by the client, and perform voiceprint recognition on the target voice to obtain target voiceprint features;

[0010] Performing attention fusion on the target micro-expression feature and the target voiceprint feature to obtain a target fusion feature;

[0011] Authenticate the target object based on the target fusion feature to obtain a legitimate object;

[0012] A digital signature is generated for the legal object, and an account is opened for the legal object based on the digital signature.

[0013] Optionally, performing attention fusion on the target micro-expression feature and the target voiceprint feature to obtain a target fusion feature includes:

[0014] Perform timestamp matching based on the face collection timestamp corresponding to the target micro-expression feature and the voice collection timestamp corresponding to the target voiceprint feature to obtain a matching timestamp;

[0015] Attention fusion is performed on the target micro-expression feature and the target voiceprint feature that belong to the matching timestamp to obtain the target fusion feature.

[0016] Optionally, performing attention fusion on the target micro-expression feature and the target voiceprint feature that both belong to the matching timestamp to obtain the target fusion feature includes:

[0017] Multiplying a preset query weight matrix by the target voiceprint feature belonging to the matching timestamp to obtain a voiceprint query vector;

[0018] Multiplying a preset key weight matrix by the target micro-expression feature belonging to the matching timestamp to obtain a micro-expression key vector;

[0019] Multiplying a preset key weight matrix by the target micro-expression feature belonging to the matching timestamp to obtain a micro-expression value vector;

[0020] Vector fusion is performed according to the voiceprint query vector, the micro-expression key vector, and the micro-expression value vector to obtain the target fusion feature.

[0021] Optionally, performing vector fusion according to the voiceprint query vector, the micro-expression key vector, and the micro-expression value vector to obtain the target fusion feature includes:

[0022] Performing weight calculation on the voiceprint query vector and the micro-expression key vector to obtain a voiceprint micro-expression interaction attention score;

[0023] Performing nonlinear activation processing on the voiceprint microexpression interaction attention score to obtain a voiceprint microexpression interaction weight vector;

[0024] A weighted sum is performed on the voiceprint micro-expression interaction weight vector and the micro-expression value vector to obtain the target fusion feature.

[0025] Optionally, performing micro-expression recognition on the target face to obtain target micro-expression features includes:

[0026] Performing eyebrow detection on the target face to obtain an upward displacement feature of the eyebrow;

[0027] Performing mouth corner detection on the target face to obtain mouth corner drooping rate features;

[0028] Performing blink detection on the target face to obtain blink interval time features;

[0029] The target micro-expression feature is obtained by performing feature fusion based on the eyebrow center upward displacement feature, the mouth corner drooping rate feature and the blink interval time feature.

[0030] Optionally, the authenticating the target object based on the target fusion feature to obtain a legitimate object includes: authenticating the target object based on the target fusion feature using a pre-trained deep fake attack defense model to obtain a legitimate object;

[0031] The method further includes pre-training the deep fake attack defense model, specifically comprising:

[0032] Constructing a training data set, the training data set including normal samples and adversarial samples, wherein the normal samples are samples in which both the sample face and the sample voice of the sample subject are non-forged, and the adversarial samples are samples in which at least one of the sample face and the sample voice of the sample subject is non-forged, and the number of the normal samples is greater than the number of the adversarial samples;

[0033] Authentication is performed on the normal sample using a preset initial depth model to obtain a first sample object verification result, and loss calculation is performed based on the first sample object detection result to obtain first loss data;

[0034] Authentication is performed based on the adversarial sample using a preset initial deep model to obtain a second sample object verification result, and loss calculation is performed based on the second sample object verification result to obtain second loss data;

[0035] generating disturbance-resistant constraint loss data according to the second loss data;

[0036] Performing loss fusion according to the first loss data, the second loss data, and the anti-disturbance constraint loss data to obtain target loss data;

[0037] The parameters of the initial deep model are adjusted using the target loss data to obtain the deep fake attack defense model.

[0038] Optionally, the adversarial sample includes a first adversarial sample and a second adversarial sample, the first adversarial sample is a sample in which either a sample face or a sample voice is not forged, the second adversarial sample is a sample in which both a sample face and a sample voice are not forged, and the second loss data includes first adversarial loss data calculated based on the first adversarial sample and second adversarial loss data calculated based on the second adversarial sample;

[0039] Generating the anti-disturbance constraint loss data according to the second loss data includes:

[0040] Setting a first adversarial weight for the first adversarial example and a second adversarial weight for the second adversarial example, wherein the first adversarial weight is greater than the second adversarial weight;

[0041] Multiplying the first adversarial weight and the first adversarial loss data to obtain first adversarial disturbance constraint loss data;

[0042] Multiplying the second adversarial weight and the second adversarial loss data to obtain second adversarial disturbance constraint loss data;

[0043] The anti-disturbance constraint loss data is obtained by adding the first anti-disturbance constraint loss data and the second anti-disturbance constraint loss data.

[0044] To achieve the above objectives, a second aspect of the embodiments of the present application provides a multimodal fusion online account opening device, which is applied to an account opening server. The device includes:

[0045] a request response module, configured to respond to an account opening request sent by a client and send a connection success notification message to the client, wherein the connection success notification message is used to turn on a camera and a microphone, wherein the camera is used to capture a target face of a target object, and the microphone is used to capture a target voice of the target object;

[0046] A micro-expression recognition module is used to obtain the target face sent by the client, and perform micro-expression recognition on the target face to obtain target micro-expression features;

[0047] a voiceprint recognition module, configured to obtain the target voice sent by the client, and perform voiceprint recognition on the target voice to obtain target voiceprint features;

[0048] A feature fusion module, configured to perform attention fusion on the target micro-expression feature and the target voiceprint feature to obtain a target fusion feature;

[0049] An identity authentication module, configured to authenticate the target object based on the target fusion feature to obtain a legitimate object;

[0050] The online account opening module is used to generate a digital signature for the legal object and open an account for the legal object based on the digital signature.

[0051] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory storing a computer program, and the processor implementing the multimodal fusion online account opening method described in the first aspect when executing the computer program.

[0052] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, it implements the multimodal fusion online account opening method described in the first aspect above.

[0053] The multimodal fusion online account opening method and device, electronic device, and storage medium proposed in this application do not require the client to first collect face and voice data and then directly send the online account opening information containing the face and voice data to the account opening server during the overall online account opening process. This would present certain difficulties for the account opening server in fusing the face and voice data. Instead, after receiving the account opening request from the client, this application first sends a connection success notification message to the client. The client then activates its camera and microphone in response to the connection success notification message. This allows for nearly simultaneous face and voice data collection, reducing the difficulty for the account opening server to subsequently fuse the received target face and target audio data. Furthermore, the account opening server performs micro-expression recognition on the target face to obtain target micro-expression features, and voiceprint recognition on the target voice to obtain target voiceprint features. The target micro-expression features and target voiceprint features are then subjected to attention fusion to obtain target fusion features. This target fusion feature effectively represents the fusion information of the target subject's face and voice data within similar acquisition timestamps. Furthermore, the target subject is authenticated based on the target fusion feature to verify the subject's legitimacy, improving the accuracy and reliability of authentication. Finally, a digital signature is generated for the legal object, and an account is opened for the legal object based on the digital signature. In summary, this application can improve the accuracy and reliability of online account opening.

[0054] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flowchart of the multimodal fusion online account opening method provided in an embodiment of the present application;

[0056] Figure 2yes Figure 1 Flowchart of step 104 in FIG.

[0057] Figure 3 yes Figure 2 Flowchart of step 202 in FIG.

[0058] Figure 4 yes Figure 3 Flowchart of step 304 in FIG.

[0059] Figure 5 This is a flowchart of a multimodal fusion online account opening method provided by another embodiment of the present application;

[0060] Figure 6 yes Figure 5 Flowchart of step 504 in FIG.

[0061] Figure 7 This is a block diagram of the module structure of the multimodal fusion online account opening device provided in an embodiment of the present application;

[0062] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0064] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0066] First, let’s analyze some of the terms used in this application:

[0067] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0068] Natural language processing (NLP): NLP uses computers to process, understand, and apply human languages (such as Chinese and English). A branch of artificial intelligence, NLP is an interdisciplinary field between computer science and linguistics, often referred to as computational linguistics. Natural language processing encompasses grammatical analysis, semantic analysis, and discourse comprehension. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information and image processing, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It encompasses data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and linguistics research related to language computing.

[0069] Traditional multimodal fusion online account opening methods include: (1) dual-modal authentication methods based on face + voiceprint, but face recognition relies on coarse-grained classification (such as "happy" and "angry"), and cannot detect micro-expressions as short as 0.2 seconds (such as fear, signs of deception), and cannot identify coercion or fraud scenes. (2) Dynamic face recognition technology is used, but the face alone is easy to be forged (such as 3D printed masks, AI voice synthesis), especially the defense capability against AI synthetic video attacks (such as Deepfake) is insufficient, and the financial anti-fraud defense is poor. (3) Technology that optimizes the voiceprint collection process based on face recognition often uses serial verification (face first, then voiceprint), resulting in a fragmented user experience and the inability to capture cross-modal correlation features (such as the trembling of the corners of the mouth when speaking nervously). In other words, multimodal data is not synchronized in real time, and the response delay is as high as 1.2 seconds.

[0070] Based on this, the embodiments of the present application propose a multimodal fusion online account opening method, a multimodal fusion online account opening device, an electronic device and a computer-readable storage medium, which can improve the accuracy and reliability of online account opening.

[0071] The multimodal fusion online account opening method provided in the embodiments of the present application can be applied to terminals and servers, and can also be software running on the server. The server can be configured as an independent physical server, or as a server cluster or distributed system consisting of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the multimodal fusion online account opening method, etc., but is not limited to the above forms.

[0072] The present application can be used in many general or special computer system environments or configurations. For example: server computers, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0073] The embodiments of the present application provide a multimodal fusion online account opening method, a multimodal fusion online account opening device, an electronic device, and a computer-readable storage medium, which are specifically illustrated by the following embodiments. First, the multimodal fusion online account opening method in the embodiments of the present application is described.

[0074] It should be noted that in each specific embodiment of the present application, when it comes to processing data related to the user's identity or characteristics, such as the user's image data, the user's permission or consent will be obtained first, and the collection, use and processing of this data will comply with relevant laws, regulations and standards.

[0075] Reference Figure 1 , Figure 1 This is an optional flowchart of the multimodal fusion online account opening method provided in an embodiment of the present application. The method can be applied to an account opening server, and the method may include but is not limited to steps 101 to 106.

[0076] Step 101: In response to an account opening request sent by a client, a connection success notification message is sent to the client. The connection success notification message is used to turn on a camera and a microphone. The camera is used to capture a target face of a target object, and the microphone is used to capture a target voice of the target object.

[0077] Step 102: Obtain the target face sent by the client, and perform micro-expression recognition on the target face to obtain target micro-expression features;

[0078] Step 103: Acquire the target voice sent by the client, perform voiceprint recognition on the target voice, and obtain the target voiceprint feature;

[0079] Step 104: performing attention fusion on the target micro-expression feature and the target voiceprint feature to obtain a target fusion feature;

[0080] Step 105: Authenticate the target object based on the target fusion feature to obtain a legitimate object;

[0081] Step 106: Generate a digital signature for the legal object and open an account for the legal object based on the digital signature.

[0082] In steps 101 to 106 shown in the embodiment of the present application, after receiving the account opening request sent by the client, a connection success notification message is first sent to the client. The client will turn on the camera and microphone based on the connection success notification message. In this way, face collection and voice collection are almost synchronous, reducing the difficulty of the account opening server in subsequently fusing the received target face and target audio. Furthermore, the account opening server performs micro-expression recognition on the target face to obtain target micro-expression features, and performs voiceprint recognition on the target voice to obtain target voiceprint features; then, attention fusion is performed on the target micro-expression features and the target voiceprint features to obtain target fusion features; in this way, the target fusion features can effectively represent the fusion information of the target object's face and voice within almost similar collection timestamps. Furthermore, the target object is authenticated based on the target fusion features to obtain a legitimate object, thereby improving the accuracy and reliability of identity authentication. Finally, a digital signature is generated for the legitimate object, and an account is opened for the legitimate object based on the digital signature. In summary, the present application can improve the accuracy and reliability of online account opening.

[0083] For example, in a fintech bank account opening scenario, a user might wish to register for a personal bank account with Bank A. In this case, the user can use a mobile app or web platform to send their facial and voice ID to Bank A's account opening server. The server then authenticates the user based on their facial and voice ID and, upon successful authentication, allocates the user a personal bank account. If the authentication fails, the account opening is denied.

[0084] In another example, in the fintech insurance account opening scenario, a user might wish to register for an insurance account with Insurance Company B. In this case, the user can use a mobile app or web platform to send their facial and voice ID to Insurance Company B's account opening server. The server then authenticates the user based on their facial and voice ID and, upon successful authentication, allocates the user an insurance account. If the authentication fails, the account opening is denied.

[0085] In step 101 of some embodiments, in response to an account opening request sent by the client, a connection success notification message is sent to the client. An account opening request is a request from a target object to register a new account. In conjunction with the above, the account opening request can be a bank account opening request, an insurance account opening request, or an account opening request in other scenarios, which is not specifically limited in this application. The account opening request generally includes information such as the object identifier of the target object, the identifier of the account opening party (such as Bank A or Insurance Company B), and the account opening type (such as a personal account or a corporate account).

[0086] The account opening server is a server that provides account opening services. For example, the account opening server may be a server that provides bank account opening services, or a server that provides insurance account opening services, or a server that provides both bank account opening services and insurance account opening services.

[0087] After receiving the account opening request, the account opening server can establish a communication connection with the client, generate a successful connection notification message, and then send the successful connection notification message to the client. The successful connection notification message is used to turn on the camera and microphone. The camera is used to capture the target person's face, and the microphone is used to capture the target person's voice.

[0088] It should be noted that the client turns on the camera and microphone almost at the same time, and the camera's acquisition timestamp is almost aligned with the microphone's acquisition timestamp, for example, the error is <10 milliseconds, achieving face and voice synchronization.

[0089] In step 102 of some embodiments, a target face sent by the client is obtained, and micro-expression recognition is performed on the target face to obtain target micro-expression features. Micro-expression recognition can use an optical flow method.

[0090] In one embodiment, step 102 may include: performing eyebrow detection on the target face to obtain an eyebrow upward displacement feature; performing mouth corner detection on the target face to obtain a mouth corner drooping rate feature; performing blink detection on the target face to obtain a blink interval time feature; performing feature fusion based on the eyebrow upward displacement feature, the mouth corner drooping rate feature, and the blink interval time feature to obtain a target micro-expression feature.

[0091] In one embodiment, performing eyebrow center detection on a target face to obtain an upward displacement feature of the eyebrow center includes: using a face detection algorithm to locate the face in the image; extracting facial key points, particularly feature points around the eyebrows, using a facial landmark extractor; calculating the average of the left and right eyebrow feature points to obtain the eyebrow center position coordinates; and comparing the eyebrow center position coordinates in two frames of images to calculate their vertical displacement. If the y-value of the eyebrow center coordinate significantly decreases (i.e., moves upward on the y-axis) in images at different time points, it can be determined that the eyebrow center is upwardly raised.

[0092] In one embodiment, the mouth corners of the target face are detected to obtain the mouth corner droop rate characteristics, including: using a face detection algorithm to locate the position of the face in the image; extracting facial key points, especially feature points around the mouth corners, through a facial landmark point extractor; obtaining the feature point coordinates of the left and right mouth corners. Usually, the left mouth corner corresponds to the 48th feature point, and the right mouth corner corresponds to the 54th feature point; compare the mouth corner coordinates in the two frames of images and calculate their changes on the y-axis. If the y value of the mouth corner coordinate increases significantly (i.e., moves downward) in consecutive images, it can be inferred that the mouth corner is drooping; the detected drooping displacement is combined with the time interval to obtain the mouth corner drooping rate. For example, the frame rate is used to calculate the drooping change per unit time.

[0093] In one embodiment, blink detection is performed on a target face to obtain blink interval characteristics, including: using a face detection algorithm to locate the face in the image; extracting facial key points, particularly feature points around the eyes, using a facial landmark extractor; determining whether the eyes are closed by analyzing the positions of the eye feature points, for example, by determining whether a blink is present based on the ratio of the vertical height to the horizontal length of the eye (the distance between the upper and lower eyebrow points and the upper and lower eye points); and recording a timestamp when the eyes are detected to transition from open to closed and back to open. This can be continuously monitored using a sliding window approach; if eye closure is detected within a certain time window, it can be determined as a blink.

[0094] In one example, the target micro-expression feature can be expressed as f micro =[Δx eyebrow ,Δy lip ,τ blink ]∈R 12 , where Δx eyebrow Indicates the upward displacement characteristic of the eyebrow center, Δy 1ip represents the mouth corner drooping rate feature, τ blink Indicates the inter-blink time feature.

[0095] In step 103 of some embodiments, the target voice sent by the client is obtained, and voiceprint recognition is performed on the target voice to obtain the target voiceprint feature. Voiceprint recognition can be achieved through a pre-trained voiceprint recognition model. The target voiceprint feature can be expressed as f voice ∈R 40 .

[0096] In step 104 of some embodiments, attention fusion is performed on the target micro-expression feature and the target voiceprint feature to obtain a target fusion feature. Attention fusion can be achieved through an interactive attention mechanism.

[0097] In one embodiment, referring to Figure 2 , step 104 may include:

[0098] Step 201, performing timestamp matching based on the face collection timestamp corresponding to the target micro-expression feature and the voice collection timestamp corresponding to the target voiceprint feature to obtain a matching timestamp;

[0099] Step 202 : Attention fusion is performed on the target micro-expression feature and the target voiceprint feature that belong to the matching timestamp to obtain a target fusion feature.

[0100] In step 201, the matching timestamp refers to the timestamp when the face collection timestamp and the voice collection timestamp are the same. For example, if the face collection timestamp and the voice collection timestamp are the same, either the face collection timestamp or the voice collection timestamp is used as the matching timestamp.

[0101] The benefit of this embodiment is that before attention fusion, timestamp matching is performed first, which can achieve synchronous fusion of multiple modalities, capture cross-modal correlation features (such as trembling of the corners of the mouth when the voice is nervous), improve fusion accuracy, and thus improve identity authentication accuracy.

[0102] In one embodiment, referring to Figure 3 , step 202 may include:

[0103] Step 301: Multiply the preset query weight matrix by the target voiceprint feature belonging to the matching timestamp to obtain a voiceprint query vector;

[0104] Step 302: multiplying a preset key weight matrix by the target micro-expression feature belonging to the matching timestamp to obtain a micro-expression key vector;

[0105] Step 303: multiplying the preset key weight matrix by the target micro-expression feature belonging to the matching timestamp to obtain a micro-expression value vector;

[0106] Step 304 : performing vector fusion based on the voiceprint query vector, the micro-expression key vector, and the micro-expression value vector to obtain a target fusion feature.

[0107] In step 301, the voiceprint query vector is as follows:

[0108] Q=f voice W q , where Q is the voiceprint query vector, f voice Refers to the target voiceprint feature; W Q is the trainable query weight matrix.

[0109] In step 302, the text layout key vector is as follows:

[0110] K=f micro W K , where K refers to the key vector of micro-expression, f micro refers to the target micro-expression characteristics; W K refers to the trainable key weight matrix.

[0111] In step 303, the text layout key vector is as follows:

[0112] V=f micro W k , where V refers to the micro-expression value vector, f micro refers to the target micro-expression characteristics; W V refers to the trainable key weight matrix.

[0113] The benefit of the embodiment of the above steps 301 to 304 is that by setting the key weight matrix to calculate the micro-expression key vector and the micro-expression value vector respectively, and then performing vector fusion with the voiceprint query vector, this setting method greatly improves the accuracy of attention fusion, that is, improves the accuracy of feature interaction.

[0114] In one embodiment, referring to Figure 4 , step 304 may include:

[0115] Step 401: weight the voiceprint query vector and the micro-expression key vector to obtain a voiceprint micro-expression interaction attention score;

[0116] Step 402: Perform nonlinear activation processing on the voiceprint micro-expression interaction attention score to obtain a voiceprint micro-expression interaction weight vector;

[0117] Step 403: Perform weighted summation on the voiceprint micro-expression interaction weight vector and the micro-expression value vector to obtain the target fusion feature.

[0118] Specifically, the target fusion features can be shown as follows:

[0119] Here, d refers to the attention dimension and can be set to 64.

[0120] The benefit of the embodiment of steps 401 to 403 is that the accuracy of vector fusion is improved.

[0121] In step 105 of some embodiments, the target object is authenticated based on the target fusion features to obtain a legitimate object. A pre-trained deepfake attack defense model can be used to authenticate the target object based on the target fusion features to obtain a legitimate object. If either the target face or the target voice of the target object is forged, the authentication fails and the target object is determined to be illegitimate.

[0122] It should be noted that in this application, even if the target face and target voice of the target object are truly collected and not forged, if the target object's voice is nervous or the corners of the mouth tremble, the identity authentication cannot pass and the target object will be judged as an illegal object.

[0123] In one embodiment, before step 106, the multimodal fusion online registration method may further include: performing physiological signal consistency detection: capturing facial blood oxygen changes (rPPG signals) of the target object through a camera to verify the physiological characteristics of the living body; if the liveness verification fails, determining that the target object is an illegal object.

[0124] In some embodiments, in step 106, a digital signature is generated for the legal subject, and an account is opened for the legal subject based on the digital signature. For example, in a bank account opening scenario, a bank account digital signature is generated for the legal user, and a bank account is allocated to the legal user based on the bank account digital signature. For another example, in an insurance account opening scenario, an insurance account digital signature is generated for the legal user, and an insurance account is allocated to the legal user based on the insurance account digital signature.

[0125] In one embodiment, referring to Figure 5 Before step 105, the multimodal fusion online account opening method may further include:

[0126] Step 501: construct a training data set, which includes normal samples and adversarial samples;

[0127] Step 502: Authenticate the normal sample using a preset initial depth model to obtain a first sample object verification result, and perform loss calculation based on the first sample object detection result to obtain first loss data;

[0128] Step 503: Authenticate the adversarial sample using a preset initial deep model to obtain a second sample object verification result, and calculate the loss based on the second sample object verification result to obtain second loss data;

[0129] Step 504: generating anti-disturbance constraint loss data based on the second loss data;

[0130] Step 505: performing loss fusion based on the first loss data, the second loss data, and the anti-disturbance constraint loss data to obtain target loss data;

[0131] Step 506: Adjust the parameters of the initial deep model using the target loss data to obtain a deep fake attack defense model.

[0132] In step 501, a normal sample is a sample face and a sample voice of a sample subject, both of which are genuine. An adversarial sample is a sample face and a sample voice of a sample subject, at least one of which is genuine. The number of normal samples is greater than the number of adversarial samples. For example, the ratio of the number of normal samples to the number of adversarial samples is 4 to 1.

[0133] A normal sample has a first label value (e.g., 1) or a second label value (e.g., 0.1). The first label value indicates that the sample object can pass identity authentication. The second label value indicates that the sample object cannot pass identity authentication.

[0134] The adversarial sample has a third label value (e.g., 0). The third label value indicates that the sample object cannot pass identity authentication. The first label value, the second label value, and the third label value decrease in sequence.

[0135] In step 502, a normal sample is authenticated using a preset initial deep model to obtain a first sample object authentication result. The first sample object authentication result indicates whether the sample object has passed or failed authentication. Loss calculation based on the first sample object detection result specifically calculates a cross entropy loss (or mean square error loss, etc.) based on the first sample object detection result and the first label value or the second label value to obtain first loss data.

[0136] In step 503, the adversarial sample is authenticated using a preset initial deep model to obtain a second sample object verification result. The second sample object verification result indicates whether the sample object has passed or failed authentication. Loss calculation based on the second sample object detection result specifically calculates the cross entropy loss (or mean square error loss, etc.) based on the second sample object detection result and the third label value to obtain second loss data. The initial deep model can be a convolutional neural network or other model.

[0137] At step 504, adversarial perturbation constraint loss data is generated based on the second loss data. The adversarial examples include a first adversarial example and a second adversarial example. The first adversarial example is a sample in which either a face or voice sample is genuine, and the second adversarial example is a sample in which both a face and voice sample are genuine. The second loss data includes first adversarial loss data calculated based on the first adversarial example and second adversarial loss data calculated based on the second adversarial example. The adversarial perturbation constraint loss data can be generated based on the first adversarial loss and the second adversarial loss.

[0138] In one embodiment, referring to Figure 6 , step 504 may include:

[0139] Step 601: setting a first adversarial weight for a first adversarial example and a second adversarial weight for a second adversarial example, wherein the first adversarial weight is greater than the second adversarial weight;

[0140] Step 602: multiplying the first adversarial weight and the first adversarial loss data to obtain first adversarial disturbance constraint loss data;

[0141] Step 603: multiply the second adversarial weight and the second adversarial loss data to obtain second adversarial disturbance constraint loss data;

[0142] Step 604 : Add the first anti-disturbance constraint loss data and the second anti-disturbance constraint loss data to obtain anti-disturbance constraint loss data.

[0143] The benefit of this embodiment is that a higher weight is set for the first adversarial sample with a lower degree of forgery than for the second adversarial sample, thereby improving the model's ability to recognize individually forged faces or voices and further improving the accuracy of identity verification.

[0144] In step 505, loss fusion is performed based on the first loss data, the second loss data, and the anti-disturbance constraint loss data to obtain target loss data. Loss fusion can be performed by directly adding the three loss data or weighted addition.

[0145] In step 506, the parameters of the initial deep model are adjusted using the target loss data to obtain a deepfake attack defense model. Specifically, the gradients of the parameters of the initial deep model are calculated based on the target loss data; based on the calculated gradients and the learning rate (a hyperparameter that controls the adjustment step), the parameters of the initial deep model are updated until a stopping condition is met (such as the loss falling below a certain threshold, reaching the maximum number of iterations, or the loss data converges), and then the deepfake attack defense model is obtained.

[0146] In summary, the present application can achieve at least the following beneficial effects: (1) Significantly improved security: The detection rate of Deepfake video attacks has been increased from 85% of traditional systems to 99.3% (based on the Celeb-DF dataset test). (2) Micro-expression analysis increases the recognition rate of coercive scenarios (such as users being controlled by a third party) to 91.5%. (3) Real-time optimization: The multimodal parallel processing architecture reduces end-to-end latency from 1.2 seconds to 300ms, improving processing efficiency. (4) Enhanced user experience: Users only need to look directly at the camera when reading random numbers, without having to repeat the operation multiple times, and the account opening success rate is increased by 25%. (5) Compliance is guaranteed.

[0147] See also Figure 7 The embodiment of the present application also provides a multi-modal fusion online account opening device, which is applied to the account opening server and can implement the above-mentioned multi-modal fusion online account opening method. Figure 7 This is a block diagram of the module structure of the multimodal fusion online account opening device provided in an embodiment of the present application. The device includes:

[0148] The request response module 701 is configured to respond to an account opening request sent by a client and send a connection success notification message to the client. The connection success notification message is used to turn on the camera and microphone. The camera is used to capture the target face of the target object, and the microphone is used to capture the target voice of the target object.

[0149] The micro-expression recognition module 702 is used to obtain the target face sent by the client, and perform micro-expression recognition on the target face to obtain target micro-expression features;

[0150] The voiceprint recognition module 703 is used to obtain the target voice sent by the client, and perform voiceprint recognition on the target voice to obtain the target voiceprint feature;

[0151] A feature fusion module 704 is used to perform attention fusion on the target micro-expression feature and the target voiceprint feature to obtain a target fusion feature;

[0152] The identity authentication module 705 is used to authenticate the target object based on the target fusion feature to obtain a legitimate object;

[0153] The online account opening module 706 is used to generate a digital signature for a legal object and open an account for the legal object based on the digital signature.

[0154] In one embodiment, the multimodal fusion online account opening device further includes: a model training module for pre-training a deep fake attack defense model.

[0155] It should be noted that the specific implementation of the multimodal fusion online account opening device is basically the same as the specific embodiment of the multimodal fusion online account opening method mentioned above, and will not be repeated here.

[0156] The present application also provides an electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, the aforementioned multimodal fusion online account opening method is implemented. The electronic device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0157] See also Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0158] The processor 801 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0159] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called by the processor 801 to execute the multimodal fusion online account opening method of the embodiments of this application.

[0160] Input / output interface 803, used to implement information input and output;

[0161] Communication interface 804, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0162] Bus 805 , which transmits information between various components of the device (e.g., processor 801 , memory 802 , input / output interface 803 , and communication interface 804 );

[0163] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0164] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned multimodal fusion online account opening method.

[0165] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0166] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0167] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0169] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0170] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0171] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0173] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0174] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0176] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A multimodal fusion online account opening method, characterized in that: Applied to an account opening server, the method includes: In response to the account opening request sent by the client, a connection success notification message is sent to the client, wherein the connection success notification message is used to turn on a camera and a microphone, wherein the camera is used to capture a target face of a target object, and the microphone is used to capture a target voice of the target object; Acquire the target face sent by the client, and perform micro-expression recognition on the target face to obtain target micro-expression features; Acquire the target voice sent by the client, and perform voiceprint recognition on the target voice to obtain target voiceprint features; Performing attention fusion on the target micro-expression feature and the target voiceprint feature to obtain a target fusion feature; Authenticate the target object based on the target fusion feature to obtain a legitimate object; A digital signature is generated for the legal object, and an account is opened for the legal object based on the digital signature.

2. The method according to claim 1, characterized in that The performing attention fusion on the target micro-expression feature and the target voiceprint feature to obtain a target fusion feature includes: Perform timestamp matching based on the face collection timestamp corresponding to the target micro-expression feature and the voice collection timestamp corresponding to the target voiceprint feature to obtain a matching timestamp; Attention fusion is performed on the target micro-expression feature and the target voiceprint feature that belong to the matching timestamp to obtain the target fusion feature.

3. The method according to claim 2, characterized in that The performing attention fusion on the target micro-expression feature and the target voiceprint feature that belong to the matching timestamp to obtain the target fusion feature includes: Multiplying a preset query weight matrix by the target voiceprint feature belonging to the matching timestamp to obtain a voiceprint query vector; Multiplying a preset key weight matrix by the target micro-expression feature belonging to the matching timestamp to obtain a micro-expression key vector; Multiplying a preset key weight matrix by the target micro-expression feature belonging to the matching timestamp to obtain a micro-expression value vector; Vector fusion is performed according to the voiceprint query vector, the micro-expression key vector, and the micro-expression value vector to obtain the target fusion feature.

4. The method according to claim 3, characterized in that The performing vector fusion according to the voiceprint query vector, the micro-expression key vector, and the micro-expression value vector to obtain the target fusion feature includes: Performing weight calculation on the voiceprint query vector and the micro-expression key vector to obtain a voiceprint micro-expression interaction attention score; Performing nonlinear activation processing on the voiceprint microexpression interaction attention score to obtain a voiceprint microexpression interaction weight vector; A weighted sum is performed on the voiceprint micro-expression interaction weight vector and the micro-expression value vector to obtain the target fusion feature.

5. The method according to any one of claims 1 to 4, characterized in that The performing micro-expression recognition on the target face to obtain target micro-expression features includes: Performing eyebrow detection on the target face to obtain an upward displacement feature of the eyebrow; Performing mouth corner detection on the target face to obtain mouth corner drooping rate features; Performing blink detection on the target face to obtain blink interval time features; The target micro-expression feature is obtained by performing feature fusion based on the eyebrow center upward displacement feature, the mouth corner drooping rate feature and the blink interval time feature.

6. The method according to any one of claims 1 to 4, characterized in that Authenticating the target object based on the target fusion feature to obtain a legitimate object includes: authenticating the target object based on the target fusion feature using a pre-trained deep fake attack defense model to obtain a legitimate object; The method further includes pre-training the deep fake attack defense model, specifically comprising: Constructing a training data set, the training data set including normal samples and adversarial samples, wherein the normal samples are samples in which both the sample face and the sample voice of the sample subject are non-forged, and the adversarial samples are samples in which at least one of the sample face and the sample voice of the sample subject is non-forged, and the number of the normal samples is greater than the number of the adversarial samples; Authentication is performed on the normal sample using a preset initial depth model to obtain a first sample object verification result, and loss calculation is performed based on the first sample object detection result to obtain first loss data; Authentication is performed based on the adversarial sample using a preset initial deep model to obtain a second sample object verification result, and loss calculation is performed based on the second sample object verification result to obtain second loss data; generating disturbance-resistant constraint loss data according to the second loss data; Performing loss fusion according to the first loss data, the second loss data, and the anti-disturbance constraint loss data to obtain target loss data; The parameters of the initial deep model are adjusted using the target loss data to obtain the deep fake attack defense model.

7. The method according to claim 6, characterized in that The adversarial sample includes a first adversarial sample and a second adversarial sample, the first adversarial sample is a sample in which either a sample face or a sample voice is not forged, the second adversarial sample is a sample in which both a sample face and a sample voice are not forged, and the second loss data includes first adversarial loss data calculated based on the first adversarial sample and second adversarial loss data calculated based on the second adversarial sample; Generating the anti-disturbance constraint loss data according to the second loss data includes: Setting a first adversarial weight for the first adversarial example and a second adversarial weight for the second adversarial example, wherein the first adversarial weight is greater than the second adversarial weight; Multiplying the first adversarial weight and the first adversarial loss data to obtain first adversarial disturbance constraint loss data; Multiplying the second adversarial weight and the second adversarial loss data to obtain second adversarial disturbance constraint loss data; The anti-disturbance constraint loss data is obtained by adding the first anti-disturbance constraint loss data and the second anti-disturbance constraint loss data.

8. A multimodal fusion online account opening device, characterized in that: Applied to an account opening server, the device includes: a request response module, configured to respond to an account opening request sent by a client and send a connection success notification message to the client, wherein the connection success notification message is used to turn on a camera and a microphone, wherein the camera is used to capture a target face of a target object, and the microphone is used to capture a target voice of the target object; A micro-expression recognition module is used to obtain the target face sent by the client, and perform micro-expression recognition on the target face to obtain target micro-expression features; a voiceprint recognition module, configured to obtain the target voice sent by the client, and perform voiceprint recognition on the target voice to obtain target voiceprint features; A feature fusion module, configured to perform attention fusion on the target micro-expression feature and the target voiceprint feature to obtain a target fusion feature; An identity authentication module, configured to authenticate the target object based on the target fusion feature to obtain a legitimate object; The online account opening module is used to generate a digital signature for the legal object and open an account for the legal object based on the digital signature.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images