Risk diagnosis method and device based on multiple modes, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a risk diagnosis method and device based on multiple modes, electronic equipment and a storage medium, and belongs to the technical field of financial science and technology and artificial intelligence. The method comprises the following steps: performing face attribute detection on original user image data, performing face action detection on original user video data, performing text feature extraction on original business text data, and fusing detected face attribute features, face action features and business text features to obtain target risk features; and performing risk diagnosis on the target risk features. According to the embodiment of the invention, the face attribute features, the face action features and the business text features are fused to obtain the target risk features, risk diagnosis is carried out on the target risk features, the model can capture the association relationship among the text, image and video data, subjective rating errors of the same user caused by the difference of diagnosers are avoided, and the user experience is improved. Therefore, the accuracy of user risk diagnosis is improved.

Description

Technical Field

[0001] The present application relates to the fields of financial technology and artificial intelligence technology, and in particular to a multimodal risk diagnosis method and device, electronic device and storage medium. Background Art

[0002] At present, user data is usually manually reviewed for user qualifications, thereby conducting risk assessments on users. For example, in the credit approval of financial technology scenarios, it often involves the verification of user identity information and the review of user qualifications to conduct anti-fraud detection on users during the credit approval process. However, in the manual review process, user risk assessments can often only be conducted based on experience, and different diagnosticians will also produce different subjective judgment results in risk diagnosis, resulting in a low accuracy rate for user risk diagnosis. Therefore, how to improve the accuracy of user risk diagnosis has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a multimodal risk diagnosis method and device, an electronic device and a storage medium, aiming to improve the accuracy of user risk diagnosis.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a multimodal risk diagnosis method, the method comprising:

[0005] Acquire multimodal user data of the target object; wherein the multimodal user data includes original user image data, original user video data and original business text data;

[0006] Using a pre-trained multimodal risk diagnosis model to perform facial attribute detection on the original user image data to obtain facial attribute features;

[0007] Performing facial action detection on the original user video data to obtain facial action features;

[0008] Performing text feature extraction on the original business text data to obtain business text features;

[0009] The facial attribute features, the facial action features and the business text features are integrated to obtain target risk features;

[0010] Perform risk diagnosis on the target risk characteristics to obtain a user risk level.

[0011] In some embodiments, the multimodal risk diagnosis model further includes an attention fusion network, which fuses the facial attribute features, the facial action features, and the business text features to obtain target risk features, including:

[0012] The facial attribute features and the facial action features are fused using the attention fusion network to obtain facial visual risk features;

[0013] The face visual risk feature is fused with the business text feature to obtain the target risk feature.

[0014] In some embodiments, the fusing the facial visual risk feature with the business text feature to obtain the target risk feature includes:

[0015] Performing deep feature extraction on the business text features to obtain deep business text features; performing deep feature extraction on the face visual risk features to obtain deep face visual risk features;

[0016] The attention fusion network is used to perform linear enhancement processing on the face visual risk features and the deep business text features to obtain text-enhanced visual features; the attention fusion network is used to perform linear enhancement processing on the deep face visual risk features and the business text features to obtain visually enhanced text features;

[0017] The text-enhanced visual features and the visual-enhanced text features are fused based on preset attention weights to obtain the target risk features.

[0018] In some embodiments, the use of the attention fusion network to perform linear enhancement processing on the face visual risk feature and the deep business text feature to obtain the text enhanced visual feature includes:

[0019] Using the attention fusion network to perform attention interaction processing on the face visual risk features and the deep business text features to obtain initial text visual interaction features;

[0020] Normalizing the initial text-visual interaction features to obtain standard text-visual interaction features;

[0021] The standard text visual interaction feature is subjected to linear enhancement processing to obtain the text enhanced visual feature.

[0022] In some embodiments, the using the attention fusion network to fuse the facial attribute features and the facial action features to obtain the facial visual risk features includes:

[0023] Using the attention fusion network to obtain the attribute risk score of the facial attribute feature; using the attention fusion network to obtain the facial risk score of the facial action feature;

[0024] The facial attribute features and the facial action features are fused according to the attribute risk score and the facial risk score to obtain the facial visual risk features.

[0025] In some embodiments, the multimodal risk diagnosis model further includes a facial action detection network, and the facial action detection is performed on the original user video data to obtain facial action features, including:

[0026] Extracting facial expressions from the original user video data using the facial action detection network to obtain facial expression features;

[0027] Performing facial key point detection on the original user video data to obtain facial key point coordinates;

[0028] Extracting facial motion features from the original user video data according to the facial key point coordinates to obtain facial motion features;

[0029] The facial expression feature and the facial movement feature are fused to obtain the facial action feature.

[0030] In some embodiments, the multimodal risk diagnosis model includes a face attribute detection network, and the use of the pre-trained multimodal risk diagnosis model to perform face attribute detection on the original user image data to obtain face attribute features includes:

[0031] Performing face detection on the original user image data using the face attribute detection network to obtain a user face image;

[0032] Perform attribute detection on the user's facial image to obtain the facial attribute features.

[0033] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a multi-modal risk diagnosis device, the device comprising:

[0034] A multimodal data acquisition module, used to acquire multimodal user data of a target object; wherein the multimodal user data includes original user image data, original user video data and original business text data;

[0035] A face attribute detection module, used to perform face attribute detection on the original user image data using a pre-trained multimodal-based risk diagnosis model to obtain face attribute features;

[0036] A facial action detection module, used to perform facial action detection on the original user video data to obtain facial action features;

[0037] A text feature extraction module is used to extract text features from the original business text data to obtain business text features;

[0038] A feature fusion module, used to fuse the facial attribute features, the facial action features and the business text features to obtain target risk features;

[0039] The risk diagnosis module is used to perform risk diagnosis on the target risk characteristics to obtain the user risk level.

[0040] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0041] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0042] The multimodal risk diagnosis method and device, electronic device and storage medium proposed in the present application obtain multimodal user data of the target object; wherein the multimodal user data includes original user image data, original user video data and original business text data; use a pre-trained multimodal risk diagnosis model to perform face attribute detection on the original user image data to obtain face attribute features; perform facial action detection on the original user video data to obtain facial action features; perform text feature extraction on the original business text data to obtain business text features; fuse facial attribute features, facial action features and business text features to obtain target risk features; perform risk diagnosis on the target risk features to obtain the user risk level. This application first uses a pre-trained multimodal-based risk diagnosis model to perform facial attribute detection on the original user image data, which can extract the user's facial attribute information, and performs facial action detection on the original user video data, which can analyze the user's facial dynamic behavior in the video, providing important clues for assessing user risks and facilitating the subsequent improvement of the accuracy of user risk diagnosis; secondly, text feature extraction is performed on the original business text data, which can provide further clues for predicting user behavior patterns and potential risks; finally, the facial attribute features, facial action features and business text features are integrated to obtain the target risk features, and risk diagnosis is performed on the target risk features. The model can capture the correlation between text, image and video data, avoiding subjective rating errors for the same user due to differences in diagnosticians, thereby improving the accuracy of user risk diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of a multimodal risk diagnosis method provided in an embodiment of the present application;

[0044] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0045] Figure 3 yes Figure 1 Flow chart of step S103 in FIG.

[0046] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0047] Figure 5 yes Figure 4 Flow chart of step S401 in FIG.

[0048] Figure 6 yes Figure 4 Flow chart of step S402 in FIG.

[0049] Figure 7 yes Figure 6 Flowchart of step S602 in FIG.

[0050] Figure 8 is a schematic diagram of the structure of a multi-modal risk diagnosis device provided in an embodiment of the present application;

[0051] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] First, some nouns involved in this application are analyzed:

[0056] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0057] Based on this, the embodiments of the present application provide a multimodal risk diagnosis method and device, an electronic device and a storage medium, aiming to improve the accuracy of user risk diagnosis.

[0058] The multimodal risk diagnosis method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the multimodal risk diagnosis method in the embodiments of the present application is described.

[0059] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0060] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, processing / interaction systems, mechatronics, etc. Artificial intelligence software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0061] The multimodal risk diagnosis method provided in the embodiment of the present application relates to the fields of financial technology and artificial intelligence technology. The multimodal risk diagnosis method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a multimodal risk diagnosis method, etc., but is not limited to the above forms.

[0062] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0063] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user image data, user video data, and business text data, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or jump to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0064] Figure 1 is an optional flow chart of a multimodal risk diagnosis method provided in an embodiment of the present application. Figure 1The method may include but is not limited to steps S101 to S106.

[0065] Step S101, obtaining multimodal user data of a target object; wherein the multimodal user data includes original user image data, original user video data and original business text data.

[0066] Step S102: Use the pre-trained multimodal risk diagnosis model to perform facial attribute detection on the original user image data to obtain facial attribute features.

[0067] Step S103: performing facial action detection on the original user video data to obtain facial action features.

[0068] Step S104: extracting text features from the original business text data to obtain business text features.

[0069] Step S105, fusing the facial attribute features, facial action features and business text features to obtain target risk features.

[0070] Step S106, performing risk diagnosis on the target risk characteristics to obtain the user risk level.

[0071] Steps S101 to S106 illustrated in the embodiment of the present application are performed by acquiring multimodal user data of the target object; wherein the multimodal user data includes original user image data, original user video data and original business text data; using a pre-trained multimodal-based risk diagnosis model to perform facial attribute detection on the original user image data to obtain facial attribute features; performing facial action detection on the original user video data to obtain facial action features; performing text feature extraction on the original business text data to obtain business text features; fusing facial attribute features, facial action features and business text features to obtain target risk features; performing risk diagnosis on the target risk features to obtain a user risk level. This application first uses a pre-trained multimodal-based risk diagnosis model to perform facial attribute detection on the original user image data, which can extract the user's facial attribute information, and performs facial action detection on the original user video data, which can analyze the user's facial dynamic behavior in the video, providing important clues for assessing user risks and facilitating the subsequent improvement of the accuracy of user risk diagnosis; secondly, text feature extraction is performed on the original business text data, which can provide further clues for predicting user behavior patterns and potential risks; finally, the facial attribute features, facial action features and business text features are integrated to obtain the target risk features, and risk diagnosis is performed on the target risk features. The model can capture the correlation between text, image and video data, avoiding subjective rating errors for the same user due to differences in diagnosticians, thereby improving the accuracy of user risk diagnosis.

[0072] In step S101 of some embodiments, specifically, the multimodal user data includes original user image data, original user video data, and original business text data.

[0073] Specifically, original user image data refers to image data containing user portraits; original user video data refers to video data containing user portraits; and original business text data refers to business data determined based on actual business conditions.

[0074] For example, in the credit approval of the financial technology scenario, the target object may be the user applying for credit approval, the original user image data may be the registration photo of the user applying for credit approval when applying for a loan, and the original user video data may be the face-to-face interview video of the user applying for credit approval and the approver when applying for a loan; the original user text data may be the credit information of the user applying for credit approval, and the user credit information includes but is not limited to information such as user identity, user address, user social relations, and user labor relations.

[0075] Specifically, the multimodal user data of the target object can be obtained from the enterprise backend database.

[0076] In step S102 of some embodiments, specifically, the pre-trained multimodal-based risk diagnosis model includes a face attribute detection network, a face action detection network, a text feature extraction network, an attention fusion network, and a risk diagnosis network.

[0077] Specifically, the face attribute detection network is used to perform face attribute detection on the original user image data to extract face attribute information; the face action detection network is used to perform face action detection on the user video data, which can analyze the user's facial dynamic behavior in the video, providing important clues for assessing user risks; the text feature extraction network is used to extract text features from the original business text data, which can provide further clues for predicting user behavior patterns and potential risks; the attention fusion network is used to fuse face attribute features, facial action features and business text features, which helps the model capture the user information correlation relationship between text, image and video data, and avoids the subjective rating error of the same user caused by differences in diagnosticians; the risk diagnosis network is used to perform risk diagnosis on the target risk features, which can accurately assess the risk probability of the user.

[0078] See also Figure 2 In some embodiments, step S102 includes but is not limited to steps S201 to S202:

[0079] Step S201, using a face attribute detection network to perform face detection on original user image data to obtain a user face image.

[0080] Step S202: perform attribute detection on the user's face image to obtain face attribute features.

[0081] In step S201 of some embodiments, the multimodal risk diagnosis model includes a face attribute detection network; wherein the face attribute detection network may be a CNN network.

[0082] Specifically, the user face image refers to an image of the user's facial area.

[0083] For example, in the credit approval field of financial technology scenarios, the approval system will require applicants to upload a clear facial photo, which will be sent to the face detection network to detect the user's facial area and obtain the user's facial image.

[0084] Specifically, the face attribute detection network can be used to perform convolutional feature recognition on the original user image data, and the feature recognition can be used to detect the face contour, eyebrows, nose, eyes and other key parts of the user image, and determine the user's face area features by locating the key area. The user's face area features are input into the activation function in the fully connected layer for activation processing to obtain the user's face image.

[0085] In an optional embodiment of the present application, before performing face detection on the original user image data, the original user image data may be pre-processed, and the pre-processed user image may be used as the original user image data. The pre-processing may include light compensation, grayscale conversion, histogram equalization, normalization, geometric correction, filtering, and sharpening on the original user image data.

[0086] In step S202 of some embodiments, specifically, the facial attribute features refer to attribute information of the user's face, including but not limited to the user's age, user's gender, user's eye bags, user's weight, user's hair, user's skin color, and whether the user wears glasses.

[0087] Specifically, the user's face image is convolutionally pooled through a face attribute detection network to obtain a feature map, and the feature map is upsampled to obtain a feature sampling map, the feature sampling map and the feature map are spliced ​​to obtain facial geometry information and facial texture information of the user's face, and the facial geometry information and facial texture information are input into an activation function for activation processing to obtain facial attribute features.

[0088] Specifically, upsampling refers to sampling the feature map to a specified resolution. For example, a (256, 256, 12) user face image is processed by convolution pooling to obtain a (9, 9, 16) feature map. In order to compare the feature map with the corresponding user face image, this feature map needs to be changed to (256, 256, 12). This process is called upsampling.

[0089] Specifically, the activation function may be a ReLU function.

[0090] For example, in the loan approval field of financial technology scenarios, by evaluating the wrinkles and skin texture of the face, the age of the applicant can be predicted, and by analyzing the facial contour and the distance between feature points, the gender of the applicant can be determined.

[0091] Specifically, through attribute detection, all facial attribute features can be integrated and used in the credit approval decision-making process. For example, if the user's age and professional background do not match their income level, it may become a point that needs further investigation in the credit approval process.

[0092] In this embodiment, by performing attribute detection on the user's facial image, not only the automation level of the approval process is improved, but also the comprehensive assessment capability of the credit risk of the applicant is enhanced, which facilitates improving the accuracy of subsequent risk diagnosis.

[0093] See also Figure 3 In some embodiments, step S103 includes but is not limited to steps S301 to S304:

[0094] Step S301, extracting facial expressions from original user video data using a facial action detection network to obtain facial expression features.

[0095] Step S302: perform facial key point detection on the original user video data to obtain the coordinates of the facial key points.

[0096] Step S303: extract facial motion features from the original user video data according to the facial key point coordinates to obtain facial motion features.

[0097] Step S304, fusing the facial expression features and the facial movement features to obtain facial action features.

[0098] In step S301 of some embodiments, specifically, the facial action detection network can be a combination of a VGG network and an I3D network; wherein the VGG network is used to extract facial expression features of the original user video data, and the I3D network is used to extract facial motion features of the original user video data.

[0099] Specifically, before extracting facial expression features, a face detection algorithm (such as Haar, HOG+SVM, MTCNN algorithm, etc.) is used to locate the user's face in each frame of the original user video data, and then the expression texture features of the user's face are extracted through the convolutional layer included in the VGG network, and the expression texture features are pooled to obtain reduced-dimensional expression texture features. The reduced-dimensional expression texture features are input into an activation function (such as a ReLU function) for activation, and a fixed-length facial expression feature vector is output.

[0100] For example, in the field of credit approval, convolutional pooling can be used to analyze changes in facial expression texture features of credit applicants, that is, to analyze changes in the eyebrows, lips, eyes, and facial expression muscle groups of the user's face to determine the user's facial expression characteristics.

[0101] In step S302 of some embodiments, the facial key point coordinates refer to the coordinates of the key points of the user's face in three-dimensional space, and the key points include the eyes, nose, and mouth of the user's face.

[0102] Specifically, a facial key point detection algorithm (such as the Facemark module of dlib and OpenCV) can be used to locate the key points of the face.

[0103] In this embodiment, by performing facial key point detection on the original user video data, the positions of the key points can be accurately located, which makes it easier for the subsequent model to capture the characteristics of the facial key points changing over time, and helps to understand the complex facial movements later.

[0104] In step S303 of some embodiments, specifically, the facial motion feature refers to the movement process of key points of the face in the original user video data, and the changes in the gaze direction and head posture of the face can be determined through the movement process of the key points, thereby determining the facial motion feature.

[0105] Specifically, firstly, based on the coordinates of facial key points, the 3D convolution layer of the I3D network is used to extract facial change features in the time dimension and the spatial dimension of the original user video data, that is, the dynamic facial change features of each frame of the video image and between frames are extracted; wherein, the spatial dimension can be the spatial change position of the eyes, nose and mouth, and the time dimension refers to the movement of the spatial features over time, such as the user's smile, frown or head posture change; secondly, the 3D pooling layer is used to reduce the spatial dimension and time dimension of the extracted facial change features to ensure that the key features of the facial change features are not lost during the dimensionality reduction; finally, the reduced dimensional facial change features are input into the activation function (such as the ReLU function) for activation to obtain the facial motion features.

[0106] For example, in the field of credit approval, by extracting the facial change features of applicants during the face-to-face review, it is possible to determine whether the applicant has any facial feature changes, such as disappearing during the review process, covering the face with the arm, abnormally tilting or rotating the head, not looking directly at the camera for a long time, blinking frequently when nervous, or twitching the face when uncomfortable. Based on these facial feature changes, it is possible to determine whether the user's facial movements show nervousness or anxiety. If these facial change features appear during the user's asset verification, the user's assets may not be consistent with their claimed financial status, and the reviewer can further investigate the applicant's credit information to ensure the security of the loan.

[0107] In this embodiment, facial motion features are extracted from the original user video data according to the coordinates of facial key points. Facial motion features in the time and space dimensions can be effectively extracted from the video through the I3D network, and the features of video clips at different time points are fused, which helps the model capture richer facial action features in the video, improves the accuracy of facial action feature extraction, and facilitates the subsequent improvement of the accuracy of risk diagnosis.

[0108] In step S304 of some embodiments, specifically, facial expression features and facial motion features may be spliced ​​to use the spliced ​​features as facial action features.

[0109] In this embodiment, by fusing facial expression features and facial movement features to obtain facial action features, a more comprehensive and representative facial action feature can be formed, which further provides direct clues for identifying the emotions and psychological states of the applicant, thereby facilitating the subsequent improvement of the accuracy of risk diagnosis.

[0110] In step S104 of some embodiments, specifically, text feature extraction may be performed on the original business text data through a text feature extraction network.

[0111] Specifically, the text feature extraction network may be a Bert model.

[0112] Specifically, the Bert model first performs word segmentation on the original business text and converts the original business text into a series of tokens. These tokens are then input into the encoder contained in the Bert model, which contains multiple self-attention mechanisms. The contextual relationship of tokens in the original business text is extracted through multiple self-attention mechanisms to obtain business text features containing contextual relationships.

[0113] For example, in the field of credit approval, original business text data may include the applicant's financial report, credit score, and historical loan records. By analyzing the keywords and semantics in the applicant's financial report, the financial stability of the applicant can be evaluated. By reviewing the applicant's credit score and historical loan records, the applicant's repayment willingness and integrity can be understood, which facilitates a more accurate and objective risk assessment in subsequent risk diagnosis.

[0114] See also Figure 4 In some embodiments, step S105 includes but is not limited to steps S401 to S402:

[0115] Step S401, using the attention fusion network to fuse the facial attribute features and facial action features to obtain the facial visual risk features.

[0116] Step S402, fusing the face visual risk features with the business text features to obtain the target risk features.

[0117] See also Figure 5 In some embodiments, step S401 includes but is not limited to steps S501 to S502:

[0118] Step S501, using the attention fusion network to obtain the attribute risk score of the facial attribute feature; using the attention fusion network to obtain the facial risk score of the facial action feature.

[0119] Step S502: The facial attribute features and facial action features are integrated according to the attribute risk score and the facial risk score to obtain the facial visual risk features.

[0120] In step S501 of some embodiments, the attribute risk score can be used to characterize the degree of association between facial attribute features (such as age, gender) and credit risk. The facial risk score can be used to characterize the degree of association between facial action features (such as smiling, frowning or gaze shifting) and credit risk.

[0121] For example, in the field of credit approval, if the applicant's facial movement features are frequent blinking, nodding or facial twitching, it means that the applicant's emotional state is unstable or has potential psychological tension, which may affect the applicant's credit risk. At this time, the facial risk score will increase; if based on the applicant's facial attribute characteristics (such as age, gender, etc.), these attributes can provide additional information about the applicant's identity and current status.

[0122] Specifically, the attribute risk score of the facial attribute feature can be obtained by the following formula:

[0123] Q n =Softmax(Wn X n +b n )

[0124] Among them, Q n represents the attribute risk score of the nth facial attribute feature, W n Represents the weight matrix of the nth facial attribute features, X n represents the nth original user image data, b n Represents the bias term of the nth facial attribute feature.

[0125] Specifically, the facial risk score of the facial action feature can be obtained by the following formula:

[0126]

[0127] in, represents the facial risk score of the mth facial action feature at time t, W m represents the weight matrix of the mth facial action feature, X t represents the original user video data at time t, b m Represents the bias term of the m-th facial action feature.

[0128] In step S502 of some embodiments, specifically, the face visual risk feature may be determined by the following formula:

[0129]

[0130] Among them, F t represents the visual risk characteristics of the face at time t, represents the facial risk score of the mth facial action feature at time t, represents the mth facial action feature at time t, Q n represents the attribute risk score of the nth facial attribute feature, f n Represents the attribute features of the nth face.

[0131] For example, in credit approval, if the applicant shows nervous facial movements in the video, such as frequent blinking or avoiding eye contact, these facial movements may receive a higher facial risk score. At the same time, if the applicant looks young and lacks relevant attributes of work experience, his attribute risk score may also be high. In this case, the credit approval system will multiply these high-risk scores with the corresponding features and sum them up to derive an overall facial visual risk feature with a relatively high risk, prompting the reviewer to conduct a more careful review of the user's application.

[0132] In this embodiment, facial attribute features and facial motion features are fused according to the attribute risk score and the facial risk score. The facial visual risk features obtained at each moment integrate the user's facial dynamic features and static attribute features, so that the model can capture more comprehensive and accurate visual features, which is convenient for improving the accuracy of risk diagnosis later.

[0133] See also Figure 6 In some embodiments, step S402 includes but is not limited to steps S601 to S603:

[0134] Step S601, performing deep feature extraction on business text features to obtain deep business text features; performing deep feature extraction on facial visual risk features to obtain deep facial visual risk features.

[0135] Step S602, use the attention fusion network to perform linear enhancement processing on the facial visual risk features and the deep business text features to obtain text-enhanced visual features; use the attention fusion network to perform linear enhancement processing on the deep facial visual risk features and the business text features to obtain visually enhanced text features.

[0136] Step S603: fusing the text-enhanced visual features and the visual-enhanced text features based on the preset attention weights to obtain target risk features.

[0137] In step S601 of some embodiments, specifically, the deep business text feature refers to the feature of describing the business text in combination with the facial visual image feature; the deep facial visual risk feature refers to the feature of describing the facial visual risk in combination with the business text feature.

[0138] For example, in credit approval, the facial visual risk feature can be an image of the property certificate submitted by the applicant, which is converted into property certificate text for description, and the property certificate text is merged with the business text to obtain deep business text features.

[0139] For example, in credit approval, the business text feature can be a property certificate document, which is described by a real estate certificate image, and the real estate certificate image is merged with the facial visual risk feature to obtain a deep facial visual risk feature.

[0140] Specifically, in the process of extracting deep business text features, the facial visual risk features are first encoded through the forward network included in BiGRU to obtain the forward encoded hidden features H1, and the facial visual risk features are encoded through the backward network included in BiGRU to obtain the backward encoded hidden features H2; secondly, according to the encoding time step, the forward encoded hidden features H1 and the backward encoded hidden features H2 at each moment are fused to obtain the encoded hidden features He, which are used to represent the global information of the facial visual risk features; finally, the encoded hidden features H1 at the last moment and the backward encoded hidden features H2 at the last moment are fused, and the fused hidden features at the last moment are used as the decoding hidden features Hd.

[0141] Furthermore, in the decoding stage, the input features of the decoder are business text features. The similarity between the decoded hidden features Hd and the encoded hidden features He is calculated through the attention mechanism to obtain a hidden weighted attention vector G, which is used to indicate the importance of the contribution of the encoded face visual risk features currently output by the encoded hidden state to the generation of business text features. The hidden weighted attention vector G and the decoded hidden features Hd are concatenated, and the concatenated features are input into the feedforward network to obtain the decoded business text features output after decoding. The decoded business text features are concatenated with the business text features to obtain deep business text features.

[0142] Specifically, the method of performing deep feature extraction on facial visual risk features to obtain deep facial visual risk features is consistent with the above-mentioned method of performing deep feature extraction on business text features to obtain deep business text features, and will not be repeated here.

[0143] In this embodiment, deep feature extraction is performed on business text features to obtain deep business text features, and deep feature extraction is performed on facial visual risk features, so that data of different modalities can be analyzed and processed in a unified feature space, which is convenient for subsequent improvement of the accuracy of risk diagnosis.

[0144] See also Figure 7 In some embodiments, step S602 includes but is not limited to steps S701 to S703:

[0145] Step S701, using the attention fusion network to perform attention interaction processing on the face visual risk features and the deep business text features to obtain the initial text visual interaction features.

[0146] Step S702, normalizing the initial text visual interaction features to obtain standard text visual interaction features.

[0147] Step S703, performing linear enhancement processing on the standard text visual interaction features to obtain text enhanced visual features.

[0148] In step S701 of some embodiments, specifically, the initial text visual interaction features of each attention head can be extracted through the multi-head attention mechanism in CrossAttention included in the attention fusion layer, and the initial text visual interaction features of each attention head are fused to obtain the initial visual text features.

[0149] Specifically, the following formula can be used to describe the initial text visual interaction features extracted by each attention head:

[0150]

[0151] Among them, ATT i (F t ,T i ) represents the initial text visual interaction feature obtained by the interaction between the face visual risk feature of the i-th attention head at time t and the i-th deep business text feature, where the number of attention heads is consistent with the number of deep business text features; W q , W k and W v refers to the attention weight matrix in the attention mechanism; F t represents the visual risk characteristics of the face at time t; T i represents the i-th deep business text feature; d i Represents the i-th scaling factor, which is used to control the gradient scale in the softmax function.

[0152] Furthermore, the initial text visual interaction features can be determined by the following formula:

[0153] R=W i [ATT 1 (F t ,T i ),...,ATT j (F t ,T i )] T

[0154] Among them, R represents the initial text-visual interaction feature, W i Represents the weight matrix that integrates all attention heads, ATT 1 (F t ,T i ) shows the initial text visual interaction features obtained by the interaction between the face visual risk features of the first attention head at time t and the i-th deep business text features, ATT i (F t ,Ti ) represents the initial text visual interaction feature obtained by the interaction between the facial visual risk feature of the i-th attention head at time t and the i-th deep business text feature.

[0155] For example, in credit approval, by calculating the attention weights between visual risk features of the face (such as smiling, frowning, or gaze shifting) and deep business text features (such as financial reports, credit scores, or historical loan records), it is possible to identify which features are most critical for credit risk assessment. If the applicant's facial expression appears nervous or unnatural, and the business text shows an optimistic description of the financial situation, the model can increase the weight of the contradictory features.

[0156] In this embodiment, by utilizing the attention fusion network to perform attention interaction processing on facial visual risk features and deep business text features, the model can automatically focus on the features that are most important for risk diagnosis. Moreover, through the attention mechanism, the system can more accurately identify possible risk signals, thereby facilitating the subsequent improvement of the accuracy of risk diagnosis.

[0157] In step S702 of some embodiments, specifically, the initial text visual interaction feature R may be normalized by LayerNorm to obtain the standard text visual interaction feature R′.

[0158] In an embodiment of the present application, the initial text visual interaction features are normalized to obtain text enhanced visual features, thereby ensuring the stability of the features. When the features are processed as high-dimensional data, the mutual influence between features can be reduced, so that each feature can independently contribute to the final decision, thereby facilitating improving the accuracy of subsequent risk diagnosis.

[0159] In step S703 of some embodiments, the standard text visual interaction feature may be linearly enhanced using the following formula:

[0160] FFN(R)=W2(ReLU(W1*R′+b1))+b2

[0161] Among them, FFN(R) represents text enhanced visual features, W1 represents the linear transformation weight matrix of standard text visual interaction features, W2 represents the linear transformation matrix of standard text visual interaction features after activation processing, R′ represents standard text visual interaction features, and b1 and b2 represent bias terms.

[0162] In this embodiment, by performing linear enhancement processing on the standard text-visual interaction features and introducing a nonlinear activation function, the model further learns the nonlinear relationship between features, is able to identify subtle differences between features, and integrates text and facial visual information to provide a more comprehensive and rich feature, thereby enabling more accurate risk diagnosis.

[0163] In one embodiment of the present application, a method for obtaining visually enhanced text features by linearly enhancing deep facial visual risk features and business text features using an attention fusion network is consistent with a method for obtaining text-enhanced visual features by linearly enhancing facial visual risk features and deep business text features using an attention fusion network, and will not be repeated here.

[0164] In step S603 of some embodiments, specifically, the preset attention weights are α and β, α is used to adjust the relative importance of visually enhanced text features in risk assessment, and β is used to adjust the relative importance of text-enhanced visual features in risk assessment, where α+β=1.

[0165] Specifically, if the visual enhancement of that text feature is more important in anti-fraud risk diagnosis, a higher value is assigned to β.

[0166] For example, the text-enhanced visual feature is [0.8, 0.2, 0.5], the visual-enhanced text feature is [0.3, 0.6, 0.4], α is 0.6, and α is 0.4, then [0.8, 0.2, 0.5]x0.6 = [0.48, 0.12, 0.3], [0.3, 0.6, 0.4]x0.4 = [0.12, 0.24, 0.16], [0.48, 0.12, 0.3] + [0.12, 0.24, 0.16] = [0.60, 0.36, 0.46], then the target risk feature is [0.60, 0.36, 0.46].

[0167] In step S106 of some embodiments, the user risk levels include low-risk users, medium-risk users, and high-risk users.

[0168] Specifically, the target risk characteristics are converted into risk scores, and the user risk levels are divided according to the scores; among them, if the risk score is greater than 0.7, the user is a low-risk user, if 0.4≤risk score≤0.7, the user is a medium-risk user, and if the risk score is less than 0.4, the user is a high-risk user.

[0169] For example, the target risk feature of user A is [0.60, 0.36, 0.46], and the first dimension of the target risk feature is income, the corresponding weight is 0.5, the second dimension is the debt ratio, the corresponding weight is 0.3, and the third dimension is the historical repayment record, the corresponding weight is 0.2. The risk score is 0.5×0.60+0.3×0.36+0.2×0.46=0.5. Since 0.5<risk score≤0.7, the risk level of user A is a medium-risk user.

[0170] In this embodiment, by performing risk diagnosis on the target risk characteristics, fully automatic risk diagnosis can be achieved without human intervention, thereby improving the efficiency of user risk diagnosis and avoiding subjective rating errors for the same user due to differences among diagnosticians, thereby improving the accuracy of user risk diagnosis.

[0171] The embodiment of the present application obtains multimodal user data of the target object; wherein the multimodal user data includes original user image data, original user video data and original business text data; uses a pre-trained multimodal-based risk diagnosis model to perform face attribute detection on the original user image data to obtain face attribute features; performs facial action detection on the original user video data to obtain facial action features; performs text feature extraction on the original business text data to obtain business text features; fuses the facial attribute features, facial action features and business text features to obtain target risk features; performs risk diagnosis on the target risk features to obtain a user risk level. This application first uses a pre-trained multimodal-based risk diagnosis model to perform facial attribute detection on the original user image data, which can extract the user's facial attribute information, and performs facial action detection on the original user video data, which can analyze the user's facial dynamic behavior in the video, providing important clues for assessing user risks and facilitating the subsequent improvement of the accuracy of user risk diagnosis; secondly, text feature extraction is performed on the original business text data, which can provide further clues for predicting user behavior patterns and potential risks; finally, the facial attribute features, facial action features and business text features are integrated to obtain the target risk features, and risk diagnosis is performed on the target risk features. The model can capture the correlation between text, image and video data, avoiding subjective rating errors for the same user due to differences in diagnosticians, thereby improving the accuracy of user risk diagnosis.

[0172] See also Figure 8 The embodiment of the present application further provides a multi-modal risk diagnosis device, which can implement the multi-modal risk diagnosis method described above, and the device includes:

[0173] A multimodal data acquisition module, used to acquire multimodal user data of a target object; wherein the multimodal user data includes original user image data, original user video data and original business text data;

[0174] A face attribute detection module is used to detect face attributes of original user image data using a pre-trained multimodal risk diagnosis model to obtain face attribute features;

[0175] A facial action detection module is used to perform facial action detection on the original user video data to obtain facial action features;

[0176] A text feature extraction module is used to extract text features from original business text data to obtain business text features;

[0177] The feature fusion module is used to fuse facial attribute features, facial action features and business text features to obtain target risk features;

[0178] The risk diagnosis module is used to perform risk diagnosis on the target risk characteristics and obtain the user risk level.

[0179] The specific implementation of the multimodal risk diagnosis device is substantially the same as the specific implementation of the multimodal risk diagnosis method described above, and will not be described in detail herein.

[0180] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned multi-modal risk diagnosis method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0181] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0182] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0183] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store processing systems and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the multi-modal risk diagnosis method of the embodiment of the present application;

[0184] Input / output interface 903, used to implement information input and output;

[0185] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0186] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0187] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0188] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the multi-modality-based risk diagnosis method is implemented.

[0189] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0190] The multimodal risk diagnosis method, multimodal risk diagnosis device, electronic device and storage medium provided in the embodiments of the present application obtain multimodal user data of the target object; wherein the multimodal user data includes original user image data, original user video data and original business text data; use a pre-trained multimodal risk diagnosis model to perform facial attribute detection on the original user image data to obtain facial attribute features; perform facial movement detection on the original user video data to obtain facial movement features; perform text feature extraction on the original business text data to obtain business text features; fuse facial attribute features, facial movement features and business text features to obtain target risk features; perform risk diagnosis on the target risk features to obtain a user risk level. This application first uses a pre-trained multimodal-based risk diagnosis model to perform facial attribute detection on the original user image data, which can extract the user's facial attribute information, and performs facial action detection on the original user video data, which can analyze the user's facial dynamic behavior in the video, providing important clues for assessing user risks and facilitating the subsequent improvement of the accuracy of user risk diagnosis; secondly, text feature extraction is performed on the original business text data, which can provide further clues for predicting user behavior patterns and potential risks; finally, the facial attribute features, facial action features and business text features are integrated to obtain the target risk features, and risk diagnosis is performed on the target risk features. The model can capture the correlation between text, image and video data, avoiding subjective rating errors for the same user due to differences in diagnosticians, thereby improving the accuracy of user risk diagnosis.

[0191] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0192] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0193] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0194] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0195] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0196] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0197] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0198] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0200] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0201] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A multimodal risk diagnosis method, characterized in that: The method comprises: Acquire multimodal user data of the target object; wherein the multimodal user data includes original user image data, original user video data and original business text data; Using a pre-trained multimodal risk diagnosis model to perform facial attribute detection on the original user image data to obtain facial attribute features; Performing facial action detection on the original user video data to obtain facial action features; Performing text feature extraction on the original business text data to obtain business text features; The facial attribute features, the facial action features and the business text features are integrated to obtain target risk features; Perform risk diagnosis on the target risk characteristics to obtain a user risk level.

2. The method according to claim 1, characterized in that The multimodal risk diagnosis model further includes an attention fusion network, which fuses the facial attribute features, the facial action features, and the business text features to obtain target risk features, including: The facial attribute features and the facial action features are fused using the attention fusion network to obtain facial visual risk features; The face visual risk feature is fused with the business text feature to obtain the target risk feature.

3. The method according to claim 2, characterized in that The step of fusing the face visual risk feature with the business text feature to obtain the target risk feature includes: Performing deep feature extraction on the business text features to obtain deep business text features; performing deep feature extraction on the face visual risk features to obtain deep face visual risk features; The attention fusion network is used to perform linear enhancement processing on the face visual risk features and the deep business text features to obtain text-enhanced visual features; the attention fusion network is used to perform linear enhancement processing on the deep face visual risk features and the business text features to obtain visually enhanced text features; The text-enhanced visual features and the visual-enhanced text features are fused based on preset attention weights to obtain the target risk features.

4. The method according to claim 3, characterized in that The method of using the attention fusion network to perform linear enhancement processing on the face visual risk features and the deep business text features to obtain text enhanced visual features includes: Using the attention fusion network to perform attention interaction processing on the face visual risk features and the deep business text features to obtain initial text visual interaction features; Normalizing the initial text-visual interaction features to obtain standard text-visual interaction features; The standard text visual interaction feature is subjected to linear enhancement processing to obtain the text enhanced visual feature.

5. The method according to claim 2, characterized in that: The method of using the attention fusion network to fuse the facial attribute features and the facial action features to obtain facial visual risk features includes: Using the attention fusion network to obtain the attribute risk score of the facial attribute feature; using the attention fusion network to obtain the facial risk score of the facial action feature; The facial attribute features and the facial action features are fused according to the attribute risk score and the facial risk score to obtain the facial visual risk features.

6. The method according to any one of claims 1 to 5, characterized in that: The multimodal risk diagnosis model further includes a facial action detection network, and the facial action detection is performed on the original user video data to obtain facial action features, including: Extracting facial expressions from the original user video data using the facial action detection network to obtain facial expression features; Performing facial key point detection on the original user video data to obtain facial key point coordinates; Extracting facial motion features from the original user video data according to the facial key point coordinates to obtain facial motion features; The facial expression feature and the facial movement feature are fused to obtain the facial action feature.

7. The method according to any one of claims 1 to 5, characterized in that: The multimodal risk diagnosis model includes a face attribute detection network, and the pre-trained multimodal risk diagnosis model is used to perform face attribute detection on the original user image data to obtain face attribute features, including: Performing face detection on the original user image data using the face attribute detection network to obtain a user face image; Perform attribute detection on the user's facial image to obtain the facial attribute features.

8. A multimodal risk diagnosis device, characterized in that: The device comprises: A multimodal data acquisition module, used to acquire multimodal user data of a target object; wherein the multimodal user data includes original user image data, original user video data and original business text data; A face attribute detection module, used to perform face attribute detection on the original user image data using a pre-trained multimodal-based risk diagnosis model to obtain face attribute features; A facial action detection module, used to perform facial action detection on the original user video data to obtain facial action features; A text feature extraction module is used to extract text features from the original business text data to obtain business text features; A feature fusion module, used to fuse the facial attribute features, the facial action features and the business text features to obtain target risk features; The risk diagnosis module is used to perform risk diagnosis on the target risk characteristics to obtain the user risk level.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the multimodal risk diagnosis method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodality-based risk diagnosis method according to any one of claims 1 to 7 is implemented.

Images