Risk assessment method and device based on user description, equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence and financial science and technology, and relates to a risk assessment method based on user description, which comprises the following steps: extracting a user image sequence and user voice data from video data of a target user; inputting the user image sequence and the user voice data into the trained risk prediction model, and sequentially performing element segmentation, multi-modal feature extraction and prediction classification to obtain corresponding multi-dimensional risk tags; and performing comprehensive risk assessment based on the multi-dimensional risk tag to obtain a risk result of the target user. The invention further provides a risk assessment device and equipment based on user description and a storage medium. In addition, the invention also relates to a block chain technology, and the video data of the target user can be stored in a block chain. The risk assessment efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to a risk assessment method, apparatus, device and storage medium based on user description. Background Art

[0002] In recent years, with the rapid development of Internet technology, online businesses have become more and more numerous. In order to ensure the normal processing of online businesses, it is usually necessary to conduct risk assessments on users who initiate businesses. Especially in the current field of financial technology, the construction and optimization of risk control systems are the key to maintaining the competitiveness of financial institutions. With the rapid development of technology and the increasing complexity of the financial environment, the amount of data that financial institutions need to process has increased dramatically. Traditional risk control often relies on relatively simple statistical methods and manual review, which may be effective when the data volume is small and the structure is relatively simple. However, in the current big data environment, these methods are far from meeting the processing requirements of complex scenarios and large amounts of data, and are prone to loopholes and delays, thereby increasing the operational risks of financial institutions. In addition, with the advancement of technology, fraud methods are also constantly upgrading, making it difficult for traditional risk control systems to cope with new fraud strategies. For example, false identities generated using complex algorithms, transaction behavior imitation using high-tech means, etc., these new fraud behaviors are often difficult to effectively identify and prevent in traditional risk control systems. Especially in the field of credit approval, it often involves the verification of customer identity information, loan qualifications and other information. In addition, during remote video review, the approver needs to subjectively check the customer's scene, whether the customer information is true, and other items one by one, in order to ultimately assess whether the customer has a fraud risk. Manual approval can often only make a qualitative judgment on the fraud risk, and it is difficult to make a quantitative description. The different judgment standards of the approvers will also lead to a wide range of subjective judgment results, resulting in low efficiency and accuracy of risk assessment. Summary of the invention

[0003] The purpose of the embodiments of the present application is to propose a risk assessment method, device, equipment and storage medium based on user description, so as to solve the technical problem that risk assessment in related technologies can only be judged qualitatively and is difficult to describe quantitatively, resulting in low efficiency and accuracy of risk assessment.

[0004] In order to solve the above technical problems, the embodiment of the present application provides a risk assessment method based on user description, which adopts the following technical solution:

[0005] Acquire video data of a target user, and extract a user image sequence and user voice data from the video data;

[0006] Inputting the user image sequence and the user voice data into a trained risk prediction model, wherein the risk prediction model includes an image segmentation module, a multimodal feature extraction module, and a prediction classification module;

[0007] By means of the image segmentation module, the user image sequence is segmented into elements according to the user description categories to obtain a multi-dimensional segmented image;

[0008] Inputting the multi-dimensional segmented image and the user voice data into the multi-modal feature extraction module to perform feature extraction respectively to obtain multi-dimensional user features;

[0009] The multi-dimensional user features are classified and predicted by the prediction and classification module to obtain corresponding multi-dimensional risk labels;

[0010] A comprehensive risk assessment is performed based on the multi-dimensional risk tags to obtain the risk results of the target user.

[0011] In order to solve the above technical problems, the embodiment of the present application further provides a risk assessment device based on user description, which adopts the following technical solution:

[0012] An acquisition module, used to acquire video data of a target user, and extract a user image sequence and user voice data from the video data;

[0013] An input module, used to input the user image sequence and the user voice data into a trained risk prediction model, wherein the risk prediction model includes an image segmentation module, a multimodal feature extraction module and a prediction classification module;

[0014] An image segmentation module is used to perform element segmentation on the user image sequence according to the user description category to obtain a multi-dimensional segmented image;

[0015] A multimodal feature extraction module, used for inputting the multi-dimensional segmented image and the user voice data into the multimodal feature extraction module for feature extraction respectively, to obtain multi-dimensional user features;

[0016] A prediction and classification module, used to classify and predict the multi-dimensional user features to obtain corresponding multi-dimensional risk labels;

[0017] The risk assessment module is used to perform a comprehensive risk assessment based on the multi-dimensional risk tags to obtain the risk results of the target user.

[0018] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0019] The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the risk assessment method based on user description as described above when executing the computer-readable instructions.

[0020] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0021] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the risk assessment method based on user description as described above are implemented.

[0022] Compared with the prior art, this application has the following beneficial effects:

[0023] The present application provides a risk assessment method based on user description, which extracts user image sequences and user voice data from the target user's video data, extracts multi-dimensional user features from the user image sequences and user voice data through a trained risk prediction model, and performs classification prediction based on the multi-dimensional user features to obtain corresponding multi-dimensional risk labels, and performs comprehensive risk assessment on the multi-dimensional risk labels to obtain risk results; the present application predicts risk labels through a risk prediction model, which can improve the efficiency of risk assessment. At the same time, by evaluating the user's risk from multiple dimensions, it can accurately reflect the user's risk and greatly improve the accuracy of user risk assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0026] Figure 2 is a flow chart of an embodiment of a risk assessment method based on user description according to the present application;

[0027] Figure 3 is a schematic structural diagram of an embodiment of a risk assessment device based on user description according to the present application;

[0028] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0030] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0032] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0033] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0034] This application provides a risk assessment method based on user description, which can be applied to Figure 1 In the system architecture 100 shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0035] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0036] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0037] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0038] It should be noted that the risk assessment method based on user description provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the risk assessment device based on user description is generally set in the server / terminal device.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0040] Continue to refer Figure 2 , shows a flow chart of an embodiment of a risk assessment method based on user description according to the present application, comprising the following steps:

[0041] Step S201, obtaining video data of a target user, and extracting a user image sequence and user voice data from the video data.

[0042] The target user is the user to be risk assessed, wherein the target user is obtained according to the risk assessment request. In the field of credit approval, upon receiving the review instruction, the target user is determined according to the user identifier in the review instruction, and a video request is sent to the target user. Based on the target user's response to the video request, the video data of the target user is obtained, wherein the video data is the real-time video obtained during the review.

[0043] In this embodiment, the risk assessment method based on the user description is executed on the electronic device (eg Figure 1 The server / terminal device shown in the figure can obtain the video data of the target user through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0044] In this embodiment, the video data can be real-time or non-real-time. The essence of video data is a continuous image sequence, which is composed of frame images arranged in chronological order, and each frame image has its corresponding timestamp. The video data of the present application can be user-side video data of the requesting party (for example, the customer who initiates the loan request), or it can be composite video data synthesized by the requesting party and the requested party (for example, the credit reviewer). The present application can obtain one channel of user-side video data by having one camera shoot the customer and the customer's surrounding environment, or it can obtain multiple channels of user-side video data by having multiple cameras shoot the user and the customer's surrounding environment.

[0045] In this embodiment, the user image sequence is extracted from the video data, specifically, the video data is framed at a preset time interval to obtain the user image sequence. The preset time interval can be set to any time interval. For example, the preset time interval can be set to 0.2s. The sequence number of the frame image is represented as f, and the image sequence is represented as {f1, f2, ..., f n}, the timestamps corresponding to each frame image in the user image sequence are represented as t1, t2, ..., t n .

[0046] In this embodiment, the method for extracting the user voice data may be to decode the video file and separate the audio stream; or to use video editing software or a special audio extraction tool to separate the audio stream from the video data.

[0047] It should be emphasized that in order to further ensure the privacy and security of the target user's video data, the above-mentioned target user's video data can also be stored in a blockchain node.

[0048] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0049] Step S202, inputting the user image sequence and the user voice data into the trained risk prediction model, the risk prediction model includes an image segmentation module, a multimodal feature extraction module and a prediction classification module.

[0050] A risk prediction model is an analytical tool used to predict and assess the risks of loans, investments or other financial activities. These models are usually based on historical data and statistical methods, using machine learning techniques to identify risk factors and predict possible risk events in the future.

[0051] In this embodiment, the risk prediction model includes an image segmentation module, a multimodal feature extraction module and a prediction classification module.

[0052] Step S203: using an image segmentation module, the user image sequence is segmented into elements according to the categories described by the user to obtain a multi-dimensional segmented image.

[0053] Among them, the user description category includes three major categories and six subcategories. The three major categories are risk label category, behavior description category and compliance category. The risk label category includes two subcategories: identity verification category and scenario verification category. The behavior description category includes three subcategories: customer behavior, customer status and environmental information.

[0054] Among them, identity verification includes 3 points:

[0055] 1) The proportion of matches with non-users:

[0056] Face detection and face recognition algorithms are used to determine the number of times someone other than the customer appears in the video review. Generally, the customer is required to answer questions in a quiet and closed environment during a video review. If someone other than the customer appears on camera multiple times, there may be suspicion of gang involvement, fraud such as others directing the answers, etc.

[0057] 2) Probability of matching as non-live:

[0058] Image and video algorithms are used to determine whether the customer is non-living (photos, posters, etc.) during video review. Non-living customers have a very high risk of fraud. This type of fraud is intended to obtain a loan by disguising oneself as someone else.

[0059] 3) Face quality judgment:

[0060] Image algorithms are used to determine whether the customer's facial quality meets the audit requirements during video review. If the facial quality fails to pass the review for a long time (the image is blurred, the brightness is too high or too low, the face is incomplete), there may be a problem of fraudulent customers using it to cover up certain facial or scene abnormalities.

[0061] The scenario verification category includes 4 points:

[0062] 1) Background quality judgment:

[0063] Through the continuous frame image algorithm, it is judged whether the background of the user meets the review requirements during the video review. In view of the emergence of many technical means of changing the background in recent years, if the user deliberately hides the background of his location by changing the background or the background of the customer is unclear, there may be suspicion of fraudulent users using it to cover up abnormal scenes, gang crimes, and bystanders guiding answers.

[0064] 2) Scene item judgment:

[0065] Image algorithms are used to determine whether there are any paper or document files near the scene where the user is located during video review. If such files exist in the scene and the customer is found to frequently check the documents in subsequent action judgments, the customer may be suspected of fraud, such as falsifying his or her own information and needing to answer questions prompted by the documents.

[0066] 3) There is a third person in the scene and there is communication:

[0067] Image and video algorithms are used to determine whether there are other people around the customer and communicating with them during the video review. This type of risk mainly involves bystanders guiding or a third party suggesting that the customer answer certain questions, and the general review requirement is that there should be no other people present around. The appearance of this scene is a violation in the video review.

[0068] 4) Detection of pornographic and violent behavior:

[0069] Image and video algorithms are used to determine whether topless, pornographic, violent or terrorist behavior appears during video review. This type of risk mainly involves illegal actions. The presence of this type of indecent behavior in the recording is a serious violation.

[0070] Customer behavior includes 5 points:

[0071] 1) Eye contact movements:

[0072] The video algorithm is used to determine the time and frequency of a user's specific eye movement during video review. Specific eye movement patterns include but are not limited to focusing on a certain direction, dodging, not looking directly at the camera for a long time, blinking frequently, looking around, etc. This type of action comprehensively describes the appearance and time of the user's eye movement. The risk level of the action can be analyzed in detail based on the content of the questioning at that time.

[0073] 2) Head posture movements:

[0074] The video algorithm determines the time and frequency of a user's head movements in a specific pattern during video review. The specific pattern of head movements includes but is not limited to focusing the head in a certain direction, uncoordinated gaze and head movements, not facing the camera for a long time, and rapid head movements. This type of action comprehensively describes the occurrence and time of the user's head movement. The risk level of the action can be analyzed in detail based on the content of the questioning at that time.

[0075] 3) Hand movements:

[0076] The video algorithm determines the time and frequency of a user's hand movements in a specific pattern during video review. Specific hand movements include but are not limited to covering facial organs (eyes, ears, mouth, nose, etc.) with hands, holding cigarettes, covering hair and neck with hands, and using too many gestures during communication. This type of action comprehensively describes the occurrence and time of the user's hand movements. The risk level of the action can be analyzed in detail based on the content of the question at that time.

[0077] 4) Abnormal actions:

[0078] The video algorithm determines the time and frequency of abnormal actions of a user in a specific pattern during video review. Abnormal actions of a specific pattern include but are not limited to abnormal vision (frequently pulling the phone close to watch), abnormal hearing (frequently pulling the phone close to the ear), etc. This type of action comprehensively describes the occurrence and time of abnormal actions of users. The risk level of the action can be analyzed in detail based on the characteristics of the abnormal action.

[0079] 5) Other actions:

[0080] The video algorithm determines the time and frequency of other specific patterns of user actions during video review. Other specific patterns include but are not limited to frequent drinking of water, holding a cigarette and smoking, wearing headphones, wearing a mask, etc. This type of action comprehensively describes the appearance and time of other types of user actions. The risk level of the action can be analyzed in detail based on the characteristics of this type of action.

[0081] User status includes 4 points:

[0082] 1) User emotions:

[0083] The video algorithm determines the time and frequency of the user's specific emotional state during the video review. The specific emotional state includes but is not limited to happiness (excited, satisfied, etc.), disappointment (sad, unhappy, etc.), anger (angry, resistant, etc.), tension, anxiety, etc. This type of action comprehensively describes the emergence and time of user emotions. The risk level of the action can be analyzed in detail based on the characteristics of this type of action.

[0084] 2) User's appearance and attire:

[0085] The image algorithm is used to judge the user's current appearance and clothing during the video review. The appearance and clothing include but are not limited to the user's age, gender, hairstyle, skin color, clothing type, whether they are wearing accessories, etc. This type of description describes the user's appearance and clothing in a comprehensive manner. The risk level contained in each description category can be analyzed in detail based on the characteristics of this type.

[0086] 3) Health status:

[0087] The image algorithm is used to determine the facial attributes of the user related to the health status at the current moment in the video review, including but not limited to large eye bags, dark circles, facial swelling, facial depression, complexion, lip color, etc. This type describes the facial attributes of the user related to the health status in a comprehensive manner. The risk level contained in each attribute class can be analyzed in detail based on the characteristics of this type.

[0088] 4) Voice status class:

[0089] The voice algorithm determines the time and frequency of a user's specific voice state during video review. The specific voice state includes but is not limited to hesitation, coughing, wheezing (sighing), repeated expressions, sudden changes in pitch, etc. This type comprehensively describes the occurrence and time of various voice states of the user. The risk level contained in each state can be analyzed in detail based on its characteristics.

[0090] Image segmentation is a key technology in the field of computer vision, which involves the process of dividing an image into multiple regions or objects. In some optional implementations, the image segmentation module includes a semantic segmentation unit, a feature extraction layer, a region proposal layer, an alignment layer, a multi-task branching layer, and an output layer.

[0091] The step of performing element segmentation on the user image sequence according to the user description category by the image segmentation module to obtain a multi-dimensional segmented image includes:

[0092] Perform semantic segmentation on the user image sequence through the semantic segmentation unit to obtain a semantic segmentation image;

[0093] Perform feature extraction on the semantic segmentation image to obtain a semantic segmentation feature map;

[0094] The semantic segmentation feature map is input into the region proposal layer to extract the region of interest and obtain the candidate region feature map;

[0095] Input the candidate region feature map into the alignment layer to obtain the aligned region of interest feature map;

[0096] The aligned region of interest feature map is input into the multi-task branch layer for classification, bounding box regression and segmentation, and the target user description category, target bounding box and target instance mask image are obtained accordingly. The target instance mask image is a multi-dimensional segmentation image.

[0097] The user image sequence is preliminarily segmented through the semantic segmentation unit to improve the accuracy of feature extraction. The feature extraction layer can use a convolutional neural network to extract features from the semantic segmentation image. The region proposal layer uses the region proposal network RPN (Region Proposal Network, RPN) to extract the region of interest from the semantic segmentation feature map, and preliminarily screen out the region that may be the target as the candidate region. The candidate region feature map is input into the alignment layer, and the alignment layer is the region of interest alignment layer RoI (Region of Interest, RoI). The candidate region is aligned through the region of interest alignment layer RoI to obtain an aligned region of interest.

[0098] The multi-task branch layer is a classification branch, a bounding box regression branch, and a segmentation branch. The feature map of the aligned region of interest is input into the classification branch, the bounding box regression branch, and the segmentation branch respectively. The classification branch is used to classify the aligned region of interest and determine the target user description category. The bounding box regression branch is used to locate the boundary of the target. The segmentation branch uses a pixel-level segmentation network FCN (Fully Convolutional Network, FCN). The pixel-level segmentation network FCN is used to confirm the boundary of the target in the aligned region of interest, and generate a target instance mask image for the aligned region of interest predicted as the target, thereby separating the target from the image and realizing pixel-level segmentation. For example, the target includes objects that appear in the user image sequence, such as faces, papers, and files.

[0099] In this embodiment, the multi-dimensional segmented images include face images, scene object images, human body images, expression images, images of different body parts, user background images, etc.

[0100] In some optional implementations of this embodiment, the semantic segmentation unit includes an encoder, a decoder and a connection layer, the encoder includes a first encoding layer, a second encoding layer, a third encoding layer and a fourth encoding layer, and the decoder includes a first decoding layer, a second decoding layer, a third decoding layer and a fourth decoding layer; wherein the step of performing semantic segmentation on the user image sequence by the semantic segmentation unit to obtain a semantically segmented image includes:

[0101] Extract features from the user image sequence through the first encoding layer to obtain a first image feature map, and transmit the first image feature map to the first decoding layer;

[0102] Downsampling the first image feature map to obtain a first pooling feature, and inputting the first pooling feature into the second encoding layer;

[0103] Extracting features from the first pooled features through a second encoding layer to obtain a second image feature map, and transmitting the second image feature map to a second decoding layer;

[0104] Downsampling the second image feature map to obtain a second pooled feature, and inputting the second pooled feature into a third encoding layer;

[0105] Performing feature extraction on the second pooling feature through a third encoding layer to obtain a third image feature map, and transmitting the third image feature map to a third decoding layer;

[0106] Downsampling the third image feature map to obtain a third pooled feature, and inputting the third pooled feature into a fourth encoding layer;

[0107] Perform feature extraction on the third pooling feature through a fourth encoding layer to obtain a fourth image feature map, and transmit the fourth image feature map to a fourth decoding layer;

[0108] Down-sampling the fourth image feature map to obtain a down-sampled feature map, and inputting the down-sampled feature map into the connection layer;

[0109] Upsampling the downsampled feature map through the connection layer to obtain enhanced upsampled features, and transmitting the enhanced upsampled features to the fourth decoding layer;

[0110] The fourth decoding layer is used to concatenate the enhanced up-sampled features and the fourth image feature map to obtain a fourth-level fusion feature, and the fourth-level fusion feature is convolved to obtain a first deconvolution feature;

[0111] Upsampling the first deconvolution feature to obtain a first upsampled feature map;

[0112] The first up-sampled feature map and the third image feature map are concatenated through the third decoding layer to obtain a third-level fusion feature, and the third-level fusion feature is convolved to obtain a second deconvolution feature;

[0113] Upsampling the second deconvolution feature to obtain a second upsampled feature map;

[0114] The second up-sampled feature map and the second image feature map are concatenated through the second decoding layer to obtain a second-level fusion feature, and the second-level fusion feature is convolved to obtain a third deconvolution feature;

[0115] Upsampling the third deconvolution feature to obtain a third upsampled feature map;

[0116] The third up-sampled feature map and the first image feature map are concatenated through the first decoding layer to obtain a first-level fusion feature, and the first-level fusion feature is convolved to obtain a third deconvolution feature;

[0117] Perform a convolution operation on the third deconvolution feature and output a semantic segmentation image.

[0118] By performing semantic segmentation on the user image sequence through the semantic segmentation unit, features containing detail information can be obtained during the image feature extraction process, and the image can be segmented more accurately, thereby improving the accuracy of the semantic segmentation results and further improving the accuracy of feature extraction.

[0119] Through image segmentation, the present application can achieve fine segmentation of user image sequences, improve the accuracy and efficiency of segmentation, lay the foundation for subsequent feature extraction, increase the depth and breadth of risk prediction, and improve the efficiency and accuracy of risk prediction.

[0120] Step S204: input the multi-dimensional segmented image and the user voice data into a multi-modal feature extraction module for feature extraction to obtain multi-dimensional user features.

[0121] The multi-dimensional segmented image and user voice data are input into the multi-modal feature extraction module for multi-modal feature extraction to obtain multi-dimensional user features, which are a collection of user features of different dimensions.

[0122] In some optional implementations, the multimodal feature extraction module includes a user feature extraction unit, a scene feature extraction unit, a visual feature extraction unit, and a speech feature extraction unit.

[0123] The step of inputting the multi-dimensional segmented image and the user voice data into the multi-modal feature extraction module for feature extraction to obtain multi-dimensional user features includes:

[0124] The multi-dimensional segmented images are input into the user feature extraction unit, the scene feature extraction unit and the visual feature extraction unit respectively, and the user voice data is input into the voice feature extraction unit;

[0125] The user feature extraction unit extracts features from the multi-dimensional segmented image to obtain user attribute features;

[0126] The scene feature extraction unit extracts features from the multi-dimensional segmented image to obtain scene description features;

[0127] The visual feature extraction unit extracts features from the multi-dimensional segmented image to obtain the image visual features;

[0128] The voice feature extraction unit extracts features from the user's voice data to obtain voice features;

[0129] User attribute features, scene description features, image visual features and voice features are combined into multi-dimensional user features for output.

[0130] In this embodiment, the multi-dimensional segmented image is a segmented image of multiple dimensions, and the segmented images of different dimensions are used to predict the risk of different user description categories. For example, inputting face images, human body images, expression images, and images of different body parts into the user feature extraction unit, inputting scene object images and user background images into the scene feature extraction unit, and inputting all multi-dimensional segmented images into the visual feature extraction unit can improve the efficiency of feature extraction.

[0131] In some optional implementations of this embodiment, the user feature extraction unit includes a face feature extraction layer, a behavior feature extraction layer, and an expression feature extraction layer; wherein the step of extracting features from the multi-dimensional segmented image by the user feature extraction unit to obtain user attribute features includes:

[0132] Input the multi-dimensional segmented image into the face feature extraction layer to obtain a face feature map;

[0133] Input the multi-dimensional segmented image into the behavior feature extraction layer to obtain the user behavior feature map;

[0134] Input the multi-dimensional segmented image into the expression feature extraction layer to obtain the user expression feature map;

[0135] The face feature map, user behavior feature map and user expression feature map are output as user attribute features.

[0136] Among them, the face feature extraction layer can use HOG feature extraction, convolutional neural network, multi-task cascade CNN model, residual network deep learning model, etc. For example, the face feature extraction layer includes residual network resnet and FPN (Feature Pyramid Networks). Specifically, features are extracted through resnet, and features of different layers are sent to FPN for feature fusion, and finally a face feature map is output.

[0137] In an optional implementation, the behavior feature extraction layer uses a Bi-LSTM layer. The Bi-LSTM layer can automatically obtain data features, and use sequential and reverse recurrent neural networks to obtain two independent hidden layer representations for each input data. Then, the two hidden layer representations are subjected to certain calculations (splicing or adding) to obtain a final hidden layer representation, which is sent to the output layer for subsequent calculations. This hidden layer representation contains semantic information from both the previous and the following contexts for each data in the time series data. Among them, the Bi-LSTM layer includes an input layer, a forward hidden layer, and a backward hidden layer. Specifically, the multi-dimensional segmented image is converted into time series data in chronological order, and the time series data is respectively input into the forward hidden layer and the backward hidden layer for feature information transmission and updating, and the forward hidden features and the backward hidden features are respectively obtained; the forward hidden features and the backward hidden features are spliced ​​according to the corresponding positions to obtain the hidden state, and the hidden state is the user behavior feature map.

[0138] Among them, the forward hidden feature is The backward hidden feature is Splicing to get h t for Then we can get the complete hidden state (h1,h2,…,h n )∈R n×m .

[0139] In an optional implementation, the expression feature extraction layer can use a CNN-LSTM hybrid model. Specifically, a convolutional neural network and a long short-term memory network (LSTM) are combined to extract the spatial features of the image using CNN, and then extract the time series features through LSTM, which is suitable for dynamic expression recognition. For example, the VGG-19 skeleton is used to extract spatial features, and then the bidirectional LSTM (Bi-LSTM) is used to extract spatiotemporal features.

[0140] By extracting user attribute features of different dimensions separately, the accuracy of extracted features can be improved and user attribute features can be fully acquired.

[0141] In this embodiment, the scene verification category includes background quality judgment, scene object judgment, the presence of a third person in the scene and the presence of communication and pornographic and violent behavior detection. In one example, the scene feature extraction unit includes a target detection layer, a face detection layer, a human shape detection layer, and a behavior recognition layer. Among them, the target detection layer (such as YOLO, Faster R-CNN, SSD) detects objects in the scene; the face detection layer (such as MTCNN, FaceNet) is used to detect faces in the scene; the human shape detection layer uses a human posture estimation model (such as OpenPose) to detect human posture and determine whether there is a third person in the scene; the behavior recognition layer (such as LSTM, Transformer) is used to analyze the behavior patterns in the video and identify communication behaviors.

[0142] In this embodiment, the background quality judgment can judge the complexity and quality of the background based on the characteristics of the background area in the multi-dimensional segmented image, such as color distribution, texture, edge, etc.

[0143] In an optional implementation, the visual feature extraction unit may use a convolutional neural network (CNN), a pyramid scene parsing network (PSPNet), a residual network (ResNet) or a feature pyramid network (FPN), etc. The image visual features include color features, texture features, shape features, etc.

[0144] In this embodiment, feature extraction is performed on the user voice data to obtain voice features, which include but are not limited to duration-related features, fundamental frequency-related features, energy-related features, cepstral coefficients, and Mel-frequency cepstral coefficients. The voice feature extraction unit may use a deep feature extraction neural network.

[0145] Through multi-dimensional user feature extraction, user risk characteristics can be fully and accurately reflected, which can greatly improve the accuracy and reliability of user risk assessment.

[0146] Step S205, classify and predict the multi-dimensional user features through the prediction and classification module to obtain corresponding multi-dimensional risk labels.

[0147] Among them, the prediction and classification module includes prediction and classification units with the same number of dimensions as the multi-dimensional user features. Specifically, the multi-dimensional user features are input into the corresponding prediction and classification units for class prediction to obtain the risk labels corresponding to each dimensional user feature; the risk labels are combined and output to obtain multi-dimensional risk labels.

[0148] The prediction classification unit can be selected according to the actual classification task. For example, for face recognition tasks and liveness detection tasks, the prediction classification unit uses an SVM classifier; for expression recognition tasks, the prediction classification unit can use a fully connected layer + softmax layer.

[0149] In this embodiment, for each classification task, the correlation between the user features of each dimension is calculated, and the user features of the corresponding dimensions are fused according to the correlation and then input into the corresponding prediction classification unit.

[0150] Among them, the risk label corresponds to the user description category. For example, the risk label includes but is not limited to the proportion of times that the person is not the person himself, the probability of non-liveness, the face quality does not meet the audit requirements, the user's background meets the audit requirements, etc.

[0151] Predicting risk labels through multi-dimensional user characteristics can improve prediction accuracy and comprehensiveness, thereby improving risk assessment accuracy.

[0152] Step S206, performing a comprehensive risk assessment based on the multi-dimensional risk tags to obtain the risk results of the target user.

[0153] Specifically, the preset risk assessment rules are called to score each risk tag in the multi-dimensional risk tags to obtain a scoring result; the risk level of each risk tag is determined based on the scoring result; the risk levels are weighted and calculated to obtain the risk result of the target user.

[0154] Different risk labels correspond to different risk assessment rules. For example, different scores correspond to different ranges of the proportion of times that someone other than the real person appears; scores correspond to different degrees of verification of blurred images, too high or too low brightness, incomplete faces, etc. The risk level of each risk label is determined based on the scoring results. The risk levels include low risk, medium-low risk, medium risk, medium-high risk, high risk, very high risk, unacceptable risk, acceptable risk, potential risk, emergency risk, long-term risk and short-term risk.

[0155] The process of weighting the risk level and obtaining the risk result of the target user is as follows:

[0156] Assign a weight to each risk tag, which reflects the relative importance of each risk tag in the overall risk assessment. The weight assignment should be based on professional knowledge, historical data and risk management strategy;

[0157] Multiply the risk level of each risk label by its corresponding weight, and then sum up all the products to get the total risk value of the target user.

[0158] The calculated total risk value is mapped to the predefined risk level. For example, the user's risk level is divided into low, medium, high, etc. according to the total risk value.

[0159] Explain the calculated risk results, clarify their specific meaning to the target users, and put forward corresponding risk management recommendations.

[0160] By weighting the risk level and comprehensively considering multiple risk tags, the assessment results are more comprehensive; by allocating weights, the accuracy of risk assessment is improved and the risk status of target users is more realistically reflected; it can provide quantitative risk assessment results to support more scientific and objective decision-making; identify high-risk users or assets, prioritize risk control and mitigation measures, and reduce potential losses; and according to the risk assessment results, reasonably allocate risk management resources to improve resource utilization efficiency.

[0161] This application uses a risk prediction model to predict risk labels, which can improve the efficiency of risk assessment. At the same time, it evaluates user risks from multiple dimensions, which can accurately reflect user risks and greatly improve the accuracy of user risk assessment.

[0162] In some optional implementations, before the step of inputting the user image sequence and the user voice data into the trained risk prediction model, the following steps are further included:

[0163] Obtain the historical user image set and the corresponding voice data set of all users as the sample set, and each data in the sample set is marked with the user's real risk label;

[0164] The pre-built initial risk prediction model is trained based on the sample set and the true risk label to obtain the final risk prediction model after training.

[0165] Specifically, collect historical user image sets and corresponding voice data sets of all users in the system, where historical user images include facial images, behavioral images, images of scenes in which users are located, etc.; preprocess the historical user image set and voice data set respectively, divide the preprocessed sample set into a training set and a test set, input the training set into a pre-built initial risk prediction model to obtain a predicted risk label; calculate the loss value between the predicted risk label and the true risk label, adjust the model parameters of the initial risk prediction model based on the loss value, and obtain a model to be verified; use the test set to test the model to be verified to obtain test results; calculate the prediction accuracy based on the test results, and output the trained risk prediction model when the prediction accuracy is greater than or equal to a preset threshold.

[0166] Well-trained models can predict or classify more accurately, reducing errors and uncertainties; trained models can automate many tasks, reduce manual intervention and improve efficiency; automated and optimized decision-making processes can reduce operating costs, especially in large-scale operations, and improve user experience. In the fields of finance, healthcare, etc., trained risk prediction models can help identify and manage risks and reduce potential losses.

[0167] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0168] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0169] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0170] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a risk assessment device based on user description, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0171] like Figure 3As shown, the risk assessment device 300 based on user description in this embodiment includes: an acquisition module 301, an input module 302, an image segmentation module 303, a multimodal feature extraction module 304, a prediction and classification module 305 and a risk assessment module 306. Among them:

[0172] The acquisition module 301 is used to acquire the video data of the target user, and extract the user image sequence and the user voice data from the video data;

[0173] The input module 302 is used to input the user image sequence and the user voice data into a trained risk prediction model, wherein the risk prediction model includes an image segmentation module, a multimodal feature extraction module and a prediction classification module;

[0174] The image segmentation module 303 is used to perform element segmentation on the user image sequence according to the user description category to obtain a multi-dimensional segmented image;

[0175] The multimodal feature extraction module 304 is used to input the multi-dimensional segmented image and the user voice data into the multimodal feature extraction module for feature extraction respectively to obtain multi-dimensional user features;

[0176] The prediction and classification module 305 is used to classify and predict the multi-dimensional user features to obtain corresponding multi-dimensional risk labels;

[0177] The risk assessment module 306 is used to perform a comprehensive risk assessment based on the multi-dimensional risk tags to obtain the risk result of the target user.

[0178] It should be emphasized that in order to further ensure the privacy and security of the target user's video data, the above-mentioned target user's video data can also be stored in a blockchain node.

[0179] Based on the above-mentioned user description-based risk assessment device 300, risk label prediction is performed through a risk prediction model, which can improve the efficiency of risk assessment. At the same time, assessing the user's risk from multiple dimensions can accurately reflect the user's risk and greatly improve the accuracy of user risk assessment.

[0180] In some optional implementations, the image segmentation module 303 includes a semantic segmentation unit, a feature extraction unit, a region proposal unit, an alignment unit, and a multi-task branch unit, wherein:

[0181] The semantic segmentation unit is used to perform semantic segmentation on the user image sequence to obtain a semantic segmentation image;

[0182] The feature extraction unit is used to extract features from the semantic segmentation image to obtain a semantic segmentation feature map;

[0183] The region proposal unit is used to input the semantic segmentation feature map into the region proposal layer to extract the region of interest and obtain a candidate region feature map;

[0184] The alignment unit is used to input the candidate region feature map into the alignment layer to obtain an aligned region of interest feature map;

[0185] The multi-task branch unit is used to input the aligned region of interest feature map into the multi-task branch layer for classification, bounding box regression and segmentation, and correspondingly obtain a target user description category, a target bounding box and a target instance mask image, where the target instance mask image is a multi-dimensional segmented image.

[0186] In some optional implementations, the multimodal feature extraction module 304 includes an input submodule, a user feature extraction submodule, a scene feature extraction submodule, a visual feature extraction submodule, a speech feature extraction submodule, and an output submodule, wherein:

[0187] The input submodule is used to input the multi-dimensional segmented image into the user feature extraction unit, the scene feature extraction unit and the visual feature extraction unit respectively, and input the user voice data into the voice feature extraction unit;

[0188] The user feature extraction submodule is used to extract features from the multi-dimensional segmented image through the user feature extraction unit to obtain user attribute features;

[0189] The scene feature extraction submodule is used to extract features from the multi-dimensional segmented image through the scene feature extraction unit to obtain scene description features;

[0190] The visual feature extraction submodule is used to extract features from the multi-dimensional segmented image through the visual feature extraction unit to obtain image visual features;

[0191] The speech feature extraction submodule is used to extract features from the user speech data through the speech feature extraction unit to obtain speech features;

[0192] The output submodule is used to combine the user attribute features, the scene description features, the image visual features and the voice features into the multi-dimensional user features for output.

[0193] In some optional implementations of this embodiment, the user feature extraction unit includes a face feature extraction layer, a behavior feature extraction layer, and an expression feature extraction layer; and the user feature extraction submodule is further used for:

[0194] Inputting the multi-dimensional segmented image into the face feature extraction layer to obtain a face feature map;

[0195] Inputting the multi-dimensional segmented image into the behavior feature extraction layer to obtain a user behavior feature graph;

[0196] Inputting the multi-dimensional segmented image into the expression feature extraction layer to obtain a user expression feature map;

[0197] The face feature map, the user behavior feature map and the user expression feature map are output as user attribute features.

[0198] In some optional implementations, the prediction and classification module includes prediction and classification units whose number of dimensions corresponds to the multi-dimensional user features, and is used to:

[0199] Input the multi-dimensional user features into the corresponding prediction classification unit to perform class prediction, and obtain the risk label corresponding to each dimensional user feature;

[0200] The risk labels are combined and output to obtain multi-dimensional risk labels.

[0201] In some optional implementations, the risk assessment module 306 includes a scoring submodule, a level determination submodule, and a calculation submodule, wherein:

[0202] The scoring submodule is used to call the preset risk assessment rules to score each risk tag in the multi-dimensional risk tags to obtain a scoring result;

[0203] The level determination submodule is used to determine the risk level of each risk tag based on the scoring result;

[0204] The calculation submodule is used to perform weighted calculation on the risk levels to obtain the risk result of the target user.

[0205] In some optional implementations, the risk assessment device 300 based on user description further includes a training module for:

[0206] Obtaining historical user image sets and corresponding voice data sets of all users as sample sets, wherein each piece of data in the sample set is marked with a real risk label of the user;

[0207] The pre-constructed initial risk prediction model is trained based on the sample set and the true risk label to obtain a trained final risk prediction model.

[0208] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0209] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0210] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0211] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the risk assessment method based on user description, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0212] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as running the computer-readable instructions of the risk assessment method based on the user description.

[0213] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0214] This embodiment implements the steps of the risk assessment method based on user description in the above embodiment when the processor executes computer-readable instructions stored in the memory. Risk label prediction is performed through the risk prediction model, which can improve the efficiency of risk assessment. At the same time, the user's risk is evaluated from multiple dimensions, which can accurately reflect the user's risk and greatly improve the accuracy of user risk assessment.

[0215] The present application also provides another implementation mode, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor so that the at least one processor executes the steps of the risk assessment method based on user description as described above, and predicts risk labels through a risk prediction model, which can improve the efficiency of risk assessment. At the same time, the user's risk can be assessed from multiple dimensions, which can accurately reflect the user's risk and greatly improve the accuracy of user risk assessment.

[0216] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0217] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A risk assessment method based on user description, characterized in that: The steps include: Acquire video data of a target user, and extract a user image sequence and user voice data from the video data; Inputting the user image sequence and the user voice data into a trained risk prediction model, wherein the risk prediction model includes an image segmentation module, a multimodal feature extraction module, and a prediction classification module; By means of the image segmentation module, the user image sequence is segmented into elements according to the user description categories to obtain a multi-dimensional segmented image; Inputting the multi-dimensional segmented image and the user voice data into the multi-modal feature extraction module to perform feature extraction respectively to obtain multi-dimensional user features; The multi-dimensional user features are classified and predicted by the prediction and classification module to obtain corresponding multi-dimensional risk labels; A comprehensive risk assessment is performed based on the multi-dimensional risk tags to obtain the risk results of the target user.

2. The risk assessment method based on user description according to claim 1, characterized in that: The image segmentation module includes a semantic segmentation unit, a feature extraction layer, a region proposal layer, an alignment layer and a multi-task branch layer; the step of performing element segmentation on the user image sequence according to the user description category through the image segmentation module to obtain a multi-dimensional segmented image includes: Performing semantic segmentation on the user image sequence by the semantic segmentation unit to obtain a semantic segmentation image; Performing feature extraction on the semantic segmentation image to obtain a semantic segmentation feature map; Inputting the semantic segmentation feature map into the region proposal layer to extract the region of interest, and obtaining a candidate region feature map; Inputting the candidate region feature map into the alignment layer to obtain an aligned region of interest feature map; The aligned region of interest feature map is input into the multi-task branch layer for classification, bounding box regression and segmentation, and a target user description category, a target bounding box and a target instance mask image are obtained accordingly. The target instance mask image is a multi-dimensional segmented image.

3. The risk assessment method based on user description according to claim 1, characterized in that: The multimodal feature extraction module includes a user feature extraction unit, a scene feature extraction unit, a visual feature extraction unit and a speech feature extraction unit; The step of inputting the multi-dimensional segmented image and the user voice data into the multi-modal feature extraction module for feature extraction to obtain multi-dimensional user features comprises: Input the multi-dimensional segmented image into the user feature extraction unit, the scene feature extraction unit and the visual feature extraction unit respectively, and input the user voice data into the voice feature extraction unit; Extracting features from the multi-dimensional segmented image by the user feature extraction unit to obtain user attribute features; Extracting features from the multi-dimensional segmented image by the scene feature extraction unit to obtain scene description features; Extracting features from the multi-dimensional segmented image by the visual feature extraction unit to obtain image visual features; Extracting features from the user's voice data by the voice feature extraction unit to obtain voice features; The user attribute features, the scene description features, the image visual features and the voice features are combined into the multi-dimensional user features for output.

4. The risk assessment method based on user description according to claim 3 is characterized in that: The user feature extraction unit includes a face feature extraction layer, a behavior feature extraction layer and an expression feature extraction layer; the step of extracting features from the multi-dimensional segmented image by the user feature extraction unit to obtain user attribute features includes: Inputting the multi-dimensional segmented image into the face feature extraction layer to obtain a face feature map; Inputting the multi-dimensional segmented image into the behavior feature extraction layer to obtain a user behavior feature graph; Inputting the multi-dimensional segmented image into the expression feature extraction layer to obtain a user expression feature map; The face feature map, the user behavior feature map and the user expression feature map are output as user attribute features.

5. The risk assessment method based on user description according to claim 1, characterized in that: The prediction and classification module includes prediction and classification units with the same number of dimensions as the multi-dimensional user features; and the step of performing classification and prediction on the multi-dimensional user features through the prediction and classification module to obtain corresponding multi-dimensional risk labels includes: Input the multi-dimensional user features into the corresponding prediction classification unit to perform class prediction, and obtain the risk label corresponding to each dimensional user feature; The risk labels are combined and output to obtain multi-dimensional risk labels.

6. The risk assessment method based on user description according to claim 1, characterized in that: The step of performing a comprehensive risk assessment based on the multi-dimensional risk tags to obtain the risk result of the target user includes: Calling preset risk assessment rules to score each risk tag in the multi-dimensional risk tags to obtain a scoring result; Determine the risk level of each risk tag based on the scoring result; The risk levels are weighted and calculated to obtain the risk result of the target user.

7. The risk assessment method based on user description according to claim 1, characterized in that: Before the step of inputting the user image sequence and the user voice data into the trained risk prediction model, the method further includes: Obtaining historical user image sets and corresponding voice data sets of all users as sample sets, wherein each piece of data in the sample set is marked with a real risk label of the user; The pre-constructed initial risk prediction model is trained based on the sample set and the true risk label to obtain a trained final risk prediction model.

8. A risk assessment device based on user description, characterized in that: include: An acquisition module, used to acquire video data of a target user, and extract a user image sequence and user voice data from the video data; An input module, used to input the user image sequence and the user voice data into a trained risk prediction model, wherein the risk prediction model includes an image segmentation module, a multimodal feature extraction module and a prediction classification module; An image segmentation module is used to perform element segmentation on the user image sequence according to the user description category to obtain a multi-dimensional segmented image; A multimodal feature extraction module, used for inputting the multi-dimensional segmented image and the user voice data into the multimodal feature extraction module for feature extraction respectively, to obtain multi-dimensional user features; A prediction and classification module, used to classify and predict the multi-dimensional user features to obtain corresponding multi-dimensional risk labels; The risk assessment module is used to perform a comprehensive risk assessment based on the multi-dimensional risk tags to obtain the risk results of the target user.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the risk assessment method based on user description as described in any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the risk assessment method based on user description according to any one of claims 1 to 7 are implemented.

Images