Risk prediction method and device, equipment and storage medium

Abstract

The invention provides a risk prediction method and device, equipment and a storage medium, and relates to the field of artificial intelligence, in particular to the field of large models. The risk prediction method comprises the following steps: acquiring multi-modal complaint data, wherein the multi-modal complaint data comprises first text data and image data; inputting the image data into an image classification model to obtain a classification result of the image data; obtaining second text data corresponding to the image data according to the classification result; the second text data is used for representing the content of the image data; obtaining a fusion feature vector according to the first text data and the second text data; the fusion feature vector is input into a classifier, the type of the multi-modal complaint data is output, and the type of the multi-modal complaint data is used for representing whether the multi-modal complaint data has an abnormal risk or not. According to the invention, the accuracy of risk prediction of complaint data can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a risk prediction method, apparatus, device, and storage medium. Background Art

[0002] With the rapid development of the internet industry, online platforms are able to more conveniently provide users with a variety of online services and products. For payment transactions, users file complaints against merchants, creating complaint tickets. Typical reasons for complaints include transaction disputes with merchants, fund errors, and merchants engaging in illegal or irregular activities. Complaint tickets contain complaint data in various formats, including the user's written complaint text and accompanying images.

[0003] In related technologies, complaint forms need to be classified to identify illegal and irregular activities in the black and gray industries. Complaint form classification generally includes steps such as data preprocessing, feature extraction, model selection, model training, model evaluation, and model application. These steps are interconnected and together constitute a complete complaint form classification process. The feature extraction step extracts features relevant to the classification task from the preprocessed data. For example, text features can be extracted based on statistical features of word frequency, distributed representations based on word vectors, or pre-trained language models. The purpose of feature extraction is to convert text data into numerical vectors with fixed dimensions to facilitate subsequent processing by classification models. However, the accuracy of complaint form classification still needs to be further improved. Summary of the Invention

[0004] The present application provides a risk prediction method, apparatus, device and storage medium, which can help improve the accuracy of risk prediction of complaint data.

[0005] In a first aspect, an embodiment of the present application provides a risk prediction method, comprising:

[0006] Acquire multimodal complaint data, where the multimodal complaint data includes first text data and image data;

[0007] Inputting the image data into an image classification model to obtain a classification result of the image data;

[0008] According to the classification result, obtaining second text data corresponding to the image data; the second text data is used to represent the content of the image data;

[0009] Obtaining a fused feature vector according to the first text data and the second text data;

[0010] The fused feature vector is input into a classifier, and the type of the multimodal complaint data is output. The type of the multimodal complaint data is used to characterize whether there is an abnormal risk in the multimodal complaint data.

[0011] In a second aspect, an embodiment of the present application provides a risk prediction device, comprising:

[0012] a first acquiring unit, configured to acquire multimodal complaint data, wherein the multimodal complaint data includes first text data and image data;

[0013] An image classification model, configured to input the image data and obtain a classification result of the image data;

[0014] A second acquiring unit is configured to acquire second text data corresponding to the image data according to the classification result; the second text data is used to represent the content of the image data;

[0015] a third acquiring unit, configured to acquire a fused feature vector according to the first text data and the second text data;

[0016] A classifier is used to input the fused feature vector and output the type of the multimodal complaint data, where the type of the multimodal complaint data is used to characterize whether the multimodal complaint data has an abnormal risk.

[0017] Optionally, the second acquiring unit is specifically configured to:

[0018] If the classification result is a conversation record image, extracting the conversation text data in the image data; wherein the image data corresponding to the conversation record image type includes the conversation text data between the complained party and the complaining party;

[0019] The conversation text data is determined as the second text data.

[0020] Optionally, the second acquiring unit is specifically configured to:

[0021] If the classification result is not the conversation record image, inputting the image data and the prompt word into a generation model and outputting a text completion of the prompt word;

[0022] The text completion is determined as the second text data.

[0023] Optionally, the classification result includes a payment success picture, a conversation record picture, a QR code picture or a real-scene picture.

[0024] Optionally, the method further includes a fourth obtaining unit, configured to:

[0025] Obtaining entity information of relevant entities based on the multimodal complaint data; wherein the relevant entities include at least one of the complainant, the complained-against, the complained-against order, and the complaint form;

[0026] Acquire third text data according to the entity information;

[0027] The third acquisition unit is specifically configured to:

[0028] The fused feature vector is obtained according to the first text data, the second text data and the third text data.

[0029] Optionally, the fourth acquiring unit is further configured to:

[0030] Obtaining a text description corresponding to the category value of the multimodal complaint data;

[0031] determining the text description as fourth text data;

[0032] The third acquisition unit is specifically configured to:

[0033] The fused feature vector is obtained according to the first text data, the second text data, the third text data and the fourth text data.

[0034] Optionally, the third acquiring unit is specifically configured to:

[0035] concatenating the first text data, the second text data, the third text data, and the fourth text data to obtain long text data;

[0036] The long text data is input into a pre-trained embedded model to obtain the fused feature vector.

[0037] Optionally, the third acquiring unit is specifically configured to:

[0038] Inputting the first text data, the second text data, the third text data, and the fourth text data into a pre-trained embedded model respectively to obtain embedded vectors corresponding to the first text data, the second text data, the third text data, and the fourth text data respectively;

[0039] The fused feature vector is obtained according to the embedded vectors respectively corresponding to the first text data, the second text data, the third text data and the fourth text data.

[0040] Optionally, the third acquiring unit is specifically configured to:

[0041] Input the embedded vectors of the first text data, the second text data, the third text data, and the fourth text data into the attention mechanism model, and output the weights corresponding to the first text data, the second text data, the third text data, and the fourth text data respectively;

[0042] According to the weights corresponding to the first text data, the second text data, the third text data and the fourth text data, respectively, the embedded vectors corresponding to the first text data, the second text data, the third text data and the fourth text data are weighted averaged to obtain the fused feature vector.

[0043] Optionally, the embedded vector is determined based on the first layer output vector and the last layer output vector of the embedded model.

[0044] Optionally, a model training unit is also included to:

[0045] Obtaining a complaint data sample set, the complaint data sample set including at least one text data sample and a true label of whether each text data sample has an abnormality risk;

[0046] Obtaining a feature vector for each of the text data samples;

[0047] Inputting the feature vector of at least one of the text data samples into an initial classifier respectively, and outputting a predicted probability that at least one of the text data samples has an abnormal risk;

[0048] Determining a target loss function based on the predicted probability and the true label of at least one of the text data samples; wherein the target loss function is used to constrain the prediction accuracy to be greater than a preset value, and the number of samples without abnormal risk in at least one of the text data samples is the largest;

[0049] The parameters of the initial classifier are updated according to the target loss function to obtain the trained classifier.

[0050] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method in the first aspect.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the method in the first aspect.

[0052] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which enable a computer to execute the method in the first aspect.

[0053] In a sixth aspect, an embodiment of the present application provides a computer program, which enables a computer to execute the method in the first aspect.

[0054] In an embodiment of the present application, the image data in the multimodal complaint data is input into an image classification model to obtain a classification result of the image data. Then, based on the classification result of the image data, second text data representing the content of the image data is obtained. Then, based on the first text data and the second text data in the multimodal complaint data, a fused feature vector of the multimodal complaint data is obtained. The fused feature vector is input into a classifier to obtain the type of the multimodal complaint data. Since the classifier classifies the complaint data based on the fused feature vector obtained from the text data and image data of the multimodal complaint data, it is possible to use both text data and image data to predict whether the complaint data has an abnormal risk, which is conducive to improving the accuracy of risk prediction of the complaint data. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0056] Figure 1 A schematic diagram of an application scenario of the solution of an embodiment of the present application;

[0057] Figure 2 is a schematic flow chart of a risk prediction method according to an embodiment of the present application;

[0058] Figure 3 A schematic diagram of a conversation record image provided in an embodiment of this application;

[0059] Figure 4 A schematic diagram of an evidence picture provided for an embodiment of the present application;

[0060] Figure 5 This is a schematic diagram of feature diffusion for customer complaint forms according to an embodiment of the present application;

[0061] Figure 6 A schematic diagram of fusing multiple texts in an embodiment of the present application;

[0062] Figure 7 A schematic flowchart of the model training provided in the embodiment of the present application;

[0063] Figure 8 A schematic flow chart of risk prediction provided in an embodiment of the present application;

[0064] Figure 9 is a schematic block diagram of a risk prediction device according to an embodiment of the present application;

[0065] Figure 10 is a schematic block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0067] It should be understood that in the embodiments of the present application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information.

[0068] In the description of this application, unless otherwise specified, "at least one" means one or more, and "plurality" means two or more than two. In addition, "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0069] It should also be understood that the first, second, etc. descriptions appearing in the embodiments of the present application are only for illustration and distinction of the description objects, and there is no order. They do not represent any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0070] It should also be understood that the specific features, structures, or characteristics associated with the embodiments in the specification are included in at least one embodiment of the present application. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0071] In addition, the terms "include" and "have" and any variations thereof are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, product, or device.

[0072] The embodiments of the present application are applied to the field of artificial intelligence technology.

[0073] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also encompasses the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0074] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0075] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0076] The embodiments of the present application may involve natural language processing (NLP) in artificial intelligence technology, which is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can enable effective communication between humans and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology generally includes text processing, semantic understanding, machine translation, robot question answering, knowledge graphs and other technologies.

[0077] The embodiments of the present application may involve computer vision (CV) technology in artificial intelligence technology. Computer vision is a science that studies how to make machines "see". More specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify, monitor and measure targets, and further performs graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0078] The embodiments of the present application may also relate to machine learning (ML) in artificial intelligence technology. ML is a multi-disciplinary subject that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning generally include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formulaic learning.

[0079] The embodiments of the present application may also relate to a pre-training model in artificial intelligence technology. The pre-training model is also called a cornerstone model or a large model, which refers to a deep neural network (DNN) with a large number of parameters. It is trained on a large amount of unlabeled data, and the function approximation ability of the large-parameter DNN is used to enable the PTM to extract common features from the data. After fine-tuning, efficient parameter fine-tuning (PEFT), prompt-tuning and other technologies, it is suitable for downstream tasks. Therefore, the pre-training model can achieve ideal results in a few-shot or zero-shot scenario. According to the data modality processed, PTM can be divided into language models (ELMO, BERT, GPT), visual models (swin-transformer, ViT, V-MOE), speech models (VALL-E), multimodal models (ViBERT, CLIP, Flamingo, Gato), etc., where a multimodal model refers to a model that establishes feature representations of two or more data modalities. Pre-trained models are important tools for outputting artificial intelligence generated content (AIGC) and can also serve as a universal interface for connecting multiple specific task models.

[0080] In related technologies, it is necessary to classify complaint forms to identify illegal and irregular activities in the black and gray industries. Classifying complaint forms generally includes the following steps 1) to 6):

[0081] 1) Data Preprocessing: Before classifying complaint tickets, the raw data needs to be preprocessed. For text data, this can include removing irrelevant information (such as special symbols and punctuation), text cleaning (such as removing stop words and correcting spelling errors), and text segmentation (breaking the text into words or phrases). For numerical data, preprocessing can include filling missing values, handling outliers, and data standardization or normalization. The goal of data preprocessing is to convert the raw data into a format suitable for input into the classification model.

[0082] 2) Feature Extraction: Extract features relevant to the classification task from the preprocessed data. For text features, this can be based on statistical features such as word frequency, distributed representations based on word vectors, or contextual representations based on pretrained language models. For numerical features, the raw values ​​can be used directly or feature combinations and selections can be performed. The goal of feature extraction is to convert text and numerical data into fixed-dimensional numerical vectors for subsequent classification model processing.

[0083] 3) Model Selection: Select an appropriate model based on the specific requirements of the complaint classification task and the characteristics of the data. Existing classification models include traditional machine learning models (such as logistic regression, support vector machines, decision trees, and random forests) and deep learning models (such as convolutional neural networks, recurrent neural networks, and Transformers). The goal of model selection is to find an algorithm that achieves good performance on the complaint classification task.

[0084] 4) Model Training: The selected classification model is trained using labeled complaint data. During model training, the model learns how to correctly classify complaints based on the input feature vectors. The training process involves hyperparameter adjustment, regularization, and loss function optimization to improve model performance on both the training and validation sets.

[0085] 5) Model Evaluation: After training is complete, the model is evaluated using the test set. Common evaluation metrics include accuracy, precision, recall, and F1 score. This evaluation helps us understand the model's generalization ability on unknown data and whether it suffers from overfitting or underfitting.

[0086] 6) Model Application: Apply the trained classification model to actual complaint handling scenarios. Newly received complaints can be automatically classified using the model, improving the efficiency and accuracy of complaint handling.

[0087] The above steps are interconnected to form a complete complaint classification process. However, the above steps do not utilize the image data features in the complaint data when classifying the complaint, resulting in low accuracy in complaint classification.

[0088] In view of this, the embodiments of the present application provide a risk prediction method, apparatus, device and storage medium, which can help improve the accuracy of risk prediction of complaint data.

[0089] Specifically, multimodal complaint data can be obtained, which includes first text data and image data; the image data is input into an image classification model to obtain a classification result of the image data; second text data corresponding to the image data is obtained based on the classification result; the second text data is used to represent the content of the image data; a fused feature vector is obtained based on the first text data and the second text data; the fused feature vector is input into a classifier to output the type of the multimodal complaint data, and the type of the multimodal complaint data is used to characterize whether there is an abnormal risk in the multimodal complaint data.

[0090] In an embodiment of the present application, the image data in the multimodal complaint data is input into an image classification model to obtain a classification result of the image data. Then, based on the classification result of the image data, second text data representing the content of the image data is obtained. Then, based on the first text data and the second text data in the multimodal complaint data, a fused feature vector of the multimodal complaint data is obtained. The fused feature vector is input into a classifier to obtain the type of the multimodal complaint data. Since the classifier classifies the complaint data based on the fused feature vector obtained from the text data and image data of the multimodal complaint data, it is possible to use both text data and image data to predict whether the complaint data has an abnormal risk, which is conducive to improving the accuracy of risk prediction of the complaint data.

[0091] The embodiments of the present application can be used to predict the risk of complaint tickets for various businesses, such as complaint tickets for payment transactions. For example, a complaint ticket can be predicted to determine whether it contains abnormal risks, thereby facilitating more efficient detection of illicit and gray industries. If a complaint ticket is predicted to contain abnormal risks, the corresponding business may be exploited by illicit and gray industries for real-time illegal and irregular activities, and the corresponding complaint ticket entity is black. If a complaint ticket is predicted to contain no abnormal risks, the corresponding business falls within the normal business scope, and the corresponding complaint ticket entity is white.

[0092] Figure 1 A schematic diagram showing an application scenario of an embodiment of the present application.

[0093] like Figure 1 As shown, the application scenario involves a server 1 and a terminal device 2, and the terminal device 2 can communicate data with the server 1 through a communication network. The server 1 can be a background server of the terminal device 2.

[0094] Exemplarily, the terminal device 2 may refer to a device that has a rich range of human-computer interaction methods, has the ability to access the Internet, is typically equipped with various operating systems, and has strong processing capabilities. The terminal device may be, but is not limited to, a smartphone, tablet computer, portable laptop computer, desktop computer, wearable device, vehicle-mounted device, or other terminal device. Optionally, in an embodiment of the present application, the terminal device 2 is installed with various client applications (Application, APP) that support user complaints, including but not limited to dating apps, shopping apps, financial apps, game apps, information apps, etc.

[0095] Server 1 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server can also become a node in the blockchain.

[0096] The server may be one or more servers. When there are multiple servers, at least two servers are used to provide different services, and / or at least two servers are used to provide the same service, such as providing the same service in a load balancing manner, which is not limited in this embodiment of the present application.

[0097] The terminal device and the server can be connected directly or indirectly via wired or wireless communication, and this application does not limit this. This application does not limit the number of servers or terminal devices. The solution provided in this application can be completed by the terminal device independently, by the server independently, or by the terminal device and the server in combination, and this application does not limit this.

[0098] It should be understood that Figure 1 This is only an illustrative description and does not specifically limit the application scenarios of the embodiments of this application. For example, Figure 1 One terminal device and one server are shown as an example, but other numbers of terminal devices and servers may actually be included, and this application does not impose any limitation on this.

[0099] The following describes the technical solutions of the embodiments of the present application in detail through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0100] Figure 2 2 is a schematic flow chart of a risk prediction method 200 according to an embodiment of the present application. The method 200 can be executed by any electronic device with data processing capabilities. For example, the electronic device can be implemented as a server or a terminal device. Figure 1 The server 1 or terminal device 2 in this application is not limited to this. Figure 2 As shown, method 200 includes steps 210 to 240 .

[0101] 210. Acquire multimodal complaint data, where the multimodal complaint data includes first text data and image data.

[0102] For example, for payment transaction services, multimodal complaint data can be complaint information reported by the complainant regarding a payment transaction (such as an order) with the complainee, and can include complaint data in different modalities such as complaint text data and evidence images. Evidence images are an example of image data. The complainant can be a user, and the complainee can be a merchant, but this application does not limit this.

[0103] Optionally, the text data may include numerical data, such as the transaction amount involved in the order.

[0104] Optionally, the image data may include various types of pictures. Exemplarily, based on the content of the image data, the image data may include but is not limited to pictures of successful payment, conversation record pictures, QR code pictures, and real-life pictures.

[0105] 220, input the image data into the image classification model to obtain a classification result of the image data.

[0106] Specifically, before step 220, the image classification model can be trained based on the image data in the collected complaint data. After the training is completed, the image data obtained in step 210 is input into the image classification model to obtain the classification result of the image data.

[0107] Exemplarily, the image classification model training process includes data labeling, data preprocessing, feature extraction, model selection, model training, and model evaluation. In the data labeling step, the image data in the collected complaint data is labeled, and each image is assigned a correct category label. Optionally, the label of the image data corresponds to the type of image data, for example, including but not limited to successful payment images, conversation record images, QR code images, real-life photos, etc. This step is very critical because the performance of the model depends on high-quality labeled data. Optionally, professional data labeling tools or platforms can be used for labeling to improve the efficiency and accuracy of data labeling. In the data preprocessing step, the labeled image data is preprocessed to facilitate subsequent model training. Exemplarily, preprocessing operations may include but are not limited to image scaling, cropping, flipping, rotation, brightness and contrast adjustment, etc. Optionally, the image data can also be divided into a training set, a validation set, and a test set to facilitate subsequent model training and model evaluation. In the feature extraction step, features relevant to the classification task are extracted from the preprocessed image data. These features can be hand-crafted features, such as the scale-invariant feature transform (SIFT) and histogram of oriented gradients (HOG), or automatically learned features based on deep learning models. The goal of feature extraction is to convert image data into numerical vectors of fixed dimensions for subsequent processing by the classification model. In the model selection step, an appropriate classification model can be selected based on the specific requirements of the image classification task and the characteristics of the data. Existing classification models include traditional machine learning models (such as support vector machines and random forests) and deep learning models (such as convolutional neural networks and residual networks). The goal of model selection is to find an algorithm that achieves good performance on the image classification task. In the model training step, the selected classification model is trained using the training set. During the training process, the model learns how to correctly classify images based on the input feature vectors. The training process may involve hyperparameter adjustment, regularization, and loss function optimization to improve model performance on the training and validation sets. After model training is complete, you can evaluate the model using the test set. Common evaluation metrics include accuracy, precision, recall, and F1 score. By evaluating the model, you can understand its generalization ability on the location data and whether it has overfitting or underfitting issues.

[0108] The image classification model trained based on the above process can automatically identify the classification results of the image data in the complaint data. Optionally, the image data classification results include but are not limited to one of the following: payment success images, conversation record images, QR code images, and real-life images.

[0109] 230. Acquire second text data corresponding to the image data according to the classification result; the second text data is used to represent the content of the image data.

[0110] Exemplarily, for image data with different classification results, different methods may be used to obtain the second text data representing the content of the image data.

[0111] In some embodiments, if the classification result is a conversation record image, the conversation text data in the image data is extracted and determined as the second text data. The image data corresponding to the conversation record image includes the conversation text data between the complained party and the complaining party.

[0112] Specifically, after the image classification model identifies the category of the image data in the multimodal complaint data, the conversation text data (i.e., conversation text) can be extracted from the conversation record image.

[0113] As an implementation approach, attention-based sequence-to-sequence models can be used to extract the conversational text data. Attention-based sequence-to-sequence models are an end-to-end text recognition method that uses convolutional neural networks (CNNs) or recurrent neural networks (RNNs) to extract image features and then maps these features to character sequences through an attention mechanism. This model demonstrates good recognition performance in a variety of scenarios.

[0114] As another possible implementation method, optical character recognition (OCR) technology may be used to convert the conversation text in the image data of the conversation record picture category into editable text data, and then perform text extraction to obtain the conversation text data.

[0115] For example, for Figure 3 For conversation log images, we can extract the text from them using an attention-based sequence-to-sequence model or optical character recognition (OCR) technology: "It'll be ready soon; Please wait half an hour for the results; All my money has been deducted." This extracted text can be used as part of the text data in the multimodal complaint data. For content in the image data that is unrelated to the conversation text, such as "To receive a timely response from the other party, go turn on the notification settings," there is no need to perform text extraction.

[0116] In some embodiments, if the classification result is not a conversation record image, the image data and the prompt word are input into the generative model, a text completion of the prompt word is output, and the text completion is determined as the second text data. Optionally, the classification result in this case includes a payment success image, a QR code image, or a real-life image.

[0117] Specifically, after the image classification model identifies the category of image data in the multimodal complaint data, for images that do not belong to the conversation record image type, the image data and prompt words can be input into the generation model to output the text completion of the prompt words. Among them, the prompt words are used to guide the model to generate responses. For example, they can contain task-related information and context to help the model better understand the requirements and generate correct outputs. For example, the prompt words can be designed to contain descriptions of questions or topics to help the model generate correct answers.

[0118] For example, for pictures that do not belong to the conversation record picture type, although they contain text, the text is not structured enough, or some pictures do not have text but contain rich information that can help us qualitatively analyze the complaint form. In this case, this information can be extracted in a structured manner, that is, a generative model is used to extract the information contained in the picture.

[0119] Exemplarily, the generative model can be a large model of graph-based text. The large model of graph-based text is a multimodal pre-training model based on deep learning, which can process various data types such as text, images and speech. By pre-training on a large amount of unlabeled data, the model learns rich feature representations, thereby achieving excellent performance in various natural language processing, computer vision and speech recognition tasks. The large model of graph-based text is guided by prompt words and uses the large amount of knowledge and language rules that the pre-training model has learned to complete the task of generating text completion for prompt words. Exemplarily, the large model of graph-based text can be a large language model (LargeLanguageModel, LLM), including but not limited to the PaLM model, GPT model, LLaMa model, ChatGLMS model, etc.

[0120] For example, Figure 4 Based on the image data in the complaint data, the prompt word adjustment (prompt tuning) of the image-generated text model can be performed. For example, the instruction can be as follows:

[0121] This is a picture of evidence of a user of a certain payment service complaining about the payee after making a payment. Please summarize the picture from this perspective and think about why the user complained about the payee, what problems the payee had, and does the image contain any information about illegal or irregular behavior?

[0122] For example, Figure 4 For the image data in the image, the image-to-text model can output the following text:

[0123] Based on the image you provided, we didn't find any information indicating any illegal or irregular behavior. The image shows a payment service's complaint and evidence submission page, where a user may have submitted a complaint about the recipient and provided relevant evidence and information, including trip status and route deviations.

[0124] The reasons for users to complain about the payee may include dissatisfaction with the service, deviation from the driving route, and the payee's failure to resolve the problem in a timely manner. These reasons may be related to the payee's service attitude, service quality, itinerary arrangements, etc.

[0125] However, based on the images you provided, it is impossible to determine whether the recipient has engaged in any illegal or irregular activities. This requires further investigation and verification of relevant evidence before a conclusion can be reached.

[0126] If you feel uneasy about the recipient's behavior or suspect fraud, you can contact the payment service customer service and file a complaint. At the same time, to protect your rights, you should retain relevant evidence and information so that you can provide assistance when needed.

[0127] Based on this, the information in the image data can be extracted in a structured manner. Since the purpose of the embodiment of the present application is to qualitatively classify the complaint data and find complaint data with abnormal risks (such as illegal and irregular behaviors), the image-to-text model can also be used to help conduct a round of pre-screening. The generated text can directly provide information on whether there are abnormal risks (such as illegal and irregular behaviors).

[0128] In some embodiments, entity information of related entities may be obtained based on the multimodal complaint data, wherein the related entities include at least one of the complainant, the complained-against party, the complained-against order, and the complaint form. Based on the entity information, third text data may be obtained.

[0129] Specifically, the information of complaint data (such as complaint tickets) itself is limited, and this limited information will result in the performance of the model not being very good. Based on this, the multimodal complaint data can be diffused first, and then the features can be refined to obtain richer features. Exemplarily, multimodal complaint data is related to entities, such as the object of the complaint, the object of the complaint, the order complained, the complaint ticket, and other entities. Therefore, the multimodal complaint data can be diffused to related entities, such as the object of the complaint, the object of the complaint, the complaint order, etc., and then the information of each entity can be refined as the third text data. Optionally, the relevant information of each entity can be stored in a database.

[0130] Figure 5A schematic diagram of feature diffusion for a customer complaint is shown. First, entity diffusion can be performed on the customer complaint to obtain four related entities, namely the customer complaint itself, the original order, the merchant, and the user. Among them, the customer complaint is an example of a complaint, the original order is an example of the complained order, the merchant is an example of the complained object, and the user is an example of the complained object. Figure 5 Performing three-dimensional business diffusion on customer complaints yields the following: 1) complaint text and supporting images; 2) pre-selected complaint types and subtypes; 3) complaint amount and refund request amount. Performing two-dimensional business diffusion on original orders yields the following: 1) order payment amount and payment time; 2) order payment scenario, transaction channel, and payment method. Performing three-dimensional business diffusion on merchants yields the following: 1) whether the merchant is a key account (KA), whether it has been deemed trustworthy by customer service, and the duration of the aforementioned creation; 2) the number of complaints received in the past k days, the complaint rate, and the number of times the merchant has been labeled as blacklisted by customer service; 3) the transaction amount, credit card transaction percentage, and photo album QR code transaction percentage in the past k days. Performing two-dimensional business diffusion on users yields the following: 1) number of days since app registration; 2) the number of user complaints received in the past k days, the total amount, and the number of user complaints deemed blacklisted. Therefore, diverging feature search based on entities associated with complaint tickets can yield more features related to complaint tickets, which can be used as tertiary text data.

[0131] In some embodiments, a text description corresponding to the category value of the multimodal complaint data may also be obtained; and the text description may be determined as the fourth text data.

[0132] For example, the category features of the complaint form obtained through feature diffusion are stored in the database as category values. The corresponding Chinese meaning of the category value cannot be directly included in the third text data described above. In view of this, the embodiment of the present application adds a text description corresponding to the category value of the multimodal complaint data and determines this text description as the fourth text data.

[0133] For example, the transaction channel of a complaint ticket is a category feature, and its category value and corresponding Chinese meaning are shown in Table 1 below:

[0134] Table 1

[0135]

[0136] As a specific example, based on manual review experience, when the transaction channel is "scanning the QR code", the complaint form has a higher probability of being involved in illegal and irregular activities. Obviously, during manual review, it is the Chinese meaning corresponding to the category value "scanning the QR code" rather than the category value "4" itself that provides more information. Based on this, the Chinese meaning corresponding to the category value of the multimodal complaint data can be regarded as a kind of text data of the multimodal complaint data, that is, the fourth text data mentioned above. Exemplarily, all category values ​​of the multimodal complaint data can be converted into text descriptions corresponding to the Chinese meaning. For example, if the category value of the transaction channel is 4, then the text description is "the transaction channel is scanning the QR code."

[0137] 240. Obtain a fused feature vector based on the first text data and the second text data.

[0138] Specifically, since the second text data is obtained based on the image data in the multimodal complaint data, the fused feature vector can fuse the features of the text data and image data in the multimodal complaint data, and characterize the correlation between the text data and image data in the multimodal complaint data.

[0139] In some embodiments, after the third text data is obtained, a fused feature vector may be obtained based on the first text data, the second text data, and the third text data.

[0140] In some embodiments, after the third text data and the fourth text data are obtained, a fused feature vector may be obtained based on the first text data, the second text data, the third text data, and the fourth text data.

[0141] For example, see Figure 6 , the text currently obtained includes user complaint text, text converted from category values, conversation text, text generated by the large graph-based text model, and other text. In this case, user complaint text is the text contained in the multimodal complaint data, which is an example of the first text data; text converted from category values ​​is the descriptive text added according to the category value of the multimodal complaint data, which is an example of the fourth text data; conversation text is the text extracted from image data of the conversation record picture type, which is an example of the second text data; text generated by the large graph-based text model is the descriptive text generated for image data that is not a conversation record picture type, which is another example of the second text data. Exemplarily, other texts can be texts obtained by feature diffusion, such as merchant names, product names, etc., which is an example of the third text data.

[0142] In some embodiments, the first text data, the second text data, the third text data, and the fourth text data may be concatenated to obtain long text data; and the long text data may be input into a pre-trained embedded model to obtain a fused feature vector.

[0143] Specifically, multiple text data can be spliced ​​together to form a long text data, and then the long text data can be input into a pre-trained embedded model to obtain a fused feature vector of the entire long text data. The fused feature vector can be an embedding vector. Exemplarily, the embedded model can also be called a text representation model, which is used to represent the input text as an embedded vector. Exemplarily, the embedded model can include a word vector model, such as a Word2Vec model, a GloVe model, or a pre-trained language model, such as BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (Robustly Optimized BERT Pre-training Approach) and other models.

[0144] BERT is a Transformer-based pre-trained language model that learns rich language representations through pre-training on large amounts of unlabeled text data. It is then fine-tuned on specific tasks to improve the model's performance on various natural language processing tasks. RoBERTa is a BERT-based pre-trained language model that improves and optimizes BERT to enhance its performance on natural language processing tasks.

[0145] In some embodiments, the first text data, the second text data, the third text data, and the fourth text data can be respectively input into a pre-trained embedded model to obtain the embedded vectors corresponding to the first text data, the second text data, the third text data, and the fourth text data; and a fused feature vector can be obtained based on the embedded vectors corresponding to the first text data, the second text data, the third text data, and the fourth text data.

[0146] That is, each text data can be input into the pre-trained embedded model to obtain the corresponding embedded vectors for each text data, and then a fused feature vector can be obtained based on the embedded vectors of each text data. For example, the embedded vectors of each text data can be averaged to capture the overall information of multiple text data and obtain a fused feature vector.

[0147] In some embodiments, the embedded vector of each text data in the first text data, the second text data, the third text data, and the fourth text data, or the long text data is determined based on the first layer output vector and the last layer output vector of the embedded model.

[0148] For example, the first layer output and the last layer output of the embedded model can be averaged as the final representation vector of the input text data.

[0149] The embodiment of the present application determines the final representation vector of the input text data by using the first and last outputs of the embedded model, which can help retain both low-level and high-level features of the input text data. Specifically, the first few neural network layers in a deep neural network generally capture low-level features of the input data, such as vocabulary and grammatical information, while the later neural network layers generally capture high-level features, such as semantic and contextual information. Therefore, by averaging the first and last outputs of the embedded model, the feature information of the first and last outputs can be retained at the same time, so that the final representation vector provides a richer text representation vector that includes both low-level features and high-level features, thereby improving the performance of downstream models. Furthermore, by determining the final representation vector of the input text data by using the first and last outputs of the embedded model, the final representation vector can be made more robust to slight changes in the model's internal weights, which is beneficial for improving the model's generalization ability on different tasks and data sets. In addition, by determining the final representation vector of the input text data by using the first and last outputs of the embedded model, the model's expressive power can be increased, thereby reducing the risk of overfitting.

[0150] Optionally, the embedded vectors of the first text data, the second text data, the third text data and the fourth text data can be input into the attention mechanism model, and the weights corresponding to the first text data, the second text data, the third text data and the fourth text data can be output respectively; and according to the weights corresponding to the first text data, the second text data, the third text data and the fourth text data respectively, the embedded vectors corresponding to the first text data, the second text data, the third text data and the fourth text data respectively are weighted averaged to obtain a fused feature vector.

[0151] Specifically, the embedded vectors (such as embedding) of the first text data, the second text data, the third text data and the fourth text data can be input into an attention mechanism model, so that the attention mechanism model learns the weights corresponding to each text data. Then, the embedded vectors of the first text data, the second text data, the third text data and the fourth text data are weighted averaged according to the weights corresponding to each text data in the first text data, the second text data, the third text data and the fourth text data to obtain a final fusion feature vector. Therefore, the embodiment of the present application can adaptively capture the relationship between multiple text data, thereby learning the weight of each text data according to the relationship between multiple text data, and by weighting the embedded vectors corresponding to each text data according to the weight of each text data, it is possible to highlight the information of important text data and reduce the influence of unimportant text data.

[0152] In other embodiments, a weight can be assigned to each of the first text data, the second text data, the third text data, and the fourth text data based on the importance of the text data or other indicators, and then the embedded vectors corresponding to each text data are weighted averaged according to the weight corresponding to each text data to obtain the final fused feature vector.

[0153] In other embodiments, the embedded vectors (such as embedding) of the first text data, the second text data, the third text data, and the fourth text data can also be input into a neural network model to allow the neural network model to learn the fusion feature vectors of these text data. Exemplarily, the neural network model can include but is not limited to a fully connected layer, a convolutional neural network, a recurrent neural network, etc., without limitation. Therefore, the embodiment of the present application can capture the nonlinear relationship between multiple text data, thereby learning and obtaining the fusion feature vectors of multiple text data based on the nonlinear relationship between the multiple text data.

[0154] 250, input the fused feature vector into the classifier, and output the type of the multimodal complaint data. The type of the multimodal complaint data is used to characterize whether the multimodal complaint data has abnormal risks.

[0155] Specifically, the classifier can map the input fusion feature vector to one of the given categories, thereby realizing the prediction of the type of multimodal complaint data. Exemplarily, the types of multimodal complaint data include the presence of abnormal risk (i.e., the corresponding complaint data is black) and the absence of abnormal risk (i.e., the corresponding complaint data is white). As an example, when the predicted probability of the presence of abnormal risk is greater than 0.5, the multimodal complaint data has an abnormal risk; when the predicted probability of the presence of abnormal risk is less than 0.5, the multimodal complaint data does not have an abnormal risk. Therefore, the embodiment of the present application can realize the prediction of whether the multimodal complaint data has an abnormal risk by inputting the fusion feature vector into the classifier.

[0156] Therefore, the embodiment of the present application obtains the classification result of the image data by inputting the image data in the multimodal complaint data into the image classification model, and then obtains the second text data used to represent the content of the image data based on the classification result of the image data. Then, based on the first text data and the second text data in the multimodal complaint data, a fused feature vector of the multimodal complaint data is obtained, and the fused feature vector is input into the classifier to obtain the type of the multimodal complaint data. Since the classifier classifies the complaint data based on the fused feature vector obtained by jointly obtaining the text data and image data of the multimodal complaint data, it can be realized to use the text data and image data to jointly predict whether the complaint data has abnormal risks, which is conducive to improving the accuracy of the risk prediction of the complaint data.

[0157] In some embodiments, see Figure 7 , the classifier can be optimized according to the following steps 310 to 350 to obtain a trained classifier.

[0158] 310 , obtaining a complaint data sample set, where the complaint data sample set includes at least one complaint data sample and a true label of whether each complaint data sample has an abnormality risk.

[0159] For example, complaint data can be annotated, with each complaint data item assigned a true label indicating whether the corresponding sample has an abnormal risk, to obtain a complaint data sample set. For example, when a sample has an abnormal risk, it is a black sample; when a sample does not have an abnormal risk, it is a white sample. Alternatively, a professional data annotation tool or platform can be used for annotation, which is not limited in this application.

[0160] 320, obtain the feature vector of each complaint data sample.

[0161] Exemplarily, when the complaint data sample includes text data, the text data sample can be input into a trained embedded model to obtain a feature vector for the complaint data sample. Optionally, when the complaint data sample also includes image data, the text data corresponding to the image data can be obtained, and the feature vector for the complaint data sample can be obtained based on the text data corresponding to the image data and the text data included in the complaint data sample. Specifically, the process for obtaining this feature vector is similar to the process for obtaining the aforementioned fused feature vector, and reference can be made to the relevant description above.

[0162] 330 , input the feature vectors of at least one complaint data sample into the initial classifier, and output a predicted probability that the at least one complaint data sample has an abnormality risk.

[0163] For example, the model parameters in the initial classifier may be initial values. The initial classifier outputs the probability that the complaint data sample has an abnormal risk based on the feature vector of the input complaint data sample.

[0164] 340 Determine a target loss function based on the predicted probability and true label of at least one of the complaint data samples; wherein the target loss function is used to constrain the prediction accuracy to be greater than a preset value and to have the largest number of samples without abnormal risk.

[0165] Specifically, the objective function is used to constrain the classifier's prediction accuracy to be greater than a preset value. At the same time, the higher the proportion of input complaint data judged by the classifier as having no abnormal risk (i.e., being white), the better. This helps the model pay more attention to misclassified samples while ensuring the model's prediction accuracy, thereby reducing the probability of the model predicting complaint data without abnormal risk as having abnormal risk.

[0166] For example, the objective function can be determined according to the following formula (1):

[0167]

[0168] Among them, f(y pred ,y true ) represents the objective function, y pred The predicted probability that the complaint data sample output by the classifier has abnormal risk, y true is the true label of the complaint data sample, λ is the adjustable parameter of the classifier, and n is the number of complaint data samples.

[0169] In the above formula (1), when y pred The closer it is to 0, the greater the probability that the complaint data sample does not have abnormal risk (i.e., it is white). Based on this, the first term in formula (1) The smaller is, the greater the white ratio of the classifier is. At the same time, the second item in (1) The denominator is the total amount of white judgments made by the classifier, and the numerator is the total amount of black judged as white, so the second term The smaller , the higher the accuracy of the classifier. Therefore, when the objective function is the above formula (1), it can ensure that the prediction accuracy of the classifier is greater than the preset value, and the classifier has the highest proportion of white judgments for the input complaint data samples, that is, the number of white judgments for the complaint data samples is the largest.

[0170] As a specific example, the accuracy of the classifier can be set to be greater than or equal to 0.995. In this case, the objective function can be expressed as follows:

[0171]

[0172] The same symbols in formula (2) and formula (1) represent the same or similar meanings.

[0173] 350, the parameters of the initial classifier are updated according to the target loss function to obtain a trained classifier.

[0174] Specifically, because the objective function constrains the classifier's prediction accuracy to be greater than a preset value and to maximize the number of samples without abnormal risk, the initial classifier's parameters are updated according to the objective loss function, ensuring that the classifier achieves the maximum number of samples without abnormal risk while maintaining a prediction accuracy greater than a preset value. For example, the classifier automatically reaches a preset accuracy, such as 99.5%, without requiring adjustment of the threshold, while ensuring that the model determines the highest proportion of input complaint data as having no abnormal risk (i.e., white).

[0175] Figure 8 FIG1 shows a schematic flow chart of a risk prediction method provided by an embodiment of the present application. Figure 8 As shown, for a multimodal data sheet, basic numerical features, category features, text features, and image features can be obtained. For example, basic numerical features include the transaction amount of the original order, category features include the payment scenario and transaction channel of the original order, text features include the complaint text and the name of the merchant in the complaint, and image features include the evidence images in the complaint sheet.

[0176] For category features, the corresponding category feature values ​​may be converted into text features. For example, an artificial description of the Chinese meaning of the category feature value may be added as the corresponding text feature.

[0177] The text features derived from the type features and the text features contained in the complaint ticket can be fed into the text representation model together to obtain embedded vectors for the text features. Optionally, the text features corresponding to the type features and the text features contained in the complaint ticket can be fused together for annotation. The annotated text is then used to fine-tune the text representation model using supervised learning.

[0178] Regarding image features, one possible implementation method is to extract the conversation text data from a conversation record image (such as a screenshot) based on the image classification results. Alternatively, the conversation text data can be extracted using OCR technology or a sequence-to-sequence model based on an attention mechanism, which is not limited in this application.

[0179] Another possible implementation method is to use the large model to generate descriptive text for the image. For example, instructions and images can be input into the large model, and the instructions can guide the large model to summarize the image and extract image information.

[0180] The extracted conversation text data and the generated image description text can be fed together into a text representation model to obtain an embedded vector for the input text. Optionally, the conversation text data and the generated image description text can be fused together for annotation, and the annotated text can be used to fine-tune the text representation model.

[0181] Finally, the basic numerical features, the embedded vectors of the text features from the complaint form output by the text representation model, the embedded vectors of the text features converted from the category features, the embedded vectors of the conversation text data from the conversation record type image, and the embedded vectors of the description text from the generated image are fused and input into the classifier to perform risk prediction on the complaint form.

[0182] In one embodiment of the present application, the category features of the complaint form are converted into text data, and the various category information in the complaint form (such as complainant information, complaint object information, etc.) is converted into text form to facilitate subsequent text processing and fusion. Then, the converted category text data is fused with the text data of the complaint form itself (such as complaint content, complaint reason, etc.) to form a multi-text dataset. On this basis, large models (such as RoBERTa, GPT, etc.) can be used to model the multi-text dataset and extract the key information and features in the complaint form.

[0183] Optionally, to further improve the accuracy of the model, the embodiments of the present application improve the loss function of the downstream classification model, so that the trained classification model can automatically achieve a specific accuracy (such as 99.5%) without adjusting the threshold. Specifically, by introducing a new loss function, the model can pay more attention to misclassified samples during training, thereby improving the model's performance on the test set.

[0184] In addition, the embodiment of the present application also classifies the images in the complaint form, uses OCR technology or a sequence-to-sequence model based on an attention mechanism to extract the conversation text content classified as conversation record images, and fuses the extracted conversation text content with other text features for comprehensive modeling. Therefore, the embodiment of the present application can further enrich the information of the complaint form and improve the prediction accuracy of the model. Optionally, in order to better utilize the image information in the complaint form, the embodiment of the present application uses fine-tuning of large model instructions to summarize and describe the complaint images and extract key information from the complaint images. Specifically, by fine-tuning the large model, it is able to generate descriptive text for the complaint images, thereby converting the image information into text information. Then, the extracted image text information is fused with other text features for comprehensive modeling.

[0185] Finally, the embodiment of the present application performs weighted adaptive fusion of features obtained from different dimensions. By fusing features extracted from multiple dimensions such as text features and image features, a comprehensive fused feature vector can be obtained to represent the overall information of the complaint form. On this basis, the fused feature vector is classified using a classification model, thereby achieving the cleaning and classification of the complaint form. The embodiment of the present application has the advantages of high accuracy, high efficiency, and a high degree of automation. It can meet the needs of the financial industry for complaint form processing and has broad application prospects.

[0186] The specific embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, a variety of simple modifications can be made to the technical solution of the present application, and these simple modifications all fall within the scope of protection of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner unless there is any contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, and as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0187] It should also be understood that in the various method embodiments of the present application, the order of the sequence numbers of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. It should be understood that these sequence numbers can be interchanged where appropriate, so that the embodiments of the present application described can be implemented in an order other than those shown or described.

[0188] The above describes the method embodiment of the present application in detail. Figures 9 and 10 , describe in detail the device embodiments of the present application.

[0189] Figure 9 FIG is a schematic block diagram of the risk prediction device 10 according to an embodiment of the present application. Figure 9 As shown, the apparatus 10 may include a first acquisition unit 11 , an image classification model 12 , a second acquisition unit 13 , a third acquisition unit 14 and a classifier 15 .

[0190] A first acquiring unit 11 is configured to acquire multimodal complaint data, where the multimodal complaint data includes first text data and image data;

[0191] An image classification model 12, configured to input the image data and obtain a classification result of the image data;

[0192] A second acquiring unit 13 is configured to acquire second text data corresponding to the image data according to the classification result; the second text data is used to represent the content of the image data;

[0193] a third acquiring unit 14, configured to acquire a fused feature vector based on the first text data and the second text data;

[0194] The classifier 15 is configured to input the fused feature vector and output the type of the multimodal complaint data, where the type of the multimodal complaint data is used to characterize whether the multimodal complaint data has an abnormal risk.

[0195] Optionally, the second acquiring unit 13 is specifically configured to:

[0196] If the classification result is a conversation record image, extracting the conversation text data in the image data; wherein the image data corresponding to the conversation record image type includes the conversation text data between the complained party and the complaining party;

[0197] The conversation text data is determined as the second text data.

[0198] Optionally, the second acquiring unit 13 is specifically configured to:

[0199] If the classification result is not the conversation record image, inputting the image data and the prompt word into a generation model and outputting a text completion of the prompt word;

[0200] The text completion is determined as the second text data.

[0201] Optionally, the classification result includes a payment success picture, a conversation record picture, a QR code picture or a real-scene picture.

[0202] Optionally, the method further includes a fourth obtaining unit, configured to:

[0203] Obtaining entity information of relevant entities based on the multimodal complaint data; wherein the relevant entities include at least one of the complainant, the complained-against, the complained-against order, and the complaint form;

[0204] Acquire third text data according to the entity information;

[0205] The third obtaining unit 14 is specifically configured to:

[0206] The fused feature vector is obtained according to the first text data, the second text data and the third text data.

[0207] Optionally, the fourth acquiring unit is further configured to:

[0208] Obtaining a text description corresponding to the category value of the multimodal complaint data;

[0209] determining the text description as fourth text data;

[0210] The third obtaining unit 14 is specifically configured to:

[0211] The fused feature vector is obtained according to the first text data, the second text data, the third text data and the fourth text data.

[0212] Optionally, the third obtaining unit 14 is specifically configured to:

[0213] concatenating the first text data, the second text data, the third text data, and the fourth text data to obtain long text data;

[0214] The long text data is input into a pre-trained embedded model to obtain the fused feature vector.

[0215] Optionally, the third obtaining unit 14 is specifically configured to:

[0216] Inputting the first text data, the second text data, the third text data, and the fourth text data into a pre-trained embedded model respectively to obtain embedded vectors corresponding to the first text data, the second text data, the third text data, and the fourth text data respectively;

[0217] The fused feature vector is obtained according to the embedded vectors respectively corresponding to the first text data, the second text data, the third text data and the fourth text data.

[0218] Optionally, the third obtaining unit 14 is specifically configured to:

[0219] Input the embedded vectors of the first text data, the second text data, the third text data, and the fourth text data into the attention mechanism model, and output the weights corresponding to the first text data, the second text data, the third text data, and the fourth text data respectively;

[0220] According to the weights corresponding to the first text data, the second text data, the third text data and the fourth text data, respectively, the embedded vectors corresponding to the first text data, the second text data, the third text data and the fourth text data are weighted averaged to obtain the fused feature vector.

[0221] Optionally, the embedded vector is determined based on the first layer output vector and the last layer output vector of the embedded model.

[0222] Optionally, a model training unit is also included to:

[0223] Obtaining a complaint data sample set, the complaint data sample set including at least one text data sample and a true label of whether each text data sample has an abnormality risk;

[0224] Obtaining a feature vector for each of the text data samples;

[0225] Inputting the feature vector of at least one of the text data samples into an initial classifier respectively, and outputting a predicted probability that at least one of the text data samples has an abnormal risk;

[0226] Determining a target loss function based on the predicted probability and the true label of at least one of the text data samples; wherein the target loss function is used to constrain the prediction accuracy to be greater than a preset value, and the number of samples without abnormal risk in at least one of the text data samples is the largest;

[0227] Parameters of the initial classifier are updated according to the target loss function to obtain the trained classifier.

[0228] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 9 The device 10 shown can execute the above method embodiment, and the above and other operations and / or functions of each module in the device 10 are respectively for implementing the corresponding processes in the above method 200, which will not be repeated here for the sake of brevity.

[0229] The apparatus of the embodiment of the present application is described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, or can be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in conjunction with its hardware.

[0230] Figure 10 It is a schematic block diagram of an electronic device 30 provided in an embodiment of the present application.

[0231] like Figure 10As shown, the electronic device 30 may include:

[0232] The memory 31 and the processor 32 are configured to store computer programs and transmit the program code to the processor 32. In other words, the processor 32 can call and run the computer program from the memory 31 to implement the method in the embodiment of the present application.

[0233] For example, the processor 32 may be configured to execute the above method embodiments according to instructions in the computer program.

[0234] In some embodiments of the present application, the processor 32 may include but is not limited to:

[0235] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0236] In some embodiments of the present application, the memory 31 includes but is not limited to:

[0237] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0238] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to implement the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0239] like Figure 10 As shown, the electronic device 30 may further include:

[0240] The transceiver 33 may be connected to the processor 32 or the memory 31 .

[0241] The processor 32 can control the transceiver 33 to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 33 may include a transmitter and a receiver. The transceiver 33 may further include an antenna, and the number of antennas may be one or more.

[0242] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0243] The present application also provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment. In other words, the present application also provides a computer program product containing instructions, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment.

[0244] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0245] It is understandable that in the specific implementation of this application, when the above embodiments of this application are applied to specific products or technologies and involve relevant data such as user information, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0246] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0247] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0248] Modules described as separate components may or may not be physically separate, and components displayed as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected based on actual needs to achieve the purpose of the present embodiment. For example, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module.

[0249] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A risk prediction method, characterized in that: include: Acquire multimodal complaint data, where the multimodal complaint data includes first text data and image data; Inputting the image data into an image classification model to obtain a classification result of the image data; According to the classification result, obtaining second text data corresponding to the image data; the second text data is used to represent the content of the image data; Obtaining a fused feature vector according to the first text data and the second text data; The fused feature vector is input into a classifier, and the type of the multimodal complaint data is output. The type of the multimodal complaint data is used to characterize whether there is an abnormal risk in the multimodal complaint data.

2. The method according to claim 1, characterized in that The step of obtaining second text data corresponding to the image data according to the classification result includes: If the classification result is a conversation record image, extracting the conversation text data in the image data; wherein the image data corresponding to the conversation record image type includes the conversation text data between the complained party and the complaining party; The conversation text data is determined as the second text data.

3. The method according to claim 2, characterized in that The step of obtaining second text data corresponding to the image data according to the classification result includes: If the classification result is not the conversation record image, inputting the image data and the prompt word into a generation model and outputting a text completion of the prompt word; The text completion is determined as the second text data.

4. The method according to claim 2 or 3, characterized in that The classification results include payment success pictures, conversation record pictures, QR code pictures or real-scene pictures.

5. The method according to claim 1, wherein Also includes: Obtaining entity information of relevant entities based on the multimodal complaint data; wherein the relevant entities include at least one of the complainant, the complained-against, the complained-against order, and the complaint form; Acquire third text data according to the entity information; The step of obtaining a fused feature vector based on the first text data and the second text data includes: The fused feature vector is obtained according to the first text data, the second text data and the third text data.

6. The method according to claim 5, characterized in that Also includes: Obtaining a text description corresponding to the category value of the multimodal complaint data; determining the text description as fourth text data; The step of obtaining the fused feature vector according to the first text data, the second text data, and the third text data includes: The fused feature vector is obtained according to the first text data, the second text data, the third text data and the fourth text data.

7. The method according to claim 6, characterized in that The first text data, the second text data, the third text data, and the fourth text data are used to obtain the fused feature vector, including: concatenating the first text data, the second text data, the third text data, and the fourth text data to obtain long text data; The long text data is input into a pre-trained embedded model to obtain the fused feature vector.

8. The method according to claim 6, characterized in that The first text data, the second text data, the third text data, and the fourth text data are used to obtain the fused feature vector, including: Inputting the first text data, the second text data, the third text data, and the fourth text data into a pre-trained embedded model respectively to obtain embedded vectors corresponding to the first text data, the second text data, the third text data, and the fourth text data respectively; The fused feature vector is obtained according to the embedded vectors respectively corresponding to the first text data, the second text data, the third text data and the fourth text data.

9. The method according to claim 6, characterized in that The obtaining the fused feature vector according to the embedded vectors respectively corresponding to the first text data, the second text data, the third text data, and the fourth text data includes: Input the embedded vectors of the first text data, the second text data, the third text data, and the fourth text data into the attention mechanism model, and output the weights corresponding to the first text data, the second text data, the third text data, and the fourth text data respectively; According to the weights corresponding to the first text data, the second text data, the third text data and the fourth text data, respectively, the embedded vectors corresponding to the first text data, the second text data, the third text data and the fourth text data are weighted averaged to obtain the fused feature vector.

10. The method according to claim 8, characterized in that The embedded vector is determined according to the first layer output vector and the last layer output vector of the embedded model.

11. The method according to claim 1, characterized in that Also includes: Obtaining a complaint data sample set, the complaint data sample set including at least one text data sample and a true label of whether each text data sample has an abnormality risk; Obtaining a feature vector for each of the text data samples; Inputting the feature vector of at least one of the text data samples into an initial classifier respectively, and outputting a predicted probability that at least one of the text data samples has an abnormal risk; Determining a target loss function based on the predicted probability and the true label of at least one of the text data samples; wherein the target loss function is used to constrain the prediction accuracy to be greater than a preset value, and the number of samples without abnormal risk in at least one of the text data samples is the largest; Parameters of the initial classifier are updated according to the target loss function to obtain the trained classifier.

12. A risk prediction device, characterized in that: include: a first acquiring unit, configured to acquire multimodal complaint data, wherein the multimodal complaint data includes first text data and image data; An image classification model, configured to input the image data and obtain a classification result of the image data; A second acquiring unit is configured to acquire second text data corresponding to the image data according to the classification result; the second text data is used to represent the content of the image data; a third acquiring unit, configured to acquire a fused feature vector according to the first text data and the second text data; A classifier is used to input the fused feature vector and output the type of the multimodal complaint data, where the type of the multimodal complaint data is used to characterize whether the multimodal complaint data has an abnormal risk.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores instructions, and when the processor executes the instructions, the processor executes the method according to any one of claims 1 to 11.

14. A computer storage medium, characterized in that Used to store a computer program, the computer program comprising instructions for executing the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The method comprises a computer program code, which, when executed by an electronic device, causes the electronic device to perform the method according to any one of claims 1 to 11.

Images