Multi-mode-based account type identification method and apparatus, and electronic device

Abstract

The invention relates to the technical field of computers, provides an account type identification method and device based on multiple modes and electronic equipment, and is used for improving the security of payment based on a two-dimensional code. According to the method, resource transfer texts and image information contained in two-dimensional code pictures are fully utilized, an evaluation text generated by resource transfer operation is combined, multi-modal fusion features are obtained, accounts corresponding to the two-dimensional code pictures are divided into a plurality of account subsets based on the fusion features of the two-dimensional code pictures, aggregation analysis of the accounts is achieved, and the user experience is improved. By using the text and image information of the two-dimensional code, the dependence on resource transfer data is reduced, and the recall rate and accuracy of an account are improved; and the account type of the account subset containing the current account is determined through the keyword in the text, so that the account with the resource transfer risk can be effectively identified, corresponding measures can be taken in time, and the security of two-dimensional code payment is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and electronic device for identifying account types based on multimodality. Background Art

[0002] In resource transfer scenarios, QR codes are often used as credentials for resource transfer, such as payment codes, ride codes, etc. As a graphic code, QR codes are easily copied and misappropriated, which may lead to resource transfer risks.

[0003] At present, when transferring resources through QR codes, the relevant technology mainly identifies risky accounts based on the resource data associated with the QR code (such as payment amount, payment goods, number of payments, payment method, etc.) and risk account clustering analysis. First, abnormal accounts are identified by performing feature statistics on the resource data, and then the abnormal accounts and risk accounts with existing labels are clustered and analyzed to determine the risk type of the abnormal accounts.

[0004] However, the above methods are usually based on high-dimensional statistical features constructed based on professional experience for identification, that is, the transferred data needs to reach a certain scale in order to accurately identify abnormal accounts. When the amount of resource data is small, it is easy to miss or misjudge, which is high-risk.

[0005] On the other hand, risk account clustering analysis is based on supervised learning. Since the labels of supervised learning are pre-labeled manually, it can only identify accounts of known risk types and has poor scalability.

[0006] Therefore, improving the security of resource transfer in the form of QR codes has become an urgent issue to be addressed in resource transfer. Summary of the invention

[0007] The embodiments of the present application provide a multimodal account type identification method, device and electronic device for improving the security of resource transfer based on QR codes.

[0008] On the one hand, an embodiment of the present application provides a multi-modal account type identification method, including:

[0009] Scan the target two-dimensional code image to be processed, obtain the resource transfer text contained in the target two-dimensional code image, and obtain the evaluation text generated by the resource transfer operation based on the target two-dimensional code image;

[0010] Based on the resource transfer text and the evaluation text, obtaining text features of the target two-dimensional code image, and extracting image features of the target two-dimensional code image;

[0011] Based on the text features and image features of the target QR code image and in combination with the text features and image features of the processed multiple historical QR code images, the current account corresponding to the target QR code image and the historical accounts corresponding to the multiple historical QR code images are divided into multiple account subsets;

[0012] Based on the keywords included in the resource transfer text and the evaluation text, the account type of the account subset including the current account is determined.

[0013] On the other hand, an embodiment of the present application provides an account type identification device based on multimodality, including:

[0014] A text acquisition module, used to scan the target two-dimensional code image to be processed, obtain the resource transfer text contained in the target two-dimensional code image, and obtain the evaluation text generated by the resource transfer operation based on the target two-dimensional code image;

[0015] A feature extraction module, used to obtain text features of the target two-dimensional code image based on the resource transfer text and the evaluation text, and to extract image features of the target two-dimensional code image;

[0016] An account division module, for dividing the current account corresponding to the target QR code image and the historical accounts corresponding to the multiple historical QR code images into multiple account subsets based on the text features and image features of the target QR code image and the text features and image features of the multiple historical QR code images that have been processed;

[0017] The type determination module is used to determine the account type of the account subset including the current account based on the resource transfer text and the keywords included in the evaluation text.

[0018] Optionally, the account division module is specifically used to:

[0019] For the target two-dimensional code image and each of the multiple historical two-dimensional code images, feature splicing is performed on text features and image features of the two-dimensional code images to obtain fusion features;

[0020] Based on the fusion features of each QR code image, determine the type similarity between the accounts corresponding to every two QR code images;

[0021] Based on the determined similarities of each type, the accounts corresponding to the QR code images are divided into multiple account subsets.

[0022] Optionally, the account division module is specifically used to:

[0023] The account corresponding to each QR code image is taken as a node, and the type similarity between the accounts corresponding to every two QR code images is taken as the weight of the edge between the nodes to obtain an undirected graph;

[0024] Each node is regarded as a community, and the community modularity is calculated based on the weight of the edge between the nodes;

[0025] Based on the community modularity, the undirected graph is segmented to obtain multiple subgraphs, and each subgraph serves as an account subset.

[0026] Optionally, the account division module is specifically used to:

[0027] Calculate the initial community modularity based on the weight of the edge between every two nodes in each node;

[0028] Merging the nodes at least once, and recalculating the community modularity based on the weights of the edges between the new nodes after each merger, until the community modularity meets a preset condition;

[0029] Each community that meets the preset condition is regarded as a subgraph.

[0030] Optionally, the account division module is specifically used to:

[0031] Based on the determined similarities of each type, clustering the accounts corresponding to each of the QR code images to obtain multiple clusters;

[0032] Each cluster obtained is regarded as an account subset.

[0033] Optionally, the type determination module is specifically used to:

[0034] Extracting each keyword from the resource transfer text and the evaluation text, and comparing each keyword with a preset historical word library;

[0035] Based on each comparison result, the account type of the account subset including the current account is determined.

[0036] Optionally, the type determination module is further used to:

[0037] Obtaining candidate accounts whose confidence level is lower than a preset confidence threshold from the account subset including the current account;

[0038] Based on the accumulated historical account labels, the account type of each candidate account is obtained;

[0039] Based on the account type of each candidate account, the structure of the undirected graph or the cluster is adjusted to update the existing account type, wherein the undirected graph or the cluster is used to divide the account subsets.

[0040] Optionally, the feature extraction module is further used for:

[0041] Based on a preset text length, a text is truncated to obtain multiple text segments for text feature extraction;

[0042] Filtering the text to remove information irrelevant to resource transfer;

[0043] Based on a preset text format, the text is formatted.

[0044] On the other hand, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned multimodal-based account type identification method are implemented.

[0045] On the other hand, an embodiment of the present application provides a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by an electronic device, implement the steps of the above-mentioned multimodal-based account type identification method.

[0046] On the other hand, an embodiment of the present application provides a computer program product, comprising a computer program, which, when executed by an electronic device, implements the steps of the above-mentioned multimodal-based account type identification method.

[0047] The beneficial effects of the embodiments of the present application are as follows:

[0048] In the multimodal account type identification method, device and electronic device provided in the embodiments of the present application, by making full use of the resource transfer text and image information contained in the target QR code image itself for identification, the reliance on high-dimensional statistical features of resource transfer data is reduced, which helps to improve the accuracy of account subset division; at the same time, based on the text features obtained from the resource transfer text and the evaluation text generated by the resource transfer operation based on the target QR code image, and the image features extracted from the target QR code image itself, the current account corresponding to the target QR code image and the historical accounts corresponding to multiple historical QR code images are divided into account subsets, which improves the accuracy of cluster analysis compared with single modal features; further, the account type of the account subset containing the current account is determined by the keywords in the resource transfer text and the evaluation text, and the account subset is qualitatively characterized compared with the manually labeled type labels, which can mine accounts of unknown risk types and improve the security of QR code-based payment.

[0049] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 An application scenario diagram applicable to the embodiment of the present application;

[0052] Figure 2 An overall architecture diagram of multimodal account type identification provided in an embodiment of the present application;

[0053] Figure 3 A flowchart of a multi-modal account type identification method provided in an embodiment of the present application;

[0054] Figure 4A A schematic diagram of a QR code image for resource transfer provided in an embodiment of the present application;

[0055] Figure 4B A schematic diagram of obtaining resource transfer text provided in an embodiment of the present application;

[0056] Figure 4C A schematic diagram of obtaining evaluation text provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of a text feature extraction process provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of an image feature extraction process provided in an embodiment of the present application;

[0059] Figure 7 A flow chart of the account aggregation analysis method provided in the embodiment of the present application;

[0060] Figure 8 A flow chart of a method for dividing account subsets based on a graph community discovery method provided in an embodiment of the present application;

[0061] Fig.9A An undirected graph constructed based on an account of a QR code image provided in an embodiment of the present application;

[0062] Fig. 9B A schematic diagram of the account subset division result provided in an embodiment of the present application;

[0063] Fig.10 A flow chart of a method for determining an account type of an account subset provided in an embodiment of the present application;

[0064] Fig.11 A schematic diagram for determining the account type provided in an embodiment of the present application;

[0065] Fig.12 A flow chart of a label propagation method provided in an embodiment of the present application;

[0066] Fig.13 A schematic diagram of the label propagation process provided in an embodiment of the present application;

[0067] Fig.14 A structural diagram of a multimodal account type identification device provided in an embodiment of the present application;

[0068] Fig.15 A structural diagram of an electronic device provided in an embodiment of the present application;

[0069] Fig.16 A structural diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0071] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0072] Multimodality: Using multiple forms of information sources to describe the same thing, for example, using images and text to describe a picture at the same time.

[0073] Optical Character Recognition (OCR): A technology for extracting text from images.

[0074] Supervised learning: A machine learning method in which the samples used for model training need to be manually labeled.

[0075] Unsupervised learning: It is also a machine learning method. The samples used in model training do not need to be labeled.

[0076] Graph community discovery: A method to partition a graph into multiple subgraphs, each of which represents a community.

[0077] Community: A group of graph nodes. Nodes in the same community are densely connected, while nodes in different communities are sparsely connected.

[0078] Community modularity: an indicator used to measure the quality of community division results. The larger the community modularity, the more accurate the community division.

[0079] Clustering: A method of classification based on the differences and commonalities between data. Generally, data with commonalities will be classified into the same class (cluster), and data with differences will be classified into different classes (clusters). Commonalities and differences can be measured by similarity.

[0080] Vision Transformer (ViT): An image classification neural network that can be used for image feature extraction and classification.

[0081] BERT model (Bidirectional Encoder Representation from Transformers, BERT): a text prediction neural network that can be used for text feature extraction and classification.

[0082] Contrastive learning: A method for training a neural network, which is used in the embodiments of the present application to train a neural network for calculating the similarity between two accounts.

[0083] Resources: data that can flow between different accounts, including but not limited to funds, currencies, stocks, etc.

[0084] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0085] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify, detect and measure targets, and further performs graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0086] Natural language processing is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0087] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Compared with data mining, which seeks mutual characteristics between big data, machine learning focuses more on algorithm design, allowing computers to automatically "learn" rules from data and use the rules to predict unknown data.

[0088] The following is an overview of the design concepts of the embodiments of the present application.

[0089] At present, when transferring resources through QR codes, the relevant technology mainly identifies the risk type of abnormal accounts based on the resource data associated with the QR code and risk account clustering analysis. However, this method is based on the high-dimensional characteristics of statistical resource data for identification. When the amount of resource data is small, it is easy to produce missed judgments and misjudgments, thereby failing to accurately identify abnormal accounts and causing payment risks. In addition, the risk account clustering analysis is based on supervised learning, which can only identify abnormal accounts of known risk types and has poor scalability.

[0090] In view of this, an embodiment of the present application provides an account type identification method based on multimodality. The method obtains text features of the text modality based on the resource transfer text contained in the target QR code itself and the evaluation text associated with the target QR code, and extracts image features of the image modality from the target QR code itself, and classifies accounts based on the features of these two modalities. Since the text and image information of the target QR code itself is utilized, the dependence on the data generated when resource transfer is performed based on the QR code is reduced, and the accuracy and recall rate of account identification are improved. At the same time, the method qualitatively characterizes the account type of the account subset including the current account based on the keywords in the resource transfer text and the evaluation text corresponding to the target QR code, and can mine accounts of unknown risk types, thereby improving the security of resource transfer based on QR codes.

[0091] It should be noted that the QR codes, evaluation texts, account information, etc. used in the embodiments of this application involve relevant data such as object information. When the above embodiments of this application are applied to specific products or technologies, it is necessary to obtain the object's permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0092] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0093] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes a terminal device 110 (specifically, it may be a terminal device 1101, a terminal device 1102...), and a server 120.

[0094] In an optional implementation, the terminal device 110 and the server 120 may communicate via a wired network or a wireless network.

[0095] In the embodiment of the present application, the terminal device 110 includes but is not limited to a personal computer, a mobile phone, a tablet computer, a notebook, an e-book reader, an intelligent medical device, and a vehicle-mounted terminal, etc., and a payment client is installed on these terminal devices for transferring electronic resources between different accounts, wherein the resource transfer method can be based on a QR code or based on an account. The payment client installed on the terminal device 110 can be a payment software, or a web page, a small program, etc. The server 120 is a backend server of the payment client, or a server specifically used for resource transfer, which is not specifically limited in this application. The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0096] The multimodal account type identification method in the embodiment of the present application can be executed by an electronic device, which may be a terminal device 110 or a server 120, that is, the method can be executed by the terminal device 110 or the server 120 alone, or can be executed jointly by the terminal device 110 and the server 120.

[0097] For example, taking server execution as an example, after receiving the target QR code sent by the terminal device 110, the server-side device 120 identifies the resource transfer text in the target QR code, and obtains the evaluation text generated by the resource transfer operation based on the target QR code from the background log. The text features are extracted through the resource transfer text and the evaluation text, and the image features are extracted from the target QR code. Considering that the types of accounts in the same subset are likely to be the same and similar, therefore, combined with the text features and image features of the processed multiple historical QR codes, the current account corresponding to the target QR code and the historical accounts corresponding to the multiple historical QR codes are subjected to account aggregation analysis to obtain multiple account subsets. Further, based on the resource transfer text and the evaluation text of the target QR code, the account subset containing the current account is qualitatively characterized to determine the account type of the account subset, thereby mining accounts of unknown risk types. When it is determined that the account subset containing the current account is at risk, the resource transfer operation based on the target QR code is stopped, thereby improving the security of resource transfer based on the QR code.

[0098] It should be noted that Figure 1 The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0099] In an embodiment of the present application, when there are multiple servers, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; as disclosed in the embodiment of the present application, the multi-modal account type identification method, the QR code, resource transfer text, evaluation text, account information, etc. involved therein can be saved on the blockchain.

[0100] The following describes the multimodal account type identification method provided by the exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate the understanding of the spirit and principles of the present application, and the implementation methods of the present application are not limited in this regard.

[0101] See also Figure 2 , which is an overall architecture diagram of multimodal account type identification provided in an embodiment of the present application, utilizes a QR code image for resource transfer and an evaluation text generated by a resource transfer operation based on a QR code, and adopts computer vision and natural language processing technology to fuse image and text information, thereby identifying risky accounts when performing resource transfer based on a QR code and exploring potential resource transfer risks.

[0102] In this architecture, it mainly includes data acquisition layer, similarity calculation layer, aggregation mining layer and type qualitative layer. The data acquisition layer is mainly used to obtain QR code images and texts. The text includes resource transfer text obtained by OCR recognition of QR code images and evaluation text corresponding to QR code images. The similarity calculation layer is mainly used to extract image features from QR code images, as well as text features from resource transfer text and complaint text, and fuse the features of image modality and text modality, perform comparative learning based on the fused features, and calculate the type similarity between accounts corresponding to each QR code image; among them, the network for image feature extraction includes but is not limited to Convolutional Neural Networks (CNN), ViT network, etc., and the network for text feature extraction includes but is not limited to Transformer network, BERT network, Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network, etc. The cluster mining layer performs cluster analysis based on the type similarity between accounts, and divides the accounts corresponding to each QR code image into multiple account subsets; among them, account clustering can use cluster analysis methods (such as K-means clustering algorithm, density-based spatial clustering of applications with noise, DBSCAN, etc.), and can also use graph community discovery methods (such as page ranking (PageRank) method, Louvain algorithm, etc.). The type qualitative layer is used to extract keywords from resource transfer texts and complaint texts, and compare the keywords with the preset vocabulary to determine the account type of each account subset; and based on the historically accumulated account type labels, the labels of unknown accounts can be updated and propagated throughout the account structure.

[0103] When performing cluster analysis on accounts, the identified historical QR code images can be introduced. In this way, the multimodal account type identification method provided in the embodiment of the present application can characterize the account corresponding to a single QR code image, and can also characterize the accounts corresponding to batches of QR code images.

[0104] Taking the input of a QR code image as an example, the process of the multimodal account type identification method provided in the embodiment of the present application is as follows: Figure 3 As shown, the process is executed by an electronic device, which may be Figure 1 The terminal device in can also be Figure 1 The process mainly includes the following steps:

[0105] S301: The electronic device scans the target QR code image to be processed, obtains the resource transfer text contained in the target QR code image, and obtains the evaluation text generated by performing the resource transfer operation based on the target QR code image.

[0106] like Figure 4A The figure shows a schematic diagram of a QR code image used for resource transfer, wherein the QR code image includes an image and a text description.

[0107] In order to make full use of the information contained in the QR code itself, after obtaining the target QR code image, the electronic device pre-processes the target QR code image, crops the target QR code image into an image of a preset size and normalizes it so that the overall distribution of the QR code image is consistent. After the pre-processing is completed, the target QR code image is subjected to OCR recognition to extract the resource transfer text contained in the target QR code image.

[0108] For example, Figure 4A After OCR recognition of the target QR code image shown, the resource transfer text obtained is "Scan code to pay ￥200.00 parking space management fee 200￥ property company", such as Figure 4B shown.

[0109] Usually, for QR code images that pose payment risks, the payment recipient may comment on the phenomenon of resource transfer based on the QR code image on the complaint platform. These comments can better reflect whether the account corresponding to the QR code image is reasonable and legal. Therefore, after obtaining the target QR code image, the electronic device can also obtain the evaluation text generated by the resource transfer operation based on the target QR code image from the background log.

[0110] For example, Figure 4C As shown, based on Figure 4A The multiple evaluation texts generated by the resource transfer operation of the target QR code image shown are: "After collecting the property fee every year, they also collect the parking space management fee. Isn't this random charging?", "The random charging phenomenon is too serious", "I personally feel that the charges are unreasonable."

[0111] S302: The electronic device obtains text features of the target two-dimensional code image based on the resource transfer text and the evaluation text, and extracts image features of the target two-dimensional code image.

[0112] After obtaining the resource transfer text and the evaluation text, the electronic device may perform a preprocessing operation on at least one of the resource transfer text and the evaluation text to improve the accuracy of subsequent account recall and classification.

[0113] Optionally, the text preprocessing operation includes at least one of the following operations:

[0114] Truncate a text based on a preset text length to obtain multiple text segments for text feature extraction;

[0115] Filter a text to remove information unrelated to resource transfer (such as: Uniform Resource Locator (URL), emoji, special symbols, etc.);

[0116] Convert the format of a text based on a preset text format (such as: convert English uppercase to lowercase, convert traditional Chinese characters to simplified Chinese characters, etc.).

[0117] Taking review text as an example, the following describes the preprocessing process of the text.

[0118] For example, when the review text is "After collecting property management fees every year, a separate parking space management fee is also charged. Isn't this arbitrary charging?", to avoid the text being too long and exceeding the processing length of the network model and the limit of hardware resources, the text length is set to 30 bytes. However, the review text "After collecting property management fees every year, a separate parking space management fee is also charged. Isn't this arbitrary charging?" exceeds 30 bytes and needs to be truncated. Since each character occupies 2 bytes, therefore, "After collecting property management fees every year, a separate parking space management fee is also charged. Isn't this arbitrary charging?" is split into two text segments: "After collecting property management fees every year, a separate parking space management fee is also charged" and "Isn't this arbitrary charging?".

[0119] Another example, when the review text is "The phenomenon of arbitrary charging is too serious (ˇ_ˇ)", it contains special symbols and needs to be text-filtered to remove the special symbol "(ˇ_ˇ)" to obtain the filtered review text "The phenomenon of arbitrary charging is too serious".

[0120] Another example, when the review text is "Personally, I feel that the charge is not 郃理", it contains the traditional Chinese character "郃". Therefore, the text needs to be format-converted to obtain the review text in simplified Chinese characters "Personally, I feel that the charge is unreasonable".

[0121] After preprocessing the resource transfer text and the review text, extract text features based on the trained network model.

[0122] Taking the use of the BERT model for text feature extraction as an example, such as Figure 5The figure shows a schematic diagram of the text feature extraction process. In the natural language processing process, the words in the text are usually represented by one-dimensional vectors. Therefore, the neural network in the BERT model takes the one-dimensional vectors of each word in the text as input. After a series of complex transformations, it outputs a one-dimensional word vector as the semantic representation of the text to obtain text features. In the transformation process, the value of the text vector is automatically learned by the BERT model to represent the global semantic information of the text and integrate it with the semantic information of a single word. Since the semantic information carried by words appearing in different positions in the text is different, the BERT model adds a different vector to the words in different positions for distinction.

[0123] The target QR code image contains not only text information but also image information, so the electronic device will also extract image features from the target QR code image.

[0124] Take the ViT model for image feature extraction as an example. Figure 6 As shown in the figure, the target QR code image is divided into 3*3 image blocks (patches), and each image block is linearly projected into a vector of fixed length, and then input into the Transformer for encoding to obtain image features. Similarly, similar to text feature extraction, each image block also needs to be added to its position in the target QR code image to obtain complete image features.

[0125] S303: The electronic device divides the current account corresponding to the target QR code image and the historical accounts corresponding to the multiple historical QR code images into multiple account subsets based on the text features and image features of the target QR code image and the text features and image features of the multiple processed historical QR code images.

[0126] Considering that the types of accounts in the same subset are more likely to be the same or similar, we can perform clustering analysis by adding processed historical QR code images. During the clustering analysis, the neural network trained by multimodal comparative learning can calculate the type similarity between the accounts corresponding to each QR code image.

[0127] In one example, a contrastive learning method can be used to train a feature extraction network for images and texts, and a fully connected network can be used to fuse the features of both image and text modalities to obtain a complete neural network for calculating similarity, thereby performing clustering analysis based on similarity.

[0128] See also Figure 7 , which is the implementation flow chart of clustering analysis. The process mainly includes the following steps:

[0129] S3031: For each of the target two-dimensional code images and the multiple historical two-dimensional code images, text features and image features of the two-dimensional code are concatenated to obtain fused features.

[0130] Taking the target QR code image and one of the multiple historical QR code images as an example, after obtaining the image features and text features of the QR code image through multimodal feature extraction, the image features and text features are spliced ​​through multiple fully connected layers to obtain the fusion features of the two modalities of image and text.

[0131] Optionally, the multimodal feature fusion process can be implemented using a contrastive language-image pre-training (CLIP) method based on contrastive text-image pairs.

[0132] S3032: Based on the fusion features of each QR code image, determine the type similarity between the accounts corresponding to every two QR code images.

[0133] Optionally, the type similarity between two accounts can be calculated using cosine similarity. Since each fused feature contains both the image information of the QR code and the text information of the QR code, the accuracy of the similarity calculation can be improved.

[0134] S3033: Based on the determined similarities of each type, the accounts corresponding to each QR code image are divided into multiple account subsets.

[0135] After obtaining the type similarity between the accounts corresponding to each two QR code images, cluster analysis can be performed on each account to explore potential resource transfer risks and improve payment security.

[0136] In one example, a graph community discovery method may be used to perform clustering analysis, including but not limited to PageRank algorithm and Louvain algorithm.

[0137] Taking the Louvain algorithm as an example, see Figure 8 , which is a flow chart of the method for dividing account subsets based on the graph community discovery method, mainly includes the following steps:

[0138] S3033_1: Take the account corresponding to each QR code image as a node, and take the type similarity between the accounts corresponding to every two QR codes as the weight of the edge between the nodes to obtain an undirected graph.

[0139] like Fig.9A As shown in the figure, it is an undirected graph constructed with accounts as nodes and the type similarity between accounts as the weight of the edge between two nodes.

[0140] S3033_2: Treat each node as a community and calculate the community modularity based on the weight of the edges between nodes.

[0141] S3033_3: Based on the community modularity, the undirected graph is segmented to obtain multiple subgraphs, each of which serves as an account subset.

[0142] In the specific implementation, first, the undirected graph is initialized, and the nodes representing each account are regarded as a community. Based on the weight of the edge between each two nodes in each node, the initial community modularity is calculated. Among them, the community modularity is used to measure the quality index of the community division (i.e., account subset division) results. The larger the community modularity, the more accurate the community division. In the embodiment of the present application, the value range of the community modularity is set between [-1,1], and the calculation formula is as follows:

[0143]

[0144] Among them, Q represents the community modularity, m is 1 / 2 of the total network weight, and A i,j represents the weight of the edge between node i and node j, represents the probability of node i being connected to any node, It represents the probability of node j being connected to any node. When node i and node j are in the same community, I(i, j) is 1; when node i and node j are in different communities, I(i, j) is 0.

[0145] Then, each node is merged at least once in an iterative manner. During each merging process, two nodes can be merged into a new node, and the new node is re-used as a community and the edge weights are updated. After each merge, the community modularity is recalculated using Formula 1 based on the edge weights between the merged new nodes. Repeat the above merging process. When the maximum community modularity is obtained and the preset conditions are met, the iteration is stopped, and the final communities are respectively used as subgraphs to complete the division of account subsets.

[0146] by Fig.9A Taking the undirected graph shown in the figure as an example, the division results of account subsets based on the graph community discovery method are as follows Fig. 9B As shown in the figure, the merged nodes contained in each community are output as an account subset. Among them, the connections between nodes in the same community are denser, indicating that the similarity between these nodes is strong and they are more likely to be the same account type, while the connections between nodes in different communities are sparse, indicating that the similarity between these nodes is low and they are more likely to be different account types.

[0147] In one example, cluster analysis can also be used for cluster analysis. Cluster analysis methods include but are not limited to K-means algorithm and DBSCAN algorithm. The clustering process includes: based on the determined similarities of each type, clustering the accounts corresponding to each QR code image to obtain multiple clusters, and taking each obtained cluster as an account subset.

[0148] In the specific implementation, the account corresponding to each QR code image is taken as an initial cluster. According to the type similarity between each account, the two clusters with the smallest type similarity are found, and the two clusters are merged into a new cluster. Furthermore, the average distance between all points in the two new clusters is calculated, and the average distance is used as the similarity between clusters. Then the above operation is repeated until the preset number of iterations is reached or the cluster no longer changes, and each final cluster is taken as an account subset, thereby completing hierarchical clustering. In the clustering process, accounts with high type similarity are divided into the same cluster, and accounts with low type similarity are divided into different clusters.

[0149] S304: Determine the account type of the account subset including the current account based on the keywords included in the resource transfer text and the evaluation text.

[0150] After obtaining each account subset, the account type of each account subset can be qualitatively analyzed to identify accounts with resource transfer risks and improve payment security.

[0151] In one example, a subset of accounts may be characterized based on keywords in the text, such as Fig.10 As shown, it mainly includes the following steps:

[0152] S3041: extracting each keyword from the resource transfer text and the evaluation text, and comparing each keyword with a preset historical vocabulary.

[0153] In one example, a Term Frequency-Inverse Document Frequency (TF-IDF) algorithm or a BERT model may be used to extract keywords from the resource transfer text and the evaluation text, and then the extracted keywords are compared with a preset historical vocabulary to determine whether the keywords are the same as or semantically similar to the keywords in the preset historical vocabulary.

[0154] S3042: Based on each comparison result, determine the account type of the account subset including the current account.

[0155] In one example, the historical words in the preset word library that are compared can be used to characterize the account subset including the current account, thereby determining the account type of the current account. Since the preset historical word library contains historically accumulated risk words, accounts with payment risks can be mined.

[0156] by Figure 4A Take the target QR code picture shown as an example. Fig.11 As shown, assuming that the keywords extracted from the resource transfer text and the evaluation text are "parking space management fee" and "random charges", after comparing with the historical words in the preset historical vocabulary, it is found that "random charges" is a word in the preset historical vocabulary. Therefore, the account type of the account subset including the current account is determined to be "random charges".

[0157] In some embodiments, a label propagation algorithm can also be used to update the account type of unknown accounts to be processed based on the accumulated labels (i.e., risk types) of historical accounts with resource transfer risks, and update them in the entire network structure of the cluster analysis, thereby mining a batch of new accounts with risks. For the specific process, see Fig.12 , mainly includes the following steps:

[0158] S1201: After the electronic device determines the account type of the account subset including the current account, it obtains candidate accounts in the account subset including the current account whose confidence is lower than a preset confidence threshold.

[0159] Optionally, the confidence of an account can be measured by the distance from the point to the center of the subgraph or cluster in which it is located. When the confidence of a point representing an account in a subgraph or cluster is lower than a preset confidence threshold, it indicates that it may be an account of another type, and therefore, it can be regarded as a candidate account of an unknown type.

[0160] S1202: The electronic device obtains the account type of each candidate account based on the accumulated historical account tags.

[0161] In the embodiment of the present application, by introducing the accumulated historical account tags to determine the account type of each candidate account, the tags determined based on the keywords in the text can be expanded to improve the accuracy of the account type identification. The historical account tags can be manually annotated.

[0162] For example, in actual application, object A does not have a private car, but the payment QR code bound to it is used to pay a repair fee to a 4S store. That is, the QR code image of object A is at risk of being "stolen". Therefore, the account of the QR code image bound to object A can be treated as a risk account and its account can be labeled as "stolen".

[0163] S1203: The electronic device adjusts the structure of the undirected graph or cluster based on the account type of each candidate account to update the existing account type.

[0164] Take the example of using undirected graph for cluster analysis, such as Fig.13 As shown in the figure, it is a schematic diagram of the label propagation algorithm process. Assume that the historical account labels are not introduced. Fig.10 The undirected graph is divided into three account subsets, including the account subset containing the current account, whose account type is "random charges", and the account types of the other two account subsets are "anti-money laundering" and "security". There are two candidate accounts in the "random charges" account subset. By introducing historical account labels, it is determined that the account types of these two candidate accounts may be "fraudulent credit card transactions". After propagating this label throughout the undirected graph, the undirected graph is re-divided into four subgraphs, and no more accounts with "fraudulent credit card transaction" risks are discovered.

[0165] The multimodal account type identification method provided in the embodiment of the present application makes full use of the text and image information contained in the QR code image itself, combines the evaluation text generated by the resource transfer operation, obtains multimodal fusion features, and performs cluster analysis based on the fusion features of each QR code image, thereby reducing the dependence on resource transfer data and improving the account recall rate and accuracy; at the same time, an unsupervised method is adopted to determine the account type of the account subset containing the current account through the keywords in the resource transfer text contained in the QR code image and the evaluation text generated by the resource transfer, which can effectively identify accounts with resource transfer risks, so as to take corresponding measures in time to improve the security of QR code payment, and because an unsupervised method is adopted, there is no need to rely on manually marked labels, and various account types can be more flexibly excavated to improve the accuracy of account type identification.

[0166] Based on the same technical concept, an embodiment of the present application provides a structural diagram of an account type identification device based on multimodality, which can implement the above-mentioned account type identification method based on multimodality and achieve the same technical effect.

[0167] See also Fig.14 The account type identification device includes a text acquisition module 1401, a feature extraction module 1402, an account classification module 1403 and a type determination module 1404, wherein:

[0168] The text acquisition module 1401 is used to scan the target two-dimensional code image to be processed, obtain the resource transfer text contained in the target two-dimensional code image, and obtain the evaluation text generated by the resource transfer operation based on the target two-dimensional code image;

[0169] A feature extraction module 1402 is used to obtain text features of the target two-dimensional code image based on the resource transfer text and the evaluation text, and to extract image features of the target two-dimensional code image;

[0170] An account division module 1403 is used to divide the current account corresponding to the target QR code image and the historical accounts corresponding to the multiple historical QR code images into multiple account subsets based on the text features and image features of the target QR code image and the text features and image features of the multiple processed historical QR code images;

[0171] The type determination module 1404 is used to determine the account type of the account subset including the current account based on the keywords included in the resource transfer text and the evaluation text.

[0172] Optionally, the account division module 1403 is specifically used for:

[0173] For each of the target QR code image and the multiple historical QR code images, feature splicing is performed on the text features and image features of the QR code images to obtain fusion features;

[0174] Based on the fusion features of each QR code image, determine the type similarity between the accounts corresponding to every two QR code images;

[0175] Based on the determined similarities of each type, the accounts corresponding to each QR code image are divided into multiple account subsets.

[0176] Optionally, the account division module 1403 is specifically used for:

[0177] The account corresponding to each QR code image is taken as a node, and the type similarity between the accounts corresponding to every two QR code images is taken as the weight of the edge between the nodes to obtain an undirected graph;

[0178] Treat each node as a community and calculate the community modularity based on the weight of the edges between nodes;

[0179] Based on the community modularity, the undirected graph is segmented to obtain multiple subgraphs, each of which serves as an account subset.

[0180] Optionally, the account division module 1403 is specifically used for:

[0181] Based on the weight of the edge between every two nodes in each node, the initial community modularity is calculated;

[0182] Merge each node at least once, and recalculate the community modularity based on the weight of the edge between the new nodes after each merger until the community modularity meets the preset conditions;

[0183] Each community that meets the preset conditions is regarded as a subgraph.

[0184] Optionally, the account division module 1403 is specifically used for:

[0185] Based on the determined similarities of each type, the accounts corresponding to each QR code image are clustered to obtain multiple clusters;

[0186] Each cluster obtained is regarded as an account subset.

[0187] Optionally, the type determination module 1404 is specifically used for:

[0188] Extracting keywords from resource transfer text and evaluation text, and comparing each keyword with a preset historical word library;

[0189] Based on each comparison result, the account type of the account subset including the current account is determined.

[0190] Optionally, the type determination module 1404 is further configured to:

[0191] Obtain candidate accounts whose confidence is lower than a preset confidence threshold in the account subset including the current account;

[0192] Based on the accumulated historical account labels, the account type of each candidate account is obtained;

[0193] Based on the account type of each candidate account, the structure of the undirected graph or cluster is adjusted to update the existing account type, wherein the undirected graph or cluster is used to divide the account subsets.

[0194] Optionally, the feature extraction module 1402 is further used for:

[0195] Based on a preset text length, a text is truncated to obtain multiple text segments for text feature extraction;

[0196] Filter a text to remove information irrelevant to resource transfer;

[0197] Convert a text into a new format based on the preset text format.

[0198] In the multimodal account identification device provided in the embodiment of the present application, text features of the text modality are obtained based on the resource transfer text contained in the target QR code image itself and the evaluation text generated by the resource transfer operation based on the target QR code image, and image features of the image modality are extracted from the target QR code image itself. Based on the features of these two modalities, the current account corresponding to the target QR code image and the historical accounts corresponding to multiple historical QR code images are divided, which reduces the dependence on the high-dimensional statistical features of the resource transfer data and helps to improve the accuracy of account subset division. Furthermore, the account type of the account subset including the current account is determined through the keywords in the resource transfer text and the evaluation text. Compared with the manually labeled type labels, the account subset is qualitatively characterized, which can mine accounts of unknown risk types and improve the security of QR code-based payment.

[0199] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. In one embodiment, the electronic device can be a server or a terminal device. In this embodiment, the structure of the electronic device can be as follows: Fig.15 As shown, it includes a memory 1501 , a communication module 1503 and one or more processors 1502 .

[0200] The memory 1501 is used to store computer programs executed by the processor 1502. The memory 1501 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an operating instruction set.

[0201] The memory 1501 may be a volatile memory, such as a random-access memory (RAM); the memory 1501 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1501 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1501 may be a combination of the above memories.

[0202] The processor 1502 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1502 is used to implement the above-mentioned multi-modal account type identification method when calling the computer program stored in the memory 1501 .

[0203] The communication module 1503 is used to communicate with terminal devices and other servers.

[0204] The specific connection medium between the memory 1501, the communication module 1503 and the processor 1502 is not limited in the embodiment of the present application. Fig.15 In the embodiment, the memory 1501 and the processor 1502 are connected via a bus 1504. The bus 1504 is connected to the processor 1502 via a bus 1504. Fig.15 The connections between the other components are described with bold lines, which are only for illustration and are not intended to be limiting. The bus 1504 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.15 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or one type of bus.

[0205] The memory 1501 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the multi-modal account type identification method of the embodiment of the present application. The processor 1502 is used to execute the steps of the multi-modal account type identification method.

[0206] Taking the electronic device as the terminal device as an example, its structure is as follows Fig.16 As shown, it includes: a communication component 1610, a memory 1620, a display unit 1630, a camera 1640, a sensor 1650, an audio circuit 1660, a Bluetooth module 1670, a processor 1680 and other components.

[0207] The communication component 1610 is used to communicate with the server. In some embodiments, a circuit wireless fidelity (Wireless Fidelity, WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The computer device can help the user to send and receive information through the WiFi module.

[0208] The memory 1620 can be used to store software programs and data. The processor 1680 executes various functions and data processing of the terminal device by running the software programs or data stored in the memory 1620. The memory 1620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The memory 1620 stores an operating system that enables the terminal device to run. In the present application, the memory 1620 can store an operating system and various application programs, and can also store a computer program for executing the multimodal account type identification method in the embodiment of the present application.

[0209] The display unit 1630 can also be used to display information input by the user or information provided to the user and a graphical user interface (GUI) of various menus of the terminal device 110. Specifically, the display unit 1630 may include a display screen 1632 disposed on the front of the terminal device. The display screen 1632 may be configured in the form of a liquid crystal display, a light-emitting diode, etc. The display unit 1630 may be used to display the user interface of the QR code payment in the embodiment of the present application, etc.

[0210] The display unit 1630 can also be used to receive input digital or character information and generate signal input related to user settings and function control of the terminal device. Specifically, the display unit 1630 may include a touch screen 1631 arranged on the front of the terminal device, which can collect user touch operations on or near it, such as clicking a button, dragging a scroll box, etc.

[0211] The touch screen 1631 may be covered on the display screen 1632, or the touch screen 1631 and the display screen 1632 may be integrated to realize the input and output functions of the terminal device 110, and the integrated touch screen may be referred to as a touch display screen. In the present application, the display unit 1630 may display the application and the corresponding operation steps.

[0212] The camera 1640 can be used to capture still images, and users can post comments on the images taken by the camera 1640 through the application. The camera 1640 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the processor 1680 to convert it into a digital image signal.

[0213] The terminal device may further include at least one sensor 1650, such as an acceleration sensor 1651, a distance sensor 1652, a fingerprint sensor 1653, and a temperature sensor 1654. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0214] The audio circuit 1660, the speaker 1661, and the microphone 1662 can provide an audio interface between the user and the terminal device 110. The audio circuit 1660 can transmit the electrical signal converted from the received audio data to the speaker 1661, which is converted into a sound signal for output. The terminal device 110 can also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1662 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1660 and converted into audio data, and then the audio data is output to the communication component 1610 to be sent to, for example, another terminal device 110, or the audio data is output to the memory 1620 for further processing.

[0215] The Bluetooth module 1670 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable computer device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1670 to exchange data.

[0216] The processor 1680 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 1620 and calling data stored in the memory 1620. In some embodiments, the processor 1680 may include one or more processing units; the processor 1680 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, the user interface, and the application program, and the baseband processor mainly processes wireless communication. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 1680. In the present application, the processor 1680 can run the operating system, the application program, the user interface display and the touch response, as well as the multi-modal account type identification method of the embodiment of the present application. In addition, the processor 1680 is coupled to the display unit 1630.

[0217] In some possible implementations, various aspects of the data classification method provided by the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the multimodal account type identification method according to various exemplary embodiments of the present application described above in this specification. For example, the electronic device may execute the following steps: Figure 3 Follow the steps shown in .

[0218] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0219] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0220] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0221] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0222] The computer program for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer program may be executed entirely on the user computing device, partially on the user computing device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network or a wide area network, or may be connected to an external computing device.

[0223] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0224] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0225] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A multimodal account type identification method, characterized in that: include: Scan the target two-dimensional code image to be processed, obtain the resource transfer text contained in the target two-dimensional code image, and obtain the evaluation text generated by the resource transfer operation based on the target two-dimensional code image; Based on the resource transfer text and the evaluation text, obtaining text features of the target two-dimensional code image, and extracting image features of the target two-dimensional code image; Based on the text features and image features of the target QR code image and in combination with the text features and image features of the processed multiple historical QR code images, the current account corresponding to the target QR code image and the historical accounts corresponding to the multiple historical QR code images are divided into multiple account subsets; Based on the keywords included in the resource transfer text and the evaluation text, the account type of the account subset including the current account is determined.

2. The method according to claim 1, characterized in that Based on the text features and image features of the target QR code image, combined with the text features and image features of the processed multiple historical QR code images, the current account corresponding to the target QR code image and the historical accounts corresponding to the multiple historical QR code images are divided into multiple account subsets, including: For the target two-dimensional code image and each of the multiple historical two-dimensional code images, feature splicing is performed on text features and image features of the two-dimensional code images to obtain fusion features; Based on the fusion features of each QR code image, determine the type similarity between the accounts corresponding to every two QR code images; Based on the determined similarities of each type, the accounts corresponding to the QR code images are divided into multiple account subsets.

3. The method according to claim 2, characterized in that Based on the determined similarities of each type, the accounts corresponding to each QR code image are divided into a plurality of account subsets, including: The account corresponding to each QR code image is taken as a node, and the type similarity between the accounts corresponding to every two QR code images is taken as the weight of the edge between the nodes to obtain an undirected graph; Each node is regarded as a community, and the community modularity is calculated based on the weight of the edge between the nodes; Based on the community modularity, the undirected graph is segmented to obtain multiple subgraphs, and each subgraph serves as an account subset.

4. The method according to claim 3, characterized in that The community modularity is calculated based on the weights of the edges between the nodes; Based on the community modularity, the undirected graph is segmented to obtain multiple subgraphs, including: Calculate the initial community modularity based on the weight of the edge between every two nodes in each node; Merging the nodes at least once, and recalculating the community modularity based on the weights of the edges between the new nodes after each merger, until the community modularity meets a preset condition; Each community that meets the preset condition is regarded as a subgraph.

5. The method according to claim 2, characterized in that Based on the determined similarities of each type, the accounts corresponding to each QR code image are divided into a plurality of account subsets, including: Based on the determined similarities of each type, clustering the accounts corresponding to each of the QR code images to obtain multiple clusters; Each cluster obtained is regarded as an account subset.

6. The method according to claim 1, characterized in that The determining the account type of the account subset including the current account based on the keywords included in the resource transfer text and the evaluation text includes: Extracting each keyword from the resource transfer text and the evaluation text, and comparing each keyword with a preset historical word library; Based on each comparison result, the account type of the account subset including the current account is determined.

7. The method according to claim 1, characterized in that After determining the account type of the account subset including the current account, the method further includes: Obtaining candidate accounts whose confidence level is lower than a preset confidence threshold from the account subset including the current account; Based on the accumulated historical account labels, the account type of each candidate account is obtained; Based on the account type of each candidate account, the structure of the undirected graph or the cluster is adjusted to update the existing account type, wherein the undirected graph or the cluster is used to divide the account subsets.

8. The method according to any one of claims 1 to 7, characterized in that Before obtaining the target text feature of the target two-dimensional code, at least one of the following operations is performed for at least one of the resource transfer text and the evaluation text: Based on a preset text length, a text is truncated to obtain multiple text segments for text feature extraction; Filtering the text to remove information irrelevant to resource transfer; Based on a preset text format, the text is formatted.

9. A multimodal account type identification device, characterized in that: include: A text acquisition module, used to scan the target two-dimensional code image to be processed, obtain the resource transfer text contained in the target two-dimensional code image, and obtain the evaluation text generated by the resource transfer operation based on the target two-dimensional code image; A feature extraction module, used to obtain text features of the target two-dimensional code image based on the resource transfer text and the evaluation text, and to extract image features of the target two-dimensional code image; An account division module, for dividing the current account corresponding to the target QR code image and the historical accounts corresponding to the multiple historical QR code images into multiple account subsets based on the text features and image features of the target QR code image and the text features and image features of the multiple historical QR code images that have been processed; The type determination module is used to determine the account type of the account subset including the current account based on the resource transfer text and the keywords included in the evaluation text.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1-8.

11. A computer-readable storage medium, characterized in that: It comprises a computer program. When the computer program is run on an electronic device, the computer program is used to make the electronic device execute the steps of any one of the methods of claims 1-8.

12. A computer program product, characterized in that The method comprises a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of any one of the methods described in claims 1-8.

Images