Visual question answering anti-fraud method and system based on OCR and NLP

Abstract

The present invention discloses a visual question-answering anti-fraud method and system based on OCR and NLP, which belongs to the field of data processing technology, including: S1, obtaining a target image; S2, using OCR to extract text in the image; S3, integrating and typeset the extracted text; S4, performing data cleaning on the obtained integrated text information; S5, using NLP to identify fraud information; S501, extracting text information from the integrated text, and semantically extracting key information from the text information; S502, integrating the key information in the text information, matching it with fraud information in a corpus, and obtaining a semantic matching result of the fraud information; S6, outputting a result based on the matching result. The present invention combines a target detection algorithm with text recognition, extracts text content, inputs the text content into NLP for semantic extraction, matches it with relevant fraud information, and obtains a result of whether it is fraud information.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to a visual question-answering anti-fraud method and system based on OCR and NLP. Background Art

[0002] Current anti-fraud technology mainly uses information such as phishing site domain names, accounts and mobile phone numbers that send fraudulent information, and can be used through blacklists for simple and effective supervision and defense.

[0003] However, traditional methods are difficult to effectively defend against undocumented fraud. With the development of machine learning technology, the main research direction of fraud detection is to use machine learning to discover the characteristics of the source and content of fraudulent information and make accurate judgments in real time and continuously.

[0004] The main means currently used are as follows:

[0005] 1) Through telephone testing;

[0006] Currently, scammers mostly use mobile phones to defraud account holders using pre-prepared sales pitches. However, research has found that similar behavior can be detected by identifying a specific type of number that is considered abnormal and examining subsequent behavior. If any anomalies are found, a warning can be issued to subsequent users.

[0007] 2) Illegal phishing website detection methods;

[0008] Criminals also send text messages informing users that they've won a prize or their bank account has been frozen, then directing them to visit their website to fill in relevant information and obtain a verification code. At this point, the scammers intercept the message and, taking advantage of the delay, withdraw all funds from the victim's bank account. Current solutions for this tactic primarily focus on detecting unusual websites, publishing them on relevant platforms, and using these cases to issue warnings and reminders to the public.

[0009] 3) Fraudulent number identification and early warning methods based on AI and data mining;

[0010] The technology primarily monitors user behavior profiles, selects users based on multiple features, such as network duration and original feature collection, and processes them to create a usable classification model. Based on this classification model, the target user is input into the corresponding model to determine the target user's fraudulent number risk probability.

[0011] In summary, the prior art has the following shortcomings:

[0012] 1) Narrow target area: Current fraud prevention methods mainly target the characteristics of existing fraudulent user groups, such as phone calls, text messages, etc., and then use case studies to serve as a warning to existing users. However, users are unable to identify and process existing fraud methods. If there is no such fraud method in the original case, it will be difficult to serve as a warning to users.

[0013] 2) Difficult to be accepted or used by both young and old people: Existing technologies all publish current fraud methods on the platform, so that users can learn about the fraud methods through the platform and avoid being deceived by criminals. However, this method cannot be used well by both young and old people. Summary of the Invention

[0014] In order to solve the above-mentioned technical problems existing in the known technology, the present invention provides a visual question answering anti-fraud method and system based on OCR and NLP.

[0015] The first object of the present invention is to provide a visual question answering anti-fraud method based on OCR and NLP, comprising:

[0016] S1. Get the target image;

[0017] S2. Use OCR to extract text from images; specifically:

[0018] S201. Optimize YOLOv5; replace the backbone network in YOLOv5 with GCN;

[0019] S202: Model the target image as an undirected graph and input it into GCN to extract text box features;

[0020] S203: Input the text box features extracted by GCN into the downstream task of Yolov5, and obtain the text detection box through the downstream task of Yolov5;

[0021] S204, using the CRNN model to perform text recognition on the text detection box area;

[0022] S205: Return the recognized highest confidence text to the output result of the CRNN model, obtain the text location information and the specific text information of the location, and output it to the end of the CRNN model;

[0023] S3, integrating and typeset the extracted text;

[0024] S4. performing data cleaning on the obtained integrated text information;

[0025] S5. Use NLP to identify fraudulent information; specifically:

[0026] S501, extracting text information of the integrated text and performing semantic extraction on key information in the text information;

[0027] S502: Integrate key information in the text information and match it with fraud information in the corpus to obtain a semantic matching result of the fraud information;

[0028] S6. Output the result according to the matching result.

[0029] Preferably, in S4, data cleaning includes removing spaces, line breaks, punctuation marks and special characters.

[0030] Preferably, the OCR and NLP are both set up in a cloud network.

[0031] A second object of the present invention is to provide a visual question-answering anti-fraud system based on OCR and NLP, comprising:

[0032] Data acquisition module: obtain target images;

[0033] Text recognition module: uses OCR to extract text from images; specifically includes:

[0034] Optimize units and YOLOv5; replace the backbone network in YOLOv5 with GCN;

[0035] The feature extraction unit models the target image as an undirected graph and inputs it into the GCN to extract the text box features;

[0036] Text detection box acquisition unit: The text box features extracted by GCN are input into the downstream task of Yolov5, and the text detection box is obtained through the downstream task of Yolov5;

[0037] Text recognition unit: uses the CRNN model to perform text recognition in the text detection box area;

[0038] Text information acquisition unit: returns the recognized highest confidence text to the output result of the CRNN model, obtains the text location information and the specific text information of the location, and outputs it to the end of the CRNN model;

[0039] Typesetting module: integrate and typeset the extracted text;

[0040] Cleaning module: perform data cleaning on the integrated text information obtained:

[0041] Identification module: uses NLP to identify fraudulent information; specifically:

[0042] Semantic extraction unit: extracts and integrates text information and performs semantic extraction on key information in the text information;

[0043] Information matching unit: integrates key information in text information and matches it with fraud information in the corpus to obtain semantic matching results of fraud information;

[0044] Output module: output the results according to the matching results.

[0045] Preferably, data cleaning includes removing spaces, line breaks, punctuation marks, and special characters.

[0046] Preferably, the OCR and NLP are both set up in a cloud network.

[0047] The third object of the present invention is to provide an information data processing terminal for implementing the above-mentioned visual question-answering anti-fraud method based on OCR and NLP.

[0048] The fourth object of the present invention is to provide a computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the above-mentioned visual question-answering anti-fraud method based on OCR and NLP.

[0049] The advantages and positive effects of the present invention are:

[0050] This invention improves existing general object detection methods and applies them to more natural scenarios. It combines object detection algorithms with text recognition to extract highly usable text content. This text is then fed into a cloud-based NLP model for semantic extraction, matching it with relevant fraudulent information to determine whether it is fraudulent. Users can use the mini-program to upload images, identify information, and perform voice broadcasting with a single click. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Flowchart of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned purpose, design control system and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0053] Explanation of English abbreviations:

[0054] OCR: Optical Character Recognition is a technology that converts printed characters into editable text.

[0055] NLP: Natural Language Processing is an interdisciplinary field in computer science and artificial intelligence, which aims to enable computers to understand, interpret and generate human language.

[0056] YOLOV5: You Only Look Once Version 5, is a target detection algorithm based on deep learning that can achieve real-time and efficient target detection.

[0057] CRNN: Convolutional Recurrent Neural Network (CRNN) is a deep learning model that combines convolutional neural networks and recurrent neural networks, and is often used to process sequence data.

[0058] GCN: Graph Convolutional Network, is a deep learning model for processing graph data, which can perform feature extraction and representation learning on nodes and edges.

[0059] BERT: Bidirectional Encoder Representations from Transformers (Bidirectional Encoder Representations from Transformers pre-trained model) is a pre-trained natural language processing model that can be used for various NLP tasks such as text classification and question answering systems.

[0060] The present invention utilizes existing target detection technology, text recognition technology and natural language processing technology, specifically an anti-fraud method that combines OCR and NLP. It can determine whether it is a fraudulent text message or information by directly inputting an image.

[0061] like Figure 1 Shown: A visual question answering anti-fraud method based on OCR and NLP, including:

[0062] S1. Acquire the target image. This is accomplished by using a device equipped with a high-definition camera, such as a mobile phone or tablet, to obtain the image and text information to be extracted. This information is then uploaded to the website or mobile app used in this solution for further processing. Before being uploaded to the website, the image information is stored in memory for subsequent processing.

[0063] S2, use OCR to extract text from the image;

[0064] Text detection involves saving the user-entered target image locally and uploading it to a cloud-based OCR model. This model then identifies the location of text within the image using the text's location information. During text detection, the user-provided image must first be saved locally for subsequent processing. Next, the image is uploaded to the cloud-based OCR model, which automatically recognizes the text within the image and returns its location information.

[0065] In this regard, the present application proceeds as follows:

[0066] First, the direct use of YOLOv5 for text detection is not effective, so the backbone network is replaced with the latest graph neural network method. More specifically:

[0067] This application uses the latest graph neural network (GCN) for processing. The input image is modeled as an undirected graph, which is fed into the GCN to extract text box features. The GCN-extracted features are then fed into the downstream task of Yolov5. Through the Yolov5 downstream task, the text detection box is finally obtained.

[0068] The detection process of the model is as follows: the image is input into the modified Yolov5 model, the detection box output by the Yolov5 model is used to retain the text position information, and then a highly effective detection box is obtained through operations such as deduplication.

[0069] This detection box represents the location where text may exist, which facilitates further operations.

[0070] The next stage: text recognition. After obtaining the detected text box, this application uses the CRNN model to identify the areas where text may exist, and returns the text with the highest confidence level to the output result of the model. Finally, the text location information and the specific text information of the area are output to the end of the model to wait for subsequent model processing.

[0071] S2 specifically includes:

[0072] S201. Optimize YOLOv5; replace the backbone network in YOLOv5 with GCN;

[0073] S202: Model the target image as an undirected graph and input it into GCN to extract text box features;

[0074] S203: Input the text box features extracted by GCN into the downstream task of Yolov5, and obtain the text detection box through the downstream task of Yolov5;

[0075] S204, using the CRNN model to perform text recognition on the text detection box area;

[0076] S205: Return the recognized highest confidence text to the output result of the CRNN model, obtain the text location information and the specific text information of the location, and output it to the end of the CRNN model;

[0077] S3. Integrate and format the extracted text. After obtaining the data, it is necessary to integrate and format the extracted valid text according to the position information of the previous text to obtain a valid text segment, which is convenient for the subsequent text processing model.

[0078] S4. performing data cleaning on the obtained integrated text information;

[0079] After obtaining the text information of the user input image, the information may contain some redundant information and useless information. If these noise data are not cleaned, the accuracy of the model will be affected.

[0080] The main tasks of text cleaning are as follows:

[0081] ①Remove spaces and line breaks:

[0082] Considering that spaces and line breaks may affect subsequent text processing, they need to be removed.

[0083] ②Remove punctuation and special characters:

[0084] In fraudulent text messages, punctuation marks and special characters often have no practical meaning and need to be removed.

[0085] Through the above means, all the redundant information is removed from the original data through the data cleaning stage to obtain the final required valid data.

[0086] Through this effective cleaning and formatting of data, it can be better used for downstream NLP tasks.

[0087] S5. Use NLP to identify fraudulent information;

[0088] First, OCR technology converts text in images into computer-readable text. Once the text is obtained, it can be used with a pre-trained NLP model in the cloud to determine whether it represents fraudulent information. Pre-trained models are trained on large amounts of data, acquiring general linguistic knowledge that can be used for natural language processing tasks such as classification and named entity recognition.

[0089] In the NLP model, the following steps are required to obtain the final fraud identification result:

[0090] ① Extract the text information of the text obtained by OCR and perform semantic extraction of key information for subsequent text analysis;

[0091] ②Integrate the key information in the text and match it with the fraud information in the corpus to obtain the semantic matching results of the fraud information.

[0092] S5 is specifically:

[0093] S501, extracting text information of the integrated text and performing semantic extraction on key information in the text information;

[0094] S502: Integrate key information in the text information and match it with fraud information in the corpus to obtain a semantic matching result of the fraud information;

[0095] S6. Output the result according to the matching result.

[0096] According to the matching results obtained by S5, they are effectively presented to users through manual processing to improve the user's interface experience.

[0097] In summary, the core technical contents of this application include:

[0098] ① Text detection: Improve the general target detection algorithm YOLOV5 and apply it to text detection;

[0099] ②Text recognition: Use the CRNN algorithm to perform text recognition on the obtained text detection part;

[0100] ③ Text typesetting: typesetting and restoring the text recognition part;

[0101] ④ Semantic extraction: Extract key semantic information from the typeset text using the NLP algorithm based on the BERT model;

[0102] ⑤Information matching: Match the extracted information with various fraudulent information.

[0103] ⑥ Result feedback: The probability that the information is false is given by matching similarity.

[0104] Traditional fraud detection methods typically require manual intervention, are inefficient, and are susceptible to subjective factors. This combined technology significantly improves the efficiency and accuracy of fraud detection and automates the processing of large amounts of data, reducing manual workload. This system enables us to provide users with an intelligent anti-fraud detection service. Users simply input text information in the form of images, and the system will automatically inform them whether the information is fraudulent, significantly reducing their chances of being scammed.

[0105] The main target users of this solution are the elderly or children. Through a simple and highly available user interface, it has a very good user interface usage effect for users of all ages.

[0106] A second object of the present invention is to provide a visual question-answering anti-fraud system based on OCR and NLP, comprising:

[0107] Data acquisition module: obtain target images;

[0108] Text recognition module: uses OCR to extract text from images; specifically includes:

[0109] Optimize units and YOLOv5; replace the backbone network in YOLOv5 with GCN;

[0110] The feature extraction unit models the target image as an undirected graph and inputs it into the GCN to extract the text box features;

[0111] Text detection box acquisition unit: The text box features extracted by GCN are input into the downstream task of Yolov5, and the text detection box is obtained through the downstream task of Yolov5;

[0112] Text recognition unit: uses the CRNN model to perform text recognition in the text detection box area;

[0113] Text information acquisition unit: returns the recognized highest confidence text to the output result of the CRNN model, obtains the text location information and the specific text information of the location, and outputs it to the end of the CRNN model;

[0114] Typesetting module: integrate and typeset the extracted text;

[0115] Cleaning module: performs data cleaning on the integrated text information obtained; data cleaning includes: removing spaces, line breaks, punctuation marks and special characters;

[0116] Identification module: uses NLP to identify fraudulent information; specifically:

[0117] Semantic extraction unit: extracts and integrates text information and performs semantic extraction on key information in the text information;

[0118] Information matching unit: integrates key information in text information and matches it with fraud information in the corpus to obtain semantic matching results of fraud information;

[0119] Output module: output the results according to the matching results.

[0120] The OCR and NLP are both set up in the cloud network.

[0121] An information data processing terminal is used to implement the above-mentioned visual question-answering anti-fraud method based on OCR and NLP.

[0122] A computer-readable storage medium includes instructions that, when executed on a computer, enable the computer to execute the above-mentioned visual question-answering anti-fraud method based on OCR and NLP.

[0123] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL) or wireless (e.g., infrared, wireless, microwave, etc.)) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0124] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are within the scope of the technical solution of the present invention.

Claims

1. A visual question answering anti-fraud method based on OCR and NLP, characterized by: include: S1. Get the target image; S2, use OCR to extract text from the image; Specifically include: S201. Optimize YOLOv5; replace the backbone network in YOLOv5 with GCN; S202: Model the target image as an undirected graph and input it into GCN to extract text box features; S203: Input the text box features extracted by GCN into the downstream task of Yolov5, and obtain the text detection box through the downstream task of Yolov5; S204, using the CRNN model to perform text recognition on the text detection box area; S205: Return the recognized highest confidence text to the output result of the CRNN model, obtain the text location information and the specific text information of the location, and output it to the end of the CRNN model; S3, integrating and typeset the extracted text; S4. performing data cleaning on the obtained integrated text information; S5. Use NLP to identify fraudulent information; specifically: S501, extracting text information of the integrated text and performing semantic extraction on key information in the text information; S502: Integrate key information in the text information and match it with fraud information in the corpus to obtain a semantic matching result of the fraud information; S6. Output the results based on the matching results.

2. The visual question answering anti-fraud method based on OCR and NLP according to claim 1 is characterized in that: In S4, data cleaning includes removing spaces, line breaks, punctuation marks, and special characters.

3. The visual question answering anti-fraud method based on OCR and NLP according to claim 1 is characterized in that: The OCR and NLP are both set up in the cloud network.

4. A visual question-answering anti-fraud system based on OCR and NLP, characterized by: include: Data acquisition module: obtain target images; Text recognition module: uses OCR to extract text from images; specifically includes: Optimize units and YOLOv5; replace the backbone network in YOLOv5 with GCN; The feature extraction unit models the target image as an undirected graph and inputs it into the GCN to extract the text box features; Text detection box acquisition unit: The text box features extracted by GCN are input into the downstream task of Yolov5, and the text detection box is obtained through the downstream task of Yolov5; Text recognition unit: uses the CRNN model to perform text recognition in the text detection box area; Text information acquisition unit: returns the recognized highest confidence text to the output result of the CRNN model, obtains the text location information and the specific text information of the location, and outputs it to the end of the CRNN model; Typesetting module: integrate and typeset the extracted text; Cleaning module: perform data cleaning on the integrated text information obtained: Identification module: uses NLP to identify fraudulent information; specifically: Semantic extraction unit: extracts and integrates text information and performs semantic extraction on key information in the text information; Information matching unit: integrates key information in text information and matches it with fraud information in the corpus to obtain semantic matching results of fraud information; Output module: output the results according to the matching results.

5. The visual question-answering anti-fraud system based on OCR and NLP according to claim 4 is characterized in that: In the cleaning module, data cleaning includes removing spaces, line breaks, punctuation marks, and special characters.

6. The OCR and NLP-based visual question-answering anti-fraud system according to claim 4 is characterized in that: The OCR and NLP are both set up in the cloud network.

7. An information data processing terminal, characterized in that: Used to implement the OCR and NLP-based visual question answering anti-fraud method described in any one of claims 1-3.

8. A computer-readable storage medium, characterized in that The method comprises instructions which, when executed on a computer, enable the computer to execute the visual question answering anti-fraud method based on OCR and NLP as described in any one of claims 1 to 3.

Images