Business risk monitoring method

Abstract

The invention relates to the technical field of electronic information, in particular to a business risk monitoring method. The method is applied to micro-service architecture. The method comprises the following steps: extracting keywords through a word frequency-inverse file frequency TF-IDF algorithm, constructing a deep learning model by adopting a Transform algorithm to extract high-frequency words, extracting monitoring information from preprocessed data, and finally sending out a risk early warning signal. The method has great advantages in the aspects of efficiency, accuracy, flexibility, expansibility and the like, and can continuously improve the business handling accuracy and the file handling efficiency of business personnel, thereby powerfully improving the quality and efficiency of overall business handling.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and in particular to a business risk monitoring method applied to a microservice architecture. Background Art

[0002] In the process of business handling, it is crucial to discover risks and issue early warnings as soon as possible. However, due to various reasons, such as insufficient manual review and long business handling cycles, various risks may arise during the handling process. These risks may affect the quality and effectiveness of business handling. Therefore, an efficient and accurate risk monitoring system is of great significance to ensure the quality of business handling and improve the efficiency of handling.

[0003] Traditional business risk monitoring methods are usually based on manual judgment, which has certain limitations when processing large amounts of information and differentiated businesses. With the development of deep learning technology, especially the application of Transformer and Term Frequency-Inverse Document Frequency (TF-IDF), new solutions are provided for business risk monitoring. Summary of the invention

[0004] The purpose of the present invention is to provide a business risk monitoring method, aiming to solve the problems in the above-mentioned background technology.

[0005] A business risk monitoring method comprises the following steps: The first step is data collection. Real-time collection and network collection are used to obtain the data to be monitored. Real-time collection uses tools such as Flume / Kafka to collect user access behaviors and message data recorded by the Web server in real time. Network collection collects data through online resources such as search engines, social media, and websites to obtain large-scale hot topic data. The data sources are Flume / Kafka to obtain MySQL business data in real time, crawlers to obtain online hot topic data, and real-time network transmission to obtain third-party data. The second step is data preprocessing. First, filter the data according to the set keywords, then remove duplicate texts, and finally extract keywords through word frequency-inverse file frequency TF-IDF to extract keywords from the data. To extract keywords, you need to calculate the frequency TF of the word in the document. This frequency is the number of times a word appears in the document divided by the number of all words in the document, and then calculate the inverse document frequency IDF. This value is the reciprocal of the number of documents D containing the word, and take the logarithm of the value. Finally, multiply the obtained TF and IDF to get the TF-IDF value. The third step is to build a deep learning model. The Transformer deep learning model is used to match the preprocessed data with the hot topic data, and select the messages that meet the hot topics. First, the Jieba word segmentation tool is used to segment the collected business data and hot search corpus. The word scale is initialized according to the system configuration. The word vector is initialized randomly with a vector length of 128. The word vector library is built and stored in the Faiss database. The training model uses the standard Tranformer. Each time input is made, one of the words in the sentence needs to be masked. The masked word and its context words are used as input. The predicted value is obtained through the Transformer structure, and the cross entropy loss function is constructed. After 10 rounds of iterations, the Loss value is close to the minimum, and the training ends. The fourth step is data verification. We manually label multiple pieces of data and mark out the hot topic data. We filter the data according to the classification labels and compare them with the hot topic data identified by the model. We calculate the accuracy of the model by identifying the degree of intersection of hot topics. We continuously optimize the deep learning model to narrow the difference range. When the difference range is less than 10%, the data passes the verification. The fifth step is the business risk monitoring system, which feeds back the high-frequency words and hot topic data of the entire network output by the model algorithm to the business system, marks the newly added message data, and monitors the message data in multiple dimensions, including data volume, high-frequency words, and hot topics of the entire network.

[0006] In the third step, hot topics on the Internet are collected every half hour, the model is called, a corresponding vector is generated for each hot topic, and it is stored in the Faiss vector library. The hot event vector library is dynamically updated, and the retrieval method uses Cosine similarity calculation to obtain business data associated with the hot search topic data, and select the message data that meets the hot topics.

[0007] In the fourth step, 10,000 pieces of data are manually labeled.

[0008] Beneficial Effects Keywords are extracted through the word frequency-inverse document frequency (TF-IDF) algorithm, and the Transformer algorithm is used to build a deep learning model to extract high-frequency words, extract monitoring information from the pre-processed data, and finally issue risk warning signals. It has great advantages in efficiency and accuracy, flexibility and scalability, and can continuously improve the accuracy of business processing and the efficiency of business personnel, thereby effectively improving the quality and efficiency of overall business processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a business risk monitoring flow chart of the present invention. DETAILED DESCRIPTION

[0010] The present invention is further described in conjunction with an embodiment, and provides a business risk monitoring method, wherein the business risk monitoring method involves a system including: The business risk monitoring system uses the development languages ​​Java and Python, can run on mainstream servers, and can be adapted to domestic terminals. The system front-end framework uses VUE for component development, and the back-end uses a microservice architecture built with SpringCloud and SpringCloudAlibaba frameworks. The database uses Mysql and Hive for storage. Data processing uses Azkaban to manage task scheduling with Datax and data processing services. Data cleaning, conversion, and synchronization are completed through tasks, and finally stored in Hive for subsequent processing. The Transformer algorithm model is used to calculate hot data on the entire network, and the word frequency-inverse file frequency TF-IDF algorithm is used to extract keywords and high-frequency words from business data, and finally the monitoring information is output for risk warning.

[0011] 1. Data collection module: obtain the data to be monitored. 2. Data preprocessing module: preprocess the collected data, filter the data according to the set keywords, deduplicate the duplicate text, and extract keywords through word frequency-inverse file frequency TF-IDF. 3. Build a deep learning model: Use the Transformer deep learning model to calculate the text semantic similarity between business data, third-party data and hot search topics on the entire network, and select message data that meets the hot topics. 4. Data verification: By comparing the manually annotated data with the extracted monitoring information, the accuracy of the monitoring information is verified for optimizing the deep learning model. 5. Business risk monitoring system: Feedback the high-frequency words output by the model algorithm to the business system, mark the newly added message data, so that business personnel can grasp the message trend in a timely manner during the handling process, and conduct risk monitoring and business processing for the surge in messages that have been warned.

[0012] The business risk monitoring method and system of the present invention can improve the accuracy and timeliness of risk identification in the business handling process. By mining and analyzing business data, and using algorithms such as Transformer, word frequency-inverse document frequency TF-IDF, etc., the effect of efficiently and accurately monitoring business handling risks can be achieved.

[0013] 1. Usage scenarios The overall business risk monitoring adopts a microservice architecture. The front-end framework uses VUE for component development, and the back-end adopts a microservice architecture built with SpringCloud and SpringCloudAlibaba frameworks. Mysql is used to store data. Azkaban is used to manage task scheduling for data processing, and Datax and data processing services are used to complete data cleaning, conversion, and synchronization through tasks. Finally, the data is stored in Hive for subsequent processing. Deep learning models such as Transformer, word frequency-inverse file frequency TF-IDF are used to extract monitoring information, and finally risk warnings are issued.

[0014] The first step is data collection, which uses real-time collection and network collection technologies to obtain the data to be monitored. Real-time collection uses tools such as Flume / Kafka. The Flume tool was proposed by Cloudera engineer Tom White in 2009, and Kafka was proposed by LinkedIn engineers Jay Kreps, Neha Narkhede and Jun Rao in 2011. It collects user access behavior and message data recorded by the Web server in real time. Network collection collects data through online resources such as search engines, social media, and websites to obtain large-scale hot topic data. The main sources of data are business data (Flume / Kafka obtains Mysql data in real time), online hot topic data (obtained through crawlers), and third-party data (obtained through real-time network transmission).

[0015] The second step is data preprocessing, which is mainly used to preprocess the collected data. First, filter the data according to the set keywords; second, remove duplicate texts. Netizens may leave messages with the same problem repeatedly. Filter out messages with text similarity exceeding 99% and remove duplicates. Finally, extract keywords from the data through the term frequency-inverse document frequency TF-IDF, which was proposed by Salton in the article "Term Frequency-Inverse Document Frequency" in 1988. To extract keywords, you need to calculate the frequency of the word in the document (TF). This frequency is the number of times the word appears in the document divided by the number of all words in the document. Then calculate the inverse document frequency (IDF). This value is the reciprocal of the number of documents (D) containing the word, and take the logarithm of the value. Finally, multiply the obtained TF and IDF to get the TF-IDF value. The larger this value is, the more important the word is in the document, and the more likely it is a keyword. The TF-IDF method takes into account the frequency of words in the document and the occurrence of words in the corpus. Through these two aspects of information, it more accurately reflects the importance of words in the document, thereby accurately extracting keywords from text data. Then, stop words are filtered through a configurable text stop word library, which helps to improve the accuracy of high-frequency vocabulary statistics.

[0016] The third step is to build a deep learning model. The Transformer deep learning model was proposed by the Google team in the paper "Attention is All You Need" in 2017. It is used to match preprocessed data with hot topic data and select messages that match hot topics. First, the Jieba word segmentation tool is used to segment the collected business data and hot search corpus. The word scale is initialized according to the system configuration. The word vector is initialized randomly with a vector length of 128. The word vector library is built and stored in the Faiss database. The training model uses the standard Tranformer. Each time the input is input, one of the words in the sentence needs to be masked. The masked word and its context words are used as input, and the predicted value is obtained through the Transformer structure. The cross entropy loss function is constructed. After 10 rounds of iterations, the Loss value is close to the minimum, and the training ends. Hot events on the Internet are collected every half an hour, and the model is called. A corresponding vector is generated for each hot event and stored in the Faiss vector library. The hot event vector library is dynamically updated. The retrieval method uses Cosine similarity calculation to obtain business data associated with hot search topic data and select message data that matches the hot topics.

[0017] The fourth step is data verification. By comparing the manually labeled data with the extracted monitoring information, the accuracy of the monitoring information is verified, which is mainly used to optimize the deep learning model. 10,000 pieces of data are manually labeled to mark the hot topic data. The data is filtered according to the classification label and then compared with the hot topic data identified by the model. The accuracy of the model is calculated by identifying the degree of intersection of hot topics. The difference range is narrowed through continuous optimization of the deep learning model. When the difference range is less than 10%, the data verification is passed.

[0018] The fifth step is the business risk monitoring system, which is mainly used to control the workflow of the above modules, output monitoring information through deep learning models, and issue risk warning signals. Feedback the high-frequency words and hot topic data of the whole network output by the model algorithm to the business system, mark the newly added message data, and facilitate business personnel to grasp the message trend in time during the handling process, and conduct risk monitoring and business processing for the surge messages of the warning. The existing system simply relies on data volume for monitoring, and requires manual participation to view the reasons for the surge in data volume and hot data. It is impossible to quickly process high-frequency messages and hot topic data of the whole network, resulting in a series of business handling risks such as inaccurate system warnings and untimely business responses. Through the business risk monitoring system, message data can be monitored in multiple dimensions (data volume, high-frequency words, hot topics of the whole network, etc.), and at the same time support the rapid configuration of risk monitoring indicators to realize the process of risk monitoring, thereby improving the timeliness and accuracy of monitoring information, and improving the quality and efficiency of information retrieval by business personnel in the process of business handling.

[0019] The specific beneficial effects are described as follows: In the process of business handling, manual analysis and marking of business data is a common method. The present invention marks business data through a big data algorithm model, and finally realizes intelligent risk warning. The beneficial effects are mainly reflected in the following aspects: Efficiency and accuracy: Compared with the warning information output by the algorithm model, manual identification is more time-consuming, and the accuracy of manual judgment will also be reduced when a large number of businesses are processed at the same time. However, the algorithm model can process a large amount of data quickly and may be more accurate than manual in image and attachment recognition.

[0020] Flexibility and scalability: Manual risk identification usually takes a lot of time to prepare and train business personnel, and may have difficulty understanding large-scale, high-precision business data. Algorithmic models have better flexibility and scalability, can automatically adapt to new data and tasks, and can improve accuracy and performance through continuous training and adjustment.

[0021] To sum up, intelligent risk warning is carried out through big data algorithm models, which can monitor risks and issue warnings in real time for business personnel during the case handling process, effectively ensure the accuracy of business handling, and at the same time greatly improve the handling efficiency of business personnel and improve the quality and efficiency of business handling.

[0022] Glossary Transformer: An algorithm for neural network architecture based on self-attention mechanism.

[0023] TF-IDF (Term Frequency-Inverse Document Frequency): is a feature vectorization method widely used in information retrieval and text mining. It is used to reflect the importance of a word to a document set or a document in a corpus.

Claims

1. A business risk monitoring method, characterized in that The steps include: The first step is data collection. Real-time collection and network collection are used to obtain the data to be monitored. Real-time collection uses tools such as Flume / Kafka to collect user access behaviors and message data recorded by the Web server in real time. Network collection collects data through online resources such as search engines, social media, and websites to obtain large-scale hot topic data. The data sources are Flume / Kafka to obtain MySQL business data in real time, crawlers to obtain online hot topic data, and real-time network transmission to obtain third-party data. The second step is data preprocessing. First, filter the data according to the set keywords, then remove duplicate texts, and finally extract keywords through word frequency-inverse file frequency TF-IDF to extract keywords from the data. To extract keywords, you need to calculate the frequency TF of the word in the document. This frequency is the number of times a word appears in the document divided by the number of all words in the document, and then calculate the inverse document frequency IDF. This value is the reciprocal of the number of documents D containing the word, and take the logarithm of the value. Finally, multiply the obtained TF and IDF to get the TF-IDF value. The third step is to build a deep learning model. The Transformer deep learning model is used to match the preprocessed data with the hot topic data, and select the messages that meet the hot topics. First, the Jieba word segmentation tool is used to segment the collected business data and hot search corpus. The word scale is initialized according to the system configuration. The word vector is initialized randomly with a vector length of 128. The word vector library is built and stored in the Faiss database. The training model uses the standard Tranformer. Each time the input is input, one of the words in the sentence needs to be masked. The masked word and its context words are used as input, and the predicted value is obtained through the Transformer structure. The cross entropy loss function is constructed. After 10 rounds of iterations, the Loss value is close to the minimum, and the training ends. The fourth step is data verification. We manually label multiple pieces of data and mark out the hot topic data. We filter the data according to the classification labels and compare them with the hot topic data identified by the model. We calculate the accuracy of the model by identifying the degree of intersection of hot topics. We continuously optimize the deep learning model to narrow the difference range. When the difference range is less than 10%, the data passes the verification. The fifth step is the business risk monitoring system, which feeds back the high-frequency words and hot topic data of the entire network output by the model algorithm to the business system, marks the newly added message data, and monitors the message data in multiple dimensions, including data volume, high-frequency words, and hot topics of the entire network.

2. A business risk monitoring method according to claim 1, characterized in that In the third step, hot topics on the Internet are collected every half hour, the model is called, a corresponding vector is generated for each hot topic, and it is stored in the Faiss vector library. The hot event vector library is dynamically updated, and the retrieval method uses Cosine similarity calculation to obtain business data associated with the hot search topic data, and select the message data that meets the hot topics.

3. A business risk monitoring method according to claim 1, characterized in that In the fourth step, 10,000 pieces of data are manually labeled.

Images