Open source information acquisition and processing method based on RPA technology

Abstract

The invention discloses an open-source intelligence acquisition and processing method based on an RPA technology, and the method comprises the steps: capturing economic data and statistics, financial market information, policy and regulation announcements, enterprise information, economic research reports and analysis, news reports and social media dynamics from the Internet through a distributed web crawler; extracting key information from the captured text by using an abstract extraction algorithm; translating the captured text in real time by using an intelligent translation algorithm; performing intelligent labeling on the captured text by using a labeling algorithm; classifying the grabbed texts by adopting an automatic classification algorithm; a mixed database strategy is adopted, a relational database and a document database are combined, captured texts, abstracts, translations, labels and classified data are stored in the database, and a structured intelligence resource library is formed. The method simplifies the deployment and maintenance process, is suitable for the intelligence analysis field of various scenes, and has important practical value and wide application prospect.

Description

Technical Field

[0001] The present invention relates to Internet open source intelligence collection and processing technology, and in particular to an Internet open source intelligence collection and processing method based on machine processing automation (RPA). Background Art

[0002] Internet open source intelligence, as an important part of the entire intelligence system, is an important reference for analyzing global economic trends and formulating national economic strategies. It has extremely important value and significance for national economic security and market stability.

[0003] At present, with the rapid development of mobile Internet, social networks and self-media, the characteristics of Internet open source intelligence big data are obvious. First, the amount of data is large, including the amount of collection, storage and calculation; second, the types and sources are diversified, including structured, semi-structured and unstructured data; third, the data value density is relatively low; fourth, the data grows fast, the processing speed is also fast, and the timeliness requirement is high; fifth, the accuracy and reliability of the data are low, that is, the quality of the data is not high. In addition, there are problems such as the different languages ​​and characters of website media from different countries and regions, which bring many challenges to the collection and processing of Internet open source intelligence.

[0004] The traditional Internet open source intelligence collection and processing system is based on the big data platform, which solves some problems such as data collection, cleaning, and storage. However, there are still many problems in terms of efficiency, agility, deployment and maintenance, which are as follows:

[0005] 1. Some key processing flows are completely dependent on manpower, which is inefficient. Open source intelligence mostly comes from websites, forums, public accounts, self-media, social networks and other network platforms in different countries and regions. They are in different languages, diverse in types and large in data volume. During the secondary processing after collection, data cleaning, abstract extraction, text translation, labeling, classification and other operations are required. Among these operations, a large amount of work is still done manually, which not only consumes a lot of manpower, but also has low efficiency and is difficult to adapt to the current situation of rapid information explosion.

[0006] 2. Lack of agility, and difficulty adapting to the rapid changes in collection targets. At present, the Internet, especially the mobile Internet, is developing rapidly, and various applications are emerging in an endless stream. Traditional websites, forums and other information carriers are rapidly being replaced by social networks, Twitter, Facebook, etc. At the same time, new mobile Internet applications such as Tiktok are constantly emerging, constantly grabbing the market of traditional media. Traditional Internet open source intelligence collection and processing systems are often difficult to change quickly and adapt to the rapid changes of different platforms and different targets.

[0007] 3. The system development and implementation environment are not unified, and deployment and maintenance are complicated. Traditional Internet intelligence collection and processing systems are generally developed and deployed separately from the system. Due to the inconsistency between the development and actual deployment environment, when the system is online, the development test is normal but the deployment is abnormal. In addition, different operating environments need to be installed in different environments, which will affect the deployment and maintenance of the system. Summary of the invention

[0008] The purpose of this invention is to propose an open source intelligence collection and processing method based on RPA technology to solve the shortcomings of traditional systems in terms of efficiency, agility, deployment and maintenance.

[0009] The technical solution to achieve the purpose of the present invention is: an open source intelligence collection and processing method based on RPA technology, comprising the following steps:

[0010] Through distributed web crawlers, we crawl text information related to economic data from the Internet, including economic data and statistics, financial market information, policy and regulatory announcements, corporate information, economic research reports and analysis, news reports and social media dynamics;

[0011] Use summary extraction algorithms to extract key information from the captured text, including topic identification, opinion summary, key facts, results and impacts, actions and decisions, trends, patterns and sentiment tendencies;

[0012] Use intelligent translation algorithms to translate captured text in real time;

[0013] Using labeling algorithms, the captured text is intelligently labeled, including concept labels, classification labels, summary labels, time labels, and geographic location labels;

[0014] Automatic classification algorithms are used to classify the captured text into macroeconomics, industry categories, financial markets, corporate information, economic data and statistics, policies and regulations, economic research reports, news and media, and international trade;

[0015] A hybrid database strategy is adopted, combining relational database and document database, and the captured text, summary, translation, label, and classification data are stored in the database to form a structured intelligence resource library.

[0016] Furthermore, through distributed web crawlers, text information related to economic data is captured from the Internet, including economic data and statistics, financial market information, policy and regulatory announcements, corporate information, economic research reports and analysis, news reports and social media dynamics, among which:

[0017] A distributed web crawler deploys crawler programs on multiple nodes, utilizes the computing and storage resources of these nodes, and realizes the collection of Internet data; each node runs the crawler task independently, while collaborating with other nodes, sharing the crawled URLs and data, and avoiding duplicate crawling; through a load balancing and task scheduling mechanism, the crawling tasks are allocated to ensure the full utilization of resources and the maximization of crawling efficiency;

[0018] Among them, the load balancing and task scheduling mechanism adopts the weighted least connections strategy. The weight of each node is associated with its processing capacity, and the tasks are allocated according to the effective connection number of each node. Suppose there are n nodes, and each node i has a weight w i and a current connection number c i , define the "effective connection number" e of node i i as:

[0019]

[0020] Considering multiple performance metrics comprehensively, including response time, resource usage, error rate, and task completion rate, by monitoring the performance data of each node in real time, a comprehensive performance score is calculated to dynamically adjust the weight of the node. Set the initial weight as w i , calculate the new weight w' according to the following formula i :

[0021] w′ i = w i ×(1 + λ 1 ×ΔRT i - λ 2 ×ΔRU i - λ 3 ×ΔE i + λ 4 ×CR i )

[0022] Where:

[0023] ΔRT i represents the change rate of the response time of node i. A negative value indicates a decrease in the response time, and a positive value indicates an increase;

[0024] ΔRU i represents the change in resource utilization rate. A positive value indicates an increase in resource usage, which may indicate overload;

[0025] ΔE i represents the change in error rate. A positive value indicates an increase in the error rate;

[0026] CR i represents the task completion rate, which is a positive indicator. The higher the value, the more tasks are completed; ​

[0027] λ 1 ,λ 2 ,λ 3 ,λ 4 is an adjustment coefficient used to balance the influence of different indicators on the weight.

[0028] Furthermore, the abstract extraction algorithm uses a BERT-based model to extract key information from the crawled text. The implementation steps are as follows:

[0029] (1) Input the original long text: Receive text information related to economic data, including economic data and statistics, financial market information, policy and regulatory announcements, enterprise information, economic research reports and analyses, news reports, and social media dynamics;

[0030] (2) Text preprocessing: Tokenize the original long text, remove stop words, and add special tokens for the start and end of the BERT sequence to form the preprocessed text;

[0031] (3) BERT encoding: Use the pre-trained BERT model to encode the preprocessed text to obtain the deep semantic representation of the text;

[0032] (4) Abstract strategy implementation: According to the deep semantic representation of the text, use an extraction method to determine the abstract content, clearly express the key information, and obtain the text abstract, including topic recognition, view summary, key facts, results and impacts, actions and decisions, trends, patterns, and sentiment tendencies;

[0033] (5) Abstract evaluation and optimization: Evaluate the generated text abstract using preset number of rounds, quality evaluation indicators, and improvement amplitude, and optimize the abstracts that pass the evaluation, including adjusting text fluency and removing duplicate information, to obtain a concise abstract text that retains key information;

[0034] (6) Output the abstract text: Provide a concise abstract text that retains key information to convey the main points of topic recognition, view summary, key facts, results and impacts, actions and decisions, trends, patterns, and sentiment tendencies.

[0035] Furthermore, the intelligent translation algorithm uses a BERT-based model to perform real-time translation on the crawled text. The implementation steps are as follows:

[0036] (1) Input the original long text: Receive text information related to economic data, including economic data and statistics, financial market information, policy and regulatory announcements, enterprise information, economic research reports and analyses, news reports, and social media dynamics;

[0037] (2) Text preprocessing: Tokenize the original long text, remove stop words, and add the special tokens for the start and end of the sequence in BERT to form the preprocessed text;

[0038] (3) BERT encoding: Use the pre-trained BERT model to encode the preprocessed text to obtain the deep semantic representation of the text;

[0039] (4) Sequence-to-sequence translation: Adopt a sequence-to-sequence model, namely a Transformer decoder with an attention mechanism, to generate the translation sequence of the target language according to the deep semantic representation of the text obtained by BERT encoding;

[0040] (5) Post-translation processing: Check and correct the grammar and fluency of the translation output to obtain the target language text with accurate translation and natural language;

[0041] (6) Output the translated text: Provide the target language text with accurate translation and natural language to convey the professional terms and concepts, economic forecasts and analyses, market dynamics and trends in the original text.

[0042] Furthermore, the automatic classification algorithm uses a BERT-based model to classify the crawled text, and the implementation steps are as follows:

[0043] (1) Input the original long text: Receive text information related to economic data, including economic data and statistics, financial market information, policy and regulatory announcements, enterprise information, economic research reports and analyses, news reports and social media dynamics;

[0044] (2) Text preprocessing: Tokenize the original long text, remove stop words, and add the special tokens for the start and end of the sequence in BERT to form the preprocessed text;

[0045] (3) BERT encoding: Use the pre-trained BERT model to encode the preprocessed text to obtain the deep semantic representation of the text;

[0046] (4) Add a classification layer: Add a fully connected layer on the top of the BERT model as a classifier, and this classification layer maps the deep semantic representation of the text output by BERT to different categories in the economic field, including macroeconomics, industry categories, financial markets, enterprise information, economic data and statistics, policies and regulations, economic research reports, news and media, international trade;

[0047] (5) Fine-tune the model: Use the labeled text data to fine-tune the BERT model and the fully connected layer model, optimize the parameters of the BERT model and the parameters of the fully connected layer model to form a complete classification model trained and optimized on the economic field data;

[0048] (6) Model evaluation and deployment: Evaluate the model performance on the validation set, adjust the hyperparameters, and deploy the trained model to the production environment;

[0049] (7) Output classification results: Classify the new input text and output the classification results, including macroeconomics, industry categories, financial markets, enterprise information, economic data and statistics, policies and regulations, economic research reports, news and media, international trade.

[0050] Furthermore, the tagging algorithm uses a keyword extraction model based on a word segmentation library to extract keywords or key phrases from the crawled text as intelligent tags for the text.

[0051] Furthermore, using Docker container technology, build the distributed web crawler, abstract extraction algorithm, intelligent translation algorithm, tagging algorithm, and automatic classification algorithm into container images, and provide call interfaces externally in the form of API services. The implementation steps are as follows:

[0052] (1) Container image construction: Write Dockerfiles for the distributed web crawler, abstract extraction algorithm, intelligent translation algorithm, tagging algorithm, and automatic classification algorithm respectively. Define the environment of the container in the Dockerfile, including the base image, required libraries and dependencies, copying of model files, and commands to run the model;

[0053] (2) Image building: Use the Docker command docker build -t mymodel - image to build the image according to the Dockerfile and create a reusable container image for each algorithm;

[0054] (3) Container instantiation: Start container instances from the image through the docker run command, and each container instance runs an independent model;

[0055] (4) API service containerization: Write a web server for each model using the Flask framework. The server loads the model and provides API endpoints to receive requests and return the prediction results of the model;

[0056] (5) API service exposure: Ensure that the containerized API service can be accessed through the network. Set the container port mapping in the Dockerfile to map the internal port of the container to the port of the host machine.

[0057] An open - source intelligence collection and processing system based on RPA technology implements the described open - source intelligence collection and processing method based on RPA technology to achieve open - source intelligence collection and processing based on RPA technology.

[0058] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the open-source intelligence collection and processing method based on RPA technology, achieving open-source intelligence collection and processing based on RPA technology.

[0059] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the open-source intelligence collection and processing based on RPA technology, achieving open-source intelligence collection and processing based on RPA technology.

[0060] Compared with the prior art, the significant advantages of the present invention are as follows: giving full play to the latest achievements of artificial intelligence in the fields of language intelligence, visual intelligence, etc., combining container and API service technologies, using the idea of machine processing automation, and constructing an automated system covering the entire process of intelligence collection and processing, including intelligence collection, translation, abstracting, annotation, and classification, thereby improving efficiency, enhancing agility, and simplifying deployment and maintenance. Description of the Drawings

[0061] Figure 1 is the data flow diagram of the open-source intelligence collection and processing method based on RPA technology of the present invention;

[0062] Figure 2 is the flow chart of the abstract extraction algorithm model based on BERT of the present invention;

[0063] Figure 3 is the implementation flow chart of the processing method of the present invention. Detailed Embodiments

[0064] In order to make the objectives, technical solutions, and advantages of the present application clearer, the following further details the present application with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] An open-source intelligence collection and processing method based on RPA technology includes the following steps:

[0066] Efficiently crawl text information related to economic data from the Internet through a distributed web crawler, including economic data and statistics, financial market information, policy and regulatory announcements, enterprise information, economic research reports and analyses, news reports, and social media dynamics;

[0067] These data are first processed by an abstract extraction algorithm to automatically extract key information, including topic recognition, opinion summary, key facts, results and impacts, actions and decisions, trends, patterns, and sentiment tendencies;

[0068] Use intelligent translation algorithms to perform real-time translation on the collected multilingual content, enhancing cross-lingual intelligence analysis capabilities;

[0069] By intelligently tagging the content, including concept tags, classification tags, abstract tags, time tags, geographical location tags, etc., the data is made easier to retrieve and analyze;

[0070] Then, adopt an automatic classification algorithm and set multiple categories to organize and manage the data, including macroeconomics, industry categories, financial markets, enterprise information, economic data and statistics, policies and regulations, economic research reports, news and media, international trade, etc., to help the system quickly identify and access information on specific topics or fields.

[0071] Finally, adopt a hybrid database strategy, combining relational databases and document databases, and store the processed data in the database to form a structured intelligence resource library for subsequent querying, analysis, and application.

[0072] The distributed web crawler deploys crawler programs on multiple nodes, utilizes the computing and storage resources of these nodes, and achieves efficient collection of Internet data. Each node runs the crawler task independently, while collaborating with other nodes, sharing the crawled URLs and data, and avoiding duplicate crawling. Through the load balancing and task scheduling mechanism, the crawling tasks are reasonably allocated to ensure the full utilization of resources and the maximization of crawling efficiency.

[0073] Among them, the load balancing and task scheduling mechanism adopts the weighted least connections strategy. The weight of each node is associated with its processing capacity, and the tasks are allocated according to the effective connection count of each node. Suppose there are n nodes, and each node i has a weight w i and a current connection count c i . The "effective connection count" e i of node i can be defined as:

[0074]

[0075] where: c i is the current connection count of node i; w i is the weight of node i, reflecting the processing capacity of this node.

[0076] Taking into account multiple performance metrics, including response time, resource usage, error rate, and task completion rate, by real-time monitoring the performance data of each node, a comprehensive performance score is calculated for dynamically adjusting the weight of the node. Set the initial weight as w i , and calculate the new weight according to the following formula:

[0077] w′ i = w i ×(1 + λ1 × ΔRT i - λ 2 × ΔRU i - λ 3 × ΔE i + λ 4 × CR i )

[0078] Where:

[0079] ΔRT i represents the change rate of the response time of node i. A negative value indicates a decrease in the response time, and a positive value indicates an increase.

[0080] ΔRU i represents the change in resource utilization rate. A positive value indicates an increase in resource usage, which may indicate overload.

[0081] ΔE i represents the change in error rate. A positive value indicates an increase in the error rate.

[0082] CR i represents the task completion rate, which is a positive metric. The higher the value, the more tasks are completed.

[0083] λ 1 , λ 2 , λ 3 , λ 4 is an adjustment coefficient used to balance the influence of different metrics on the weight.

[0084] In implementation, first clarify the calculation method or data collection mechanism for each metric. The response time is obtained by measuring the difference in timestamps before and after request processing; the resource utilization rate is obtained through tools or APIs provided by the operating system; the error rate is statistically analyzed by monitoring system logs; the task completion rate is calculated by counting the number of completed tasks.

[0085] Next, according to the above formula, the weight of each node will be dynamically adjusted based on its performance data. The adjustment coefficient λ needs to be optimized according to the actual operating conditions to ensure that the weight adjustment can reflect the true performance of the node and avoid overreaction and frequent fluctuations.

[0086] Finally, the new weights will be used for subsequent task allocation decisions to achieve better system performance and resource utilization. In addition, the system administrator can monitor the effect of weight adjustment and manually adjust the λ coefficient or redefine the calculation method of the metrics as needed.

[0087] The present invention relates to a variety of natural language processing models, including an abstract extraction algorithm model, a translation algorithm model, a tagging algorithm model, and a classification algorithm model. Among them, the abstract extraction algorithm model, the translation algorithm model, and the classification algorithm model all adopt models based on BERT (Bidirectional Encoder Representations from Transformers), and the tagging algorithm model is a keyword extraction model based on a word segmentation library. The keyword extraction model based on the word segmentation library is a traditional text processing method, mainly used to extract keywords or key phrases from text.

[0088] The implementation steps of the BERT-based abstract extraction algorithm model are as follows:

[0089] (1) Input the original long text: Receive long articles or documents related to the economic field, such as economic data and statistics, financial market information, policy and regulatory announcements, enterprise information, economic research reports and analyses, news reports, and social media dynamics. These texts usually contain complex economic terms and professional vocabulary.

[0090] (2) Text preprocessing: Segment the original long text, remove stop words, and add special tokens for the start and end of the BERT sequence to form the preprocessed text.

[0091] (3) BERT encoding: Use a pre-trained BERT model (a typically known BERT model) to encode the preprocessed text to obtain a deep semantic representation of the text.

[0092] (4) Abstract strategy implementation: According to the deep semantic representation of the text, adopt an extraction method to determine the abstract content, and clearly express key information in a short time to obtain the text abstract, including topic recognition, view summary, key facts, results and impacts, actions and decisions, trends, patterns, and sentiment tendencies.

[0093] (5) Abstract evaluation and optimization: Use preset rounds, quality evaluation metrics (such as ROUGE, BLEU, etc.), and improvement margins to evaluate the generated text abstract, and optimize the abstracts that pass the evaluation, including text fluency adjustment, removal of duplicate information, etc.

[0094] (6) Output the abstract text: Provide a concise abstract text that retains key information. The abstract can clearly convey the main points of topic recognition, view summary, key facts, results and impacts, actions and decisions, trends and patterns, and sentiment tendencies.

[0095] The first three steps of the implementation steps of the BERT-based translation algorithm model are the same as those of the BERT-based abstract extraction algorithm model. The subsequent steps are as follows:

[0096] (4) Sequence-to-sequence translation: A sequence-to-sequence model is adopted, namely a Transformer decoder with an attention mechanism, which generates a translation sequence in the target language based on the deep semantic representation of the text encoded by BERT. During the translation process, special attention is paid to the accuracy and formality of texts such as economic, financial, and policy documents to ensure that the translation results meet the standards of communication in the economic field.

[0097] (5) Post-translation processing: The output of machine translation is checked for grammar and fluency, and manual correction is carried out if necessary, especially when translating texts containing sensitive economic data or requiring precise expression of market analysis and prediction, corporate financial status, policy impacts, etc., to ensure the accuracy and professionalism of the translation.

[0098] (6) Output the translated text: Provide a target-language text with accurate translation and natural language, ensuring that the translation results can clearly convey the professional terms and concepts, economic forecasts and analyses, market dynamics and trends in the original text.

[0099] The first three steps of the implementation steps of the classification algorithm model based on BERT are the same as those of the abstract extraction algorithm model based on BERT. The subsequent steps are as follows:

[0100] (4) Addition of a classification layer: A fully connected layer is added on the top layer of the BERT model as a classifier, which maps the deep semantic representation of the text output by BERT to different categories in the economic field.

[0101] (5) Fine-tuning the model: Use the labeled text data to fine-tune the BERT model and the fully connected layer model, optimize the parameters of the BERT model and the fully connected layer model, and form a complete model trained and optimized on economic field data.

[0102] (6) Model evaluation and deployment: Evaluate the model performance on the validation set, adjust the hyperparameters, and deploy the trained model to the production environment. The validation set contains diverse texts in the economic field to comprehensively evaluate the generalization ability of the model.

[0103] (7) Output the classification results: Classify the new input text and output the classification results, including macroeconomics, industry categories, financial markets, corporate information, economic data and statistics, policies and regulations, economic research reports, news and media, international trade, etc.

[0104] Combining the above steps, using Docker container technology, the web crawler, abstract extraction model, translation model, tagging model, and classification model are built into container images, and then each model is published as an API service, which is achieved through the following steps:

[0105] (1) Container image building: Write Dockerfiles for the web crawler, abstract extraction model, translation model, tagging model, and classification model respectively. Define the environment of the container in the Dockerfile, including the base image, required libraries and dependencies, copying of model files, and commands to run the models.

[0106] (2) Image building: Use the Docker command docker build -t mymodel-image to build the image according to the Dockerfile, which creates a reusable container image for each model.

[0107] (3) Container instantiation: Start container instances from the image through the docker run command, and each container instance runs an independent model.

[0108] (4) API service containerization: Write a lightweight web server for each model using the Flask framework. The server loads the model and provides API endpoints to receive requests and return the prediction results of the model.

[0109] (5) API service exposure: Ensure that the containerized API service can be accessed through the network. Set the container port mapping in the Dockerfile to map the internal port of the container to the port of the host.

[0110] Through the above process, each model is encapsulated in a container and provides a call interface externally in the form of an API service, realizing the rapid deployment, expansion, and maintenance of the model service. At the same time, containerization and API-ization ensure the flexibility and portability of the model service, facilitating running and integration in different environments.

[0111] Through the above process, ensure environmental consistency in the development and deployment processes, simplify dependency management and application deployment, and realize a fast and repeatable build and distribution process; at the same time, the lightweight and isolation of containers improve resource utilization and application security, support easy horizontal expansion and high-availability deployment, facilitate the implementation of continuous integration and continuous deployment (CI / CD), strengthen team collaboration and application portability, thereby ensuring the stability and reliability of the system while improving development efficiency.

[0112] The present invention also proposes an open-source intelligence collection and processing system based on RPA technology, implements the above-mentioned open-source intelligence collection and processing method based on RPA technology, and realizes open-source intelligence collection and processing based on RPA technology.

[0113] A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the open-source intelligence collection and processing method based on RPA technology to achieve open-source intelligence collection and processing based on RPA technology.

[0114] A computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the open-source intelligence collection and processing based on RPA technology to achieve open-source intelligence collection and processing based on RPA technology.

[0115] Embodiment 1

[0116] An open-source intelligence collection and processing method based on RPA technology, as Figure 1 shown, the data processing flow is as follows:

[0117] (1) Through a distributed web crawler, receive long articles or documents related to the economic field, such as efficiently scraping economic data and statistics, financial market information, policy and regulation announcements, enterprise information, economic research reports and analyses, news reports, and social media dynamics from the Internet;

[0118] (2) These data are first processed by a summary extraction algorithm to automatically extract key information, including topic recognition, opinion summary, key facts, results and impacts, actions and decisions, trends, patterns, and sentiment tendencies;

[0119] (3) Use an intelligent translation algorithm to perform real-time translation on the collected multilingual content to enhance cross-lingual intelligence analysis capabilities;

[0120] (4) By intelligently tagging the content, including concept tags, classification tags, summary tags, time tags, geographical location tags, etc., the data is made easy to retrieve and analyze;

[0121] (5) Then adopt an automatic classification algorithm, set multiple categories to organize and manage the data, including macroeconomics, industry categories, financial markets, enterprise information, economic data and statistics, policies and regulations, economic research reports, news and media, international trade, etc., to help the system quickly identify and access information on specific topics or fields;

[0122] (6) Adopt a hybrid database strategy, combining a relational database and a document database, and store the processed data in the database to form a structured intelligence resource library for subsequent querying, analysis, and application.

[0123] Embodiment 2

[0124] As Figure 2 shown, the summary extraction algorithm process based on BERT is as follows:

[0125] (1) Input the original long text: Receive long articles or documents related to the economic field, such as economic data and statistics, financial market information, policy and regulation announcements, enterprise information, economic research reports and analyses, news reports, and social media trends. These texts usually contain complex economic terms and professional vocabulary.

[0126] (2) Text preprocessing: Tokenize the original long text, remove stop words, and add the special tokens for the start and end of the sequence in BERT to form the preprocessed text.

[0127] (3) BERT encoding: Use the pre-trained BERT model (a typical well-known BERT model) to encode the preprocessed text and obtain the deep semantic representation of the text.

[0128] (4) Abstract strategy implementation: According to the deep semantic representation of the text, adopt an extractive method to determine the abstract content, and clearly express the key information in a short time to obtain the text abstract, including topic recognition, view summary, key facts, results and impacts, actions and decisions, trends, patterns, and sentiment tendencies.

[0129] (5) Abstract evaluation and optimization: Use the preset number of rounds, quality evaluation metrics (such as ROUGE, BLEU, etc.), and improvement amplitude to evaluate the generated text abstract, and optimize the abstracts that pass the evaluation, including adjusting text fluency and removing duplicate information.

[0130] (6) Output the abstract text: Provide a concise abstract text that retains the key information. The abstract can clearly convey the key points of topic recognition, view summary, key facts, results and impacts, actions and decisions, trends, patterns, and sentiment tendencies.

[0131] The first three steps of the implementation steps of the BERT-based translation algorithm are the same as those of the BERT-based extractive summarization algorithm. The following are the subsequent steps:

[0132] (4) Sequence-to-sequence translation: Adopt a sequence-to-sequence model, namely a Transformer decoder with an attention mechanism, to generate the translation sequence of the target language according to the deep semantic representation of the text obtained by BERT encoding. Pay special attention to the accuracy and formality of texts such as economic, financial, and policy documents during the translation process to ensure that the translation results meet the standards of economic field communication.

[0133] (5) Post-translation processing: Check the grammar and fluency of the machine translation output, and perform manual correction if necessary, especially when translating texts containing sensitive economic data or requiring precise expression of market analysis and forecasts, enterprise financial conditions, policy impacts, etc., to ensure the accuracy and professionalism of the translation.

[0134] (6) Output the translated text: Provide the target language text with accurate translation and natural language, ensuring that the translation results can clearly convey the professional terms and concepts, economic forecasts and analyses, market dynamics and trends in the original text.

[0135] The first three steps of the implementation steps of the BERT-based classification algorithm are the same as those of the BERT-based extractive summarization algorithm. The subsequent steps are as follows:

[0136] (4) Adding a classification layer: Add a fully connected layer on top of the BERT model as a classifier. This classification layer maps the deep semantic representation of the text output by BERT to different categories in the economic field.

[0137] (5) Fine-tuning the model: Use the labeled text data to fine-tune the BERT model and the fully connected layer model, optimize the parameters of the BERT model and the fully connected layer model, and form a complete model trained and optimized on economic field data.

[0138] (6) Model evaluation and deployment: Evaluate the model performance on the validation set, adjust the hyperparameters, and deploy the trained model to the production environment. The validation set contains diverse texts in the economic field to comprehensively evaluate the generalization ability of the model.

[0139] (7) Output the classification results: Classify the new input text and output the classification results, including macroeconomics, industry categories, financial markets, enterprise information, economic data and statistics, policies and regulations, economic research reports, news and media, international trade, etc.

[0140] Example 3

[0141] As Figure 3 shown, the containerization implementation process is as follows:

[0142] Step 1: Prepare the container image and API service.

[0143] (1) Prepare the container image: Prepare the container image containing the required algorithm model and environment.

[0144] (2) Instantiate the container: Instantiate multiple container instances according to the scale and requirements of the acquisition task.

[0145] (3) Invoke the API service: Start each intelligent processing API service container instance, including:

[0146] 1) Automatic extractive summarization API: Extract the summary of the text content;

[0147] 2) Automatic translation API: Translate the text in a non-target language into the target language;

[0148] 3) Automatic Tagging API: Add keyword tags to the text content for easy indexing and retrieval;

[0149] 4) Automatic Classification API: Classify the text to organize the data structure.

[0150] Step 2: Implement distributed intelligence collection through crawler container instances.

[0151] (1) Distributed Web Crawler Deployment: Start the distributed web crawler to efficiently scrape data from the Internet in parallel.

[0152] (2) Data Collection and Preliminary Processing: Scrape the raw data through crawler container instances and perform preliminary cleaning and formatting.

[0153] (3) Crawl the intelligence data of the target website.

[0154] Step 3: Invoke the API service of the intelligent abstract extraction model.

[0155] (1) Invoke the intelligent abstract extraction API service interface, and pass in the data and parameters to be processed. The parameter is the upper limit value of the number of words in the extracted abstract;

[0156] (2) Run the intelligent abstract extraction processing model inside the container instance to generate the abstract result. The model uses a BERT-based abstract extraction model.

[0157] Step 4: Invoke the API service of the text translation model.

[0158] (1) Invoke the text translation API service interface, and pass in the data and parameters to be processed. The parameters are the input language, target language, and the content to be translated;

[0159] (2) Run the text translation processing model inside the container instance to generate the text translation result. The text translation model can use a self-developed BERT-based text translation model.

[0160] Step 5: Invoke the API service of the intelligent tagging model.

[0161] (1) Invoke the intelligent tagging API service interface, and pass in the data and parameters to be processed. The parameter is the number of tags;

[0162] (2) Run the intelligent tagging processing model inside the container instance to generate the tag result. The tagging model is a keyword extraction model based on a word segmentation library, and the extracted keywords are used as document tags.

[0163] Step 6: Invoke the API service of the intelligent classification model.

[0164] (1) Invoke the intelligent classification API service interface, and input the data and parameters to be processed. The parameters are the keywords extracted in Step 5.

[0165] (2) Run the intelligent classification processing model inside the container instance to generate classification results. The classification processing model adopts a BERT-based text classification model.

[0166] Step 7: Perform manual intervention.

[0167] Manual intervention and quality control: During the automated processing, set up a manual intervention link to correct possible errors in the processes of intelligent summarization, translation, tagging, and classification.

[0168] Step 8: Store the collected and processed open-source intelligence data into the database.

[0169] Store the final data into the database to form a structured intelligence resource library.

[0170] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0171] The above-described embodiments only represent several implementation manners of the present application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several deformations and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for collecting and processing open source intelligence based on RPA technology, characterized in that: The steps include: Through distributed web crawlers, we crawl text information related to economic data from the Internet, including economic data and statistics, financial market information, policy and regulatory announcements, corporate information, economic research reports and analysis, news reports and social media dynamics; Use summary extraction algorithms to extract key information from the captured text, including topic identification, opinion summary, key facts, results and impacts, actions and decisions, trends, patterns and sentiment tendencies; Use intelligent translation algorithms to translate captured text in real time; Using labeling algorithms, the captured text is intelligently labeled, including concept labels, classification labels, summary labels, time labels, and geographic location labels; Automatic classification algorithms are used to classify the captured text into macroeconomics, industry categories, financial markets, corporate information, economic data and statistics, policies and regulations, economic research reports, news and media, and international trade; A hybrid database strategy is adopted, combining relational database and document database, and the captured text, summary, translation, label, and classification data are stored in the database to form a structured intelligence resource library.

2. The open source intelligence collection and processing method based on RPA technology according to claim 1 is characterized in that: Through distributed web crawlers, we crawl text information related to economic data from the Internet, including economic data and statistics, financial market information, policy and regulatory announcements, corporate information, economic research reports and analysis, news reports and social media dynamics, including: Distributed web crawlers collect Internet data by deploying crawler programs on multiple nodes and utilizing the computing and storage resources of these nodes. Each node runs crawler tasks independently and works with other nodes to share crawled URLs and data to avoid repeated crawling. Crawling tasks are distributed through load balancing and task scheduling mechanisms to ensure full utilization of resources and maximum crawling efficiency. The load balancing and task scheduling mechanism adopts the weighted least connection strategy. The weight of each node is related to its processing capacity, and the tasks are allocated according to the number of valid connections of each node. Assuming there are n nodes, each node i has a weight w i and a current number of connections c i , define the "effective number of connections" e of node i i for: Taking into account multiple performance indicators, including response time, resource usage, error rate, and task completion rate, a comprehensive performance score is calculated by real-time monitoring of the performance data of each node, which is used to dynamically adjust the node weight. The initial weight is set to w i , calculate the new weight w' according to the following formula i : In' i =in i ×(1+λ1×ΔRT i -λ2×ΔRU i -λ3×ΔE i +λ4×CR i ) in: ΔRT i It represents the response time change rate of node i. A negative value indicates a decrease in response time, and a positive value indicates an increase. ΔRU i Indicates changes in resource usage, with positive values ​​indicating increased resource usage, possibly indicating overload; ΔE i Indicates the change in error rate, a positive value indicates an increase in error rate; CR i Indicates the task completion rate, which is a positive indicator. The higher the value, the more tasks are completed. λ1, λ2, λ3, and λ4 are adjustment coefficients used to balance the impact of different indicators on weights.

3. The open source intelligence collection and processing method based on RPA technology according to claim 1 is characterized in that: The extractive summarization algorithm uses a BERT-based model to extract key information from the captured text. The implementation steps are as follows: (1) Input original long text: Receive text information related to economic data, including economic data and statistics, financial market information, policy and regulatory announcements, corporate information, economic research reports and analysis, news reports, and social media dynamics; (2) Text preprocessing: Segment the original long text, remove stop words, and add BERT’s special markers for the start and end of the sequence to form the preprocessed text; (3) BERT encoding: Use the pre-trained BERT model to encode the pre-processed text to obtain a deep semantic representation of the text; (4) Summary strategy implementation: Based on the deep semantic representation of the text, an extractive approach is used to determine the summary content, achieve clear expression of key information, and obtain a text summary, including topic identification, point summary, key facts, results and impacts, actions and decisions, trends, patterns, and sentiment tendencies; (5) Summary evaluation and optimization: The generated text summaries are evaluated using the preset number of rounds, quality evaluation indicators, and improvement range. The summaries that pass the evaluation are optimized, including adjusting the text fluency and removing duplicate information to obtain a concise summary text that retains key information; (6) Output summary text: Provide a concise summary text that retains key information to convey the main points of topic identification, point summary, key facts, results and impacts, actions and decisions, trends, patterns and emotional tendencies.

4. The open source intelligence collection and processing method based on RPA technology according to claim 1 is characterized in that: The intelligent translation algorithm uses a BERT-based model to translate the captured text in real time. The implementation steps are as follows: (1) Input original long text: Receive text information related to economic data, including economic data and statistics, financial market information, policy and regulatory announcements, corporate information, economic research reports and analysis, news reports, and social media dynamics; (2) Text preprocessing: Segment the original long text, remove stop words, and add BERT’s special markers for the start and end of the sequence to form the preprocessed text; (3) BERT encoding: Use the pre-trained BERT model to encode the pre-processed text to obtain a deep semantic representation of the text; (4) Sequence-to-sequence translation: A sequence-to-sequence model, namely a Transformer decoder with an attention mechanism, is used to generate a translation sequence in the target language based on the deep semantic representation of the text encoded by BERT. (5) Post-translation processing: Check and correct the grammar and fluency of the translated output to obtain an accurate and natural target language text; (6) Output translated text: Provide accurate and natural-sounding target language text to convey the original text’s professional terms and concepts, economic forecasts and analysis, and market dynamics and trends.

5. The open source intelligence collection and processing method based on RPA technology according to claim 1 is characterized in that: The automatic classification algorithm uses a BERT-based model to classify the captured text. The implementation steps are as follows: (1) Input original long text: Receive text information related to economic data, including economic data and statistics, financial market information, policy and regulatory announcements, corporate information, economic research reports and analysis, news reports, and social media dynamics; (2) Text preprocessing: Segment the original long text, remove stop words, and add BERT’s special markers for the start and end of the sequence to form the preprocessed text; (3) BERT encoding: Use the pre-trained BERT model to encode the pre-processed text to obtain a deep semantic representation of the text; (4) Classification layer addition: A fully connected layer is added to the top layer of the BERT model as a classifier. This classification layer maps the deep semantic representation of the text output by BERT to different categories in the economic field, including macroeconomics, industry categories, financial markets, corporate information, economic data and statistics, policies and regulations, economic research reports, news and media, and international trade. (5) Fine-tuning the model: Use the annotated text data to fine-tune the BERT model and the fully connected layer model, optimize the parameters of the BERT model and the fully connected layer model, and form a complete classification model trained and optimized on economic field data; (6) Model evaluation and deployment: Evaluate model performance on the validation set, adjust hyperparameters, and deploy the trained model to the production environment; (7) Output classification results: Classify new input text and output classification results, including macroeconomics, industry categories, financial markets, corporate information, economic data and statistics, policies and regulations, economic research reports, news and media, and international trade.

6. The open source intelligence collection and processing method based on RPA technology according to claim 1 is characterized in that: The tagging algorithm uses a keyword extraction model based on thesaurus to extract keywords or key phrases from the captured text as intelligent tags for the text.

7. The open source intelligence collection and processing method based on RPA technology according to claim 1 is characterized in that: Using Docker container technology, the distributed web crawler, summary extraction algorithm, intelligent translation algorithm, labeling algorithm and automatic classification algorithm are built into container images, and the calling interface is provided to the outside in the form of API services. The implementation steps are as follows: (1) Container image construction: Write Dockerfiles for the distributed web crawler, summary extraction algorithm, intelligent translation algorithm, labeling algorithm, and automatic classification algorithm. Define the container environment in the Dockerfile, including the base image, required libraries and dependencies, model file copying, and commands for running the model. (2) Build the image: Use the Docker command docker build -t mymodel-image to build the image according to the Dockerfile and create a reusable container image for each algorithm; (3) Container instantiation: Start a container instance from the image using the docker run command. Each container instance runs an independent model. (4) API service containerization: Use the Flask framework to write a web server for each model, which loads the model and provides an API endpoint to receive requests and return the model's prediction results; (5) API service exposure: Ensure that the containerized API service can be accessed through the network. Set the container port mapping in the Dockerfile to map the port inside the container to the port of the host machine.

8. An open source intelligence collection and processing system based on RPA technology, implementing the open source intelligence collection and processing method based on RPA technology described in any one of claims 1-7, to achieve open source intelligence collection and processing based on RPA technology.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the open source intelligence collection and processing method based on RPA technology according to any one of claims 1 to 7 is implemented to realize open source intelligence collection and processing based on RPA technology.

10. A computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the open source intelligence collection and processing method based on RPA technology described in any one of claims 1 to 7 is implemented to realize open source intelligence collection and processing based on RPA technology.

Images