Policy data processing methods, devices, computer equipment and storage media

Abstract

This application provides a policy data processing method, apparatus, computer equipment, and storage medium, which can significantly improve the efficiency of article data processing and quickly identify policy articles from a large amount of article data. The method includes: cleaning the real-time article data to obtain N article data, where N is a positive integer greater than or equal to 1; inputting the N article data into a policy text classification model to determine whether a target policy article exists among the N article data. The policy text classification model is pre-trained on a training sample set using TEXTCNN. The policy text classification model is used to determine whether the current article is a policy article. The training sample set includes multiple policy article data and multiple non-policy article data. TEXTCNN is a convolutional neural network used for text classification; if a target policy article exists among the N article data, the industry tag corresponding to the target policy article is determined.

Description

[Technical Field]

[0001] This application belongs to the field of data processing, and specifically relates to a policy data processing method, apparatus and storage medium. [Background Technology]

[0002] Local government agencies formulate policies for different regions and industries, and these policies are distributed across numerous websites. However, these websites don't only contain policy articles; they often include content unrelated to policies, such as bidding information, contract information, and news. Therefore, it's necessary to quickly categorize these articles to identify which are policy-related.

[0003] Currently, for classification tasks, in order to pursue model accuracy, many solutions adopt classification methods based on the BERT model. However, this technology is relatively slow and inefficient for systems that need to process tens of thousands of cases a day. [Summary of the Invention]

[0004] This application provides a policy data processing method, apparatus, computer equipment, and storage medium, which can significantly improve the efficiency of article data processing and quickly identify policy articles from a large amount of article data.

[0005] The first aspect of this application provides a policy data processing method, including:

[0006] Obtain real-time article data corresponding to policy websites;

[0007] The real-time article data is cleaned to obtain N article data, where N is a positive integer greater than or equal to 1;

[0008] The N article data are input into a policy text classification model to determine whether a target policy article exists among the N article data. The policy text classification model is pre-trained on a training sample set using TEXTCNN. The policy text classification model is used to determine whether the current article is a policy article. The training sample set includes multiple policy article data and multiple non-policy article data. TEXTCNN is a convolutional neural network used for text classification.

[0009] If the target policy article exists among the N article data, then the industry tag corresponding to the target policy article is determined.

[0010] A second aspect of this application provides a policy data processing apparatus, comprising:

[0011] The acquisition unit is used to acquire real-time article data corresponding to the policy website;

[0012] A data cleaning unit is used to clean the real-time article data to obtain N article data, where N is a positive integer greater than or equal to 1;

[0013] A classification unit is used to input the N article data into a policy text classification model to determine whether a target policy article exists among the N article data. The policy text classification model is pre-trained on a training sample set using TEXTCNN. The policy text classification model is used to determine whether the current article is a policy article. The training sample set includes multiple policy article data and multiple non-policy article data. TEXTCNN is a convolutional neural network used for text classification.

[0014] The determining unit is used to determine the industry tag corresponding to the target policy article if the target policy article exists among the N article data.

[0015] In one possible design, the device further includes:

[0016] Training unit, the training unit is used for:

[0017] Step 1: Obtain the training sample set;

[0018] Step 2: Preprocess the training sample set;

[0019] Step 3: Input the target training sample into TEXTCNN to determine the output result corresponding to the target training sample. The output result is whether the target training sample is a policy article. The target training sample is any training sample in the preprocessed training sample set.

[0020] Step 4: Adjust the loss function corresponding to TEXTCNN based on the output results and the labels of the target training samples;

[0021] Step 5: Iteratively execute steps 3 to 4 based on the adjusted loss function until the preset iteration termination condition is reached;

[0022] Step 6: The TEXTCNN that reaches the preset iteration termination condition is identified as the policy text classification model.

[0023] In one possible design, the training unit is further used for:

[0024] Determine whether the number of iterations has reached a preset value. If so, determine that the preset iteration termination condition is met.

[0025] or,

[0026] Determine whether the loss function has converged; if so, then determine that the preset iteration termination condition is met.

[0027] In one possible design, the determining unit is specifically used for:

[0028] Predict the technical field corresponding to the target policy article;

[0029] The industry tag corresponding to the target policy article is determined based on the mapping relationship between the technical field and the pre-built industry tags at different levels.

[0030] In one possible design, the determining unit is further specifically used for:

[0031] The target policy article is divided into paragraphs based on line breaks;

[0032] Identify the target paragraphs in the target article that contain policy content;

[0033] The industry label corresponding to the target paragraph is determined based on the industry classification model.

[0034] In one possible design, the acquisition unit is specifically used for:

[0035] The real-time article data is retrieved from the policy website using Robotic Process Automation (RPA).

[0036] In one possible design, the data cleaning unit is specifically used for:

[0037] Remove HTML tags from the real-time article data;

[0038] The real-time article data after removing HTML tags is processed with special symbols to obtain the N article data.

[0039] A third aspect of this application provides a computer device including at least one connected processor, memory, and transceiver, wherein the memory is used to store program code, and the processor is used to call the program code in the memory to execute the steps of the vehicle diagnostic method described in the first aspect.

[0040] A fourth aspect of this application provides a computer storage medium including instructions that, when executed on a computer, cause the computer to perform the steps of the vehicle diagnostic method described in the first aspect.

[0041] In summary, it can be seen that in the embodiments provided in this application, the policy data processing device can first acquire real-time article data corresponding to the policy website, clean the data to obtain N article data, and input the N article data into a policy text classification model to determine whether a target policy article exists among the N article data. The policy text classification model is pre-trained on a training sample set using TEXTCNN. The policy text classification model is used to determine whether the current article is a policy. The training sample set includes multiple policy article data and multiple non-policy article data. TEXTCNN is a convolutional neural network used for text classification. If the target policy article exists among the N article data, the industry tag corresponding to the target policy article is determined. Therefore, in this application, the policy article contained in a large amount of article data can be determined from a pre-built TEXTCNN model. Compared with the current classification method using the BERT model, this significantly improves the efficiency of article data processing and quickly identifies policy articles from a large amount of article data. [Attached Image Description]

[0042] Figure 1 A flowchart illustrating the policy data processing method provided in this application embodiment;

[0043] Figure 2 This is a schematic diagram of the virtual structure of the policy data processing device provided in the embodiments of this application;

[0044] Figure 3 This is a schematic diagram of the hardware structure of the policy data processing device provided in an embodiment of the present invention;

[0045] Figure 4 A schematic diagram illustrating an embodiment of an electronic device provided by an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of an embodiment of a computer-readable storage medium provided in this invention.

Detailed Implementation Methods

[0047] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0048] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The division of modules appearing in this application is merely a logical division; in practical applications, other division methods may be used. For example, multiple modules may be combined or integrated into another system, or some feature vectors may be ignored or not executed. Additionally, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interface, and the indirect coupling or communication connection between modules may be electrical or other similar forms, none of which are limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separate, may or may not be physical modules, or may be distributed among multiple circuit modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this application.

[0049] The policy data processing device provided in this application embodiment will be described below from the perspective of a policy data processing device. This policy data processing device can be a server or a service unit within a server, and there is no specific limitation.

[0050] Please see Figure 1 , Figure 1 A flowchart illustrating the policy data processing apparatus provided in this application embodiment includes:

[0051] 101. Obtain real-time article data corresponding to policy websites.

[0052] In this embodiment, the policy data processing device can obtain real-time article data from a large number of policy websites. Specifically, it can use Robotic Process Automation (RPA) to crawl real-time article data from a large number of policy websites. The main function of RPA is to execute work information and business interactions through robots according to a pre-designed process. In this way, when there is a lot of work information and business interactions, RPA can efficiently solve these complex processes and save labor costs.

[0053] 102. Perform data cleaning on the real-time article data to obtain N article data.

[0054] In this embodiment, after obtaining the real-time article data corresponding to the policy website, the policy data processing device can perform data cleaning on the real-time article data. This data cleaning refers to removing HTML tags and special symbols from the real-time article data, etc., to obtain N cleaned article data, where N is an integer greater than or equal to 1.

[0055] 103. Input N articles into the policy text classification model to determine whether the target policy article exists among the N articles.

[0056] In this embodiment, after determining N article data, the policy data processing device can input each of the N article data into a pre-trained policy text classification model used to classify the article data, in order to determine whether a target policy article exists among the N article data. The policy text classification model is pre-trained using TEXTCNN on a training sample set. This model is used to determine whether the current article is a policy article. The training sample set includes multiple policy article data and multiple non-policy article data. TEXTCNN is a convolutional neural network used for text classification. The training process of the policy text classification model is described below:

[0057] Step 1: Obtain the training sample set.

[0058] In this step, the policy data processing device can first acquire a large amount of policy article data and non-policy article data, and then label the policy article data and non-policy article data. The purpose of the labeling is to enable the convolutional neural network that performs text classification to identify whether the current article belongs to the policy article category.

[0059] Step 2: Preprocess the training sample set.

[0060] In this step, the policy data processing device can preprocess each training sample in the training sample set. Preprocessing includes, but is not limited to, data cleaning, data inheritance, and data transformation. The purpose of preprocessing is to correct missing values ​​and spelling errors in each training sample so as to normalize or standardize the values.

[0061] Step 3: Input the target training sample into TEXTCNN to determine the output result corresponding to the target training sample.

[0062] In this step, the policy data processing device can randomly select a training sample from the preprocessed training sample set and input the training sample as the target training sample into TEXTCNN to obtain the output result corresponding to the target training sample. The output result indicates whether the target training sample is a policy article.

[0063] Step 4: Adjust the loss function corresponding to TEXTCNN based on the output results and the labels of the target training samples.

[0064] In this implementation, the policy data processing device can adjust the damage function corresponding to TEXTCNN based on the output result and the label of the target training sample. The purpose of the adjustment is to bring the output result and the label of the target training sample closer together as much as possible.

[0065] Step 5: Iterate through steps 3 and 4 based on the adjusted loss function until the threshold is reached to terminate the iteration.

[0066] Step 6: Determine TEXTCNN as the policy text classification model when the preset iteration termination condition is met.

[0067] During iterative training, the policy data processing device determines whether the iteration termination condition has been met. If so, it stops iterating and identifies the TEXTCNN model at the time of termination as the policy text classification model. If not, it repeats steps 3 and 4 until the preset iteration termination condition is met. Specifically, the policy data processing device can determine whether the number of iterations has reached a preset value; if so, it determines that the preset iteration termination condition has been met.

[0068] or,

[0069] The policy data processing device can determine whether the loss function corresponding to TEXTCNN has converged, that is, whether the loss function no longer changes significantly after multiple iterations. If so, it is determined that the preset iteration termination condition is met.

[0070] In practical applications, the server can also use other conditions as the iteration termination condition, which are not limited here.

[0071] 104. If a target policy article exists among N articles, then determine the industry tag of the target policy article.

[0072] In this embodiment, after determining that a target policy article exists among N article data, the policy data processing device can identify the industry to which the target policy article belongs and assign a corresponding industry tag to it. Specifically, it can predict the technical field corresponding to the target policy article and determine the corresponding industry tag based on the mapping relationship between the technical field and pre-built industry tags at different levels, and then assign the corresponding industry tag to the target policy article. That is, in actual industries, there are often inclusion relationships between industries. Therefore, the policy data processing device pre-specifies a multi-level industry tag system and the mapping relationship between different levels of industries to facilitate industry prediction for relatively coarse-grained industries. For example, some policies only mention that they are about information technology services, without specifying whether they are about artificial intelligence, software, or big data. The device will predict that the current policy is about the information technology field and then assign the corresponding industry tag to the policy based on the mapping relationship between industry tags at different levels.

[0073] In one embodiment, the policy data processing device determines the industry tags corresponding to the target policy article as including:

[0074] Divide the target policy article into paragraphs based on line breaks;

[0075] Identify the target paragraphs in the target article that contain policy content;

[0076] The industry label corresponding to the target paragraph is determined based on the industry classification model.

[0077] In this embodiment, when classifying the target policy article by industry, the process first involves filtering out paragraphs containing policy content from the main text of the policy. Specifically, paragraphs are divided based on line breaks. Then, a policy content classification model is used to identify the target paragraphs containing policy content. After obtaining these policy content paragraphs, industry tags are assigned to them based on the industry classification model. Based on the model's results, a relatively accurate industry tag can be assigned to the policy. Here, the policy data processing device defines first-level, second-level, and third-level tags, and assigns a final third-level tag to the target policy article based on the tag mapping relationship at different levels.

[0078] It should be noted that the training methods for the policy content classification model and industry classification model are similar to those for the policy text classification model. The training of the policy text classification model has been explained in detail above, and will not be repeated here.

[0079] The following explains the labels, industries, and their definitions. Please refer to Table 1, which shows the correspondence between the labels, industries, and their definitions provided in the embodiments of this application:

[0080] Table 1

[0081]

[0082]

[0083]

[0084]

[0085]

[0086] It should be noted that the industries, labels, and definitions in Table 1 above are only examples. Of course, there are other industries, labels, and definitions, which are not limited to these specific ones.

[0087] In summary, it can be seen that in the embodiments provided in this application, the policy data processing device can first acquire real-time article data corresponding to the policy website, clean the data to obtain N article data, and input the N article data into a policy text classification model to determine whether a target policy article exists among the N article data. The policy text classification model is pre-trained on a training sample set using TEXTCNN. The policy text classification model is used to determine whether the current article is a policy. The training sample set includes multiple policy article data and multiple non-policy article data. TEXTCNN is a convolutional neural network used for text classification. If the target policy article exists among the N article data, the industry tag corresponding to the target policy article is determined. Therefore, in this application, the policy article contained in a large amount of article data can be determined from a pre-built TEXTCNN model. Compared with the current classification method using the BERT model, this significantly improves the efficiency of article data processing and quickly identifies policy articles from a large amount of article data.

[0088] Please see Figure 2 , Figure 2 The virtual structural intent of the policy data processing apparatus provided in this application, the policy data processing apparatus 200 includes:

[0089] Acquisition unit 201 is used to acquire real-time article data corresponding to the policy website;

[0090] The data cleaning unit 202 is used to clean the real-time article data to obtain N article data, where N is a positive integer greater than or equal to 1;

[0091] Classification unit 203 is used to input the N article data into a policy text classification model to determine whether there is a target policy article among the N article data. The policy text classification model is pre-trained on a training sample set using TEXTCNN. The policy text classification model is used to determine whether the current article is a policy article. The training sample set includes multiple policy article data and multiple non-policy article data. TEXTCNN is a convolutional neural network used for text classification.

[0092] The determining unit 204 is used to determine the industry tag corresponding to the target policy article if the target policy article exists among the N article data.

[0093] In one possible design, the device further includes:

[0094] Training unit 205, the training unit 205 being used for:

[0095] Step 1: Obtain the training sample set;

[0096] Step 2: Preprocess the training sample set;

[0097] Step 3: Input the target training sample into TEXTCNN to determine the output result corresponding to the target training sample. The output result is whether the target training sample is a policy article. The target training sample is any training sample in the preprocessed training sample set.

[0098] Step 4: Adjust the loss function corresponding to TEXTCNN based on the output results and the labels of the target training samples;

[0099] Step 5: Iteratively execute steps 3 to 4 based on the adjusted loss function until the preset iteration termination condition is reached;

[0100] Step 6: The TEXTCNN that reaches the preset iteration termination condition is identified as the policy text classification model.

[0101] In one possible design, the training unit 205 is further used for:

[0102] Determine whether the number of iterations has reached a preset value. If so, determine that the preset iteration termination condition is met.

[0103] or,

[0104] Determine whether the loss function has converged; if so, then determine that the preset iteration termination condition is met.

[0105] In one possible design, the determining unit 204 is specifically used for:

[0106] Predict the technical field corresponding to the target policy article;

[0107] The industry tag corresponding to the target policy article is determined based on the mapping relationship between the technical field and the pre-built industry tags at different levels.

[0108] In one possible design, the determining unit 204 is further specifically used for:

[0109] The target policy article is divided into paragraphs based on line breaks;

[0110] Identify the target paragraphs in the target article that contain policy content;

[0111] The industry label corresponding to the target paragraph is determined based on the industry classification model.

[0112] In one possible design, the acquisition unit 201 is specifically used for:

[0113] The real-time article data is retrieved from the policy website using Robotic Process Automation (RPA).

[0114] In one possible design, the data cleaning unit 202 is specifically used for:

[0115] Remove HTML tags from the real-time article data;

[0116] The real-time article data after removing HTML tags is processed with special symbols to obtain the N article data.

[0117] above Figure 2 The policy data processing apparatus in this application embodiment has been described from the perspective of modular functional entities. The following is a detailed description of the policy data processing apparatus in this application embodiment from the perspective of hardware processing. Please refer to [link / reference]. Figure 3 One embodiment of the policy data processing device 300 in this application includes:

[0118] Input device 301, output device 302, processor 303, and memory 304 (where the number of processors 303 can be one or more). Figure 3 (Taking a processor 303 as an example). In some embodiments of this application, the input device 301, output device 302, processor 303, and memory 304 can be connected via a bus or other means, wherein... Figure 3 Taking the example of a connection between China and Israel via a bus.

[0119] Specifically, by calling the operation instructions stored in memory 304, processor 303 executes the following steps:

[0120] Obtain real-time article data corresponding to policy websites;

[0121] The real-time article data is cleaned to obtain N article data, where N is a positive integer greater than or equal to 1;

[0122] The N article data are input into a policy text classification model to determine whether a target policy article exists among the N article data. The policy text classification model is pre-trained on a training sample set using TEXTCNN. The policy text classification model is used to determine whether the current article is a policy article. The training sample set includes multiple policy article data and multiple non-policy article data. TEXTCNN is a convolutional neural network used for text classification.

[0123] If the target policy article exists among the N article data, then the industry tag corresponding to the target policy article is determined.

[0124] By calling the operation instructions stored in memory 304, processor 303 is also used to execute... Figure 1 Any of the methods in the corresponding embodiments.

[0125] Please see Figure 4 , Figure 4 A schematic diagram illustrating an embodiment of the electronic device provided in this application.

[0126] like Figure 4 As shown, this application embodiment provides an electronic device, including a memory 410, a processor 420, and a computer program 411 stored in the memory 410 and executable on the processor 420. When the processor 420 executes the computer program 411, it performs the following steps:

[0127] Obtain real-time article data corresponding to policy websites;

[0128] The real-time article data is cleaned to obtain N article data, where N is a positive integer greater than or equal to 1;

[0129] The N article data are input into a policy text classification model to determine whether a target policy article exists among the N article data. The policy text classification model is pre-trained on a training sample set using TEXTCNN. The policy text classification model is used to determine whether the current article is a policy article. The training sample set includes multiple policy article data and multiple non-policy article data. TEXTCNN is a convolutional neural network used for text classification.

[0130] If the target policy article exists among the N article data, then the industry tag corresponding to the target policy article is determined.

[0131] In practical implementation, when the processor 420 executes the computer program 411, it can achieve... Figure 1 Any of the corresponding implementation methods in the embodiments.

[0132] Since the electronic device described in this embodiment is a device used to implement a policy data processing device in the embodiments of this application, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the method described in the embodiments of this application. Therefore, how the electronic device implements the method in the embodiments of this application will not be described in detail here. Any device used by those skilled in the art to implement the method in the embodiments of this application is within the scope of protection of this application.

[0133] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided in this application.

[0134] like Figure 5 As shown, this embodiment provides a computer-readable storage medium 500 on which a computer program 511 is stored. When the computer program 511 is executed by a processor, it performs the following steps:

[0135] Obtain real-time article data corresponding to policy websites;

[0136] The real-time article data is cleaned to obtain N article data, where N is a positive integer greater than or equal to 1;

[0137] The N article data are input into a policy text classification model to determine whether a target policy article exists among the N article data. The policy text classification model is pre-trained on a training sample set using TEXTCNN. The policy text classification model is used to determine whether the current article is a policy article. The training sample set includes multiple policy article data and multiple non-policy article data. TEXTCNN is a convolutional neural network used for text classification.

[0138] If the target policy article exists among the N article data, then the industry tag corresponding to the target policy article is determined.

[0139] In practical implementation, when the computer program 511 is executed by the processor, it can achieve the following: Figure 1 Any of the corresponding implementation methods in the embodiments.

[0140] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0141] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] This application also provides a computer program product, which includes computer software instructions that, when executed on a processing device, cause the processing device to perform actions such as... Figure 1 The process in the corresponding embodiment.

[0146] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0147] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A policy data processing method characterized by, The method comprises the following steps: acquiring real-time article data corresponding to a policy website; performing data cleaning on the real-time article data to obtain N article data, wherein N is a positive integer greater than or equal to 1; inputting the N article data into a policy text classification model to determine whether there is a target policy article in the N article data, wherein the policy text classification model is obtained by training a training sample set by using a TEXTCNN in advance, the policy text classification model is used to determine whether a current article is a policy article, the training sample set comprises a plurality of policy article data and a plurality of non-policy article data, and the TEXTCNN is a convolutional neural network used for text classification; if the target policy article exists in the N article data, determining an industry label corresponding to the target policy article comprises: performing paragraph division on the target policy article based on a line break symbol; determining a target paragraph containing policy content in a paragraph corresponding to the target article; and determining an industry label corresponding to the target paragraph based on an industry classification model. When the target policy article is classified by industry, the paragraphs about policy content are filtered from the text of the policy, the target policy article is divided into paragraphs based on the line break symbol, the target paragraph about policy content is found based on a policy content classification model, the paragraphs about policy content are obtained, and then the industry classification model is used to label the paragraphs with industry labels; based on the result of the model, the policy is labeled with a more accurate industry label; wherein, a first-level, a second-level and a third-level label are defined, and the target policy article is finally labeled with a third-level label based on the mapping relationship between labels of different levels.

2. The method of claim 1, wherein, The method further comprises: Step 1, acquiring the training sample set; Step 2, preprocessing the training sample set; Step 3, inputting a target training sample into the TEXTCNN to determine an output result corresponding to the target training sample, wherein the output result is whether the target training sample is a policy article, and the target training sample is any one of the training sample set after preprocessing; Step 4, adjusting a loss function corresponding to the TEXTCNN based on the output result and a label of the target training sample; Step 5, iteratively executing steps 3 to 4 based on the adjusted loss function until a preset iteration termination condition is reached; Step 6, determining the TEXTCNN when the preset iteration termination condition is reached as the policy text classification model.

3. The method of claim 2, wherein, The method further comprises: determining whether the number of iterations reaches a preset value, and if so, determining that the preset iteration termination condition is met; or, determining whether the loss function converges, and if so, determining that the preset iteration termination condition is met.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: predicting a technical field corresponding to the target policy article; determining an industry label corresponding to the target policy article according to the technical field and a mapping relationship between industry labels of different levels of the industry constructed in advance.

5. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: acquiring the real-time article data corresponding to the policy website comprises: grabbing the real-time article data from the policy website through robotic process automation (RPA).

6. The method according to any one of claims 1 to 3, characterized in that, data cleaning is performed on the real-time article data to obtain N article data, including: removing HTML tags in the real-time article data; performing special symbol processing on the real-time article data after removing the HTML tags to obtain the N article data.

7. A policy data processing apparatus characterized by comprising: comprising: an acquisition unit configured to acquire real-time article data corresponding to a policy website; a data cleaning unit configured to perform data cleaning on the real-time article data to obtain N article data, wherein N is a positive integer greater than or equal to 1; a classification unit configured to input the N article data into a policy text classification model to determine whether a target policy article exists in the N article data, wherein the policy text classification model is obtained by pre-training a training sample set using a TEXTCNN, the policy text classification model is used to determine whether a current article is a policy article, the training sample set includes a plurality of policy article data and a plurality of non-policy article data, and the TEXTCNN is a convolutional neural network used for text classification; a determination unit configured to determine an industry label corresponding to the target policy article if the target policy article exists in the N article data; the determination unit is specifically configured to perform paragraph division on the target policy article based on a line feed character, determine a target paragraph containing policy content in a paragraph corresponding to the target article, and determine an industry label corresponding to the target paragraph based on an industry classification model when determining the industry label corresponding to the target policy article. the determination unit is specifically configured to filter out a paragraph about policy content from a body of the policy, perform paragraph division on the target policy article based on a line feed character, find a target paragraph about policy content based on a policy content classification model, give the paragraph an industry label based on an industry classification model after obtaining the paragraph about policy content, and give the policy a more accurate industry label based on a result of the model when performing industry division on the target policy article.

8. A computer device, comprising: comprising: at least one connected processor, memory, and transceiver, wherein the memory is configured to store program code, and the processor is configured to invoke the program code in the memory to perform the steps of the policy data processing method of any one of claims 1 to 6.

9. A computer storage medium, characterized in that comprising: instructions that, when executed on a computer, cause the computer to perform the steps of the policy data processing method of any one of claims 1 to 6.

Images