Method and device for detecting data compliance based on word vector model
By using an automated detection method based on word vector models, and calculating cosine similarity using word vector dictionaries and violation word database vector spaces, the problem of low efficiency and insufficient accuracy of manual review is solved, achieving efficient and low-cost data compliance detection.
Patent Information
- Application Number
- CN202411102378.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-15
AI Technical Summary
In existing technologies, data compliance testing mainly relies on manual review, which leads to low efficiency, high cost, and results that are affected by personal bias, making it difficult to meet the needs of large-scale data processing.
By employing a word vector model-based approach, a word vector dictionary and a violation word library vector space are constructed, and the cosine similarity between the target data and the violation word library is calculated to achieve automated data compliance detection.
It improved the accuracy of data compliance testing, reduced manual workload, lowered costs, and enhanced the objectivity and consistency of testing.
Smart Images

Figure CN118966205A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computers, and specifically, to a method and device for detecting data compliance based on a word vector model. Background Art
[0002] With the development of Internet technology, the scale of enterprise data continues to increase, the scale of data processing is also getting larger and larger, and data compliance testing is becoming more and more important.
[0003] In the prior art, data compliance detection is mainly carried out through manual review, that is, the reviewer manually reviews the data to determine whether the data complies with relevant regulations. However, manual review takes a lot of time and human resources, especially for large-scale data sets, the cost is relatively high, and enterprises or organizations need to allocate a large amount of human resources to conduct manual review; and because human thinking is subjective, the same data may have different results when tested by different reviewers, and the test results may be affected by the personal bias and subjective consciousness of the reviewer, resulting in the review results being not objective and consistent; as the amount of data increases, the efficiency and accuracy of manual review may be limited, and it is difficult to meet the needs of large-scale data processing. Summary of the invention
[0004] The embodiments of the present application provide a method and device for detecting data compliance based on a word vector model, so as to at least solve the problem of low detection accuracy in the method for detecting data compliance in the related art.
[0005] According to one embodiment of the present application, a method for detecting data compliance based on a word vector model is provided, comprising: inputting target data into a target word vector model to obtain a target word vector of the target data, wherein the target word vector model includes a word vector dictionary and an illegal word library vector space, the word vector dictionary is a data structure including words and corresponding vector representations, and is used to convert the target data into vector representations, and the illegal word library vector space is used to map illegal words to vector space; calculating the cosine similarity between the target word vector and the illegal word vector in the illegal word library vector space, wherein the cosine similarity is used to represent the degree of similarity between the target word vector and the illegal word vector in the illegal word library vector space; and detecting the compliance of the target data based on the cosine similarity.
[0006] In an exemplary embodiment, the target word vector model is determined in the following manner: constructing a violation word library, wherein the violation word library includes multiple historical words that violate data compliance; constructing a corpus, wherein the corpus includes multiple historical data that have been tested for compliance; performing data processing on the multiple historical data in the corpus to obtain a target corpus; and determining the target word vector model using the violation word library and the target corpus.
[0007] In an exemplary embodiment, data processing is performed on multiple historical data in the corpus to obtain a target corpus, including: traversing the multiple historical data in the corpus to perform regular processing on the multiple historical data using regular expressions, and performing word segmentation processing on the processed data to obtain a first corpus; using a stop word list to remove stop words in the first corpus to obtain a second corpus, wherein the stop word list is a set of words that are excluded and not analyzed and processed; performing part-of-speech tagging, word frequency statistics, and keyword extraction on the data in the second corpus to obtain the target corpus.
[0008] In an exemplary embodiment, the word vector model is trained using the illegal word library and the target corpus to obtain the target word vector model, including: inputting the illegal word library into the word vector model to convert the illegal data in the illegal word library into illegal vectors using the word vector model to obtain the target illegal word library, wherein the word vector model is a word embedding model that maps words to a high-dimensional space; loading multiple word vectors from the target illegal word library to construct the word vector dictionary; setting both the word vector dictionary and the target illegal word library into the word vector model to obtain the target word vector model.
[0009] In an exemplary embodiment, before inputting the target data into the target word vector model to obtain the target word vector of the target data, the method further includes: using a regular expression to perform regular processing on the acquired first data, and performing word segmentation processing on the processed first data to obtain second data; using a stop word list to remove stop words in the second data to obtain third data, wherein the stop word list is a set of words that are excluded and not analyzed and processed; performing part-of-speech tagging, word frequency statistics, and keyword extraction on the third data to obtain the target data.
[0010] In an exemplary embodiment, calculating the cosine similarity between the target word vector and the illegal word vector in the illegal word library vector space includes: calculating the dot product and the modulus between the target word vector and the illegal word vector; and calculating the cosine similarity using the ratio between the dot product and the modulus.
[0011] In an exemplary embodiment, detecting compliance of the target data based on the cosine similarity includes: when the cosine similarity is greater than a first threshold and less than a second threshold, determining that the probability of violation of the target data is less than or equal to a first preset probability, and obtaining a first detection result, wherein the first detection result is used to indicate that the violation of the target data is in a pending state; and when the cosine similarity is greater than the second threshold, determining that the probability of violation of the target data is greater than the first preset probability, and obtaining a second detection result, wherein the second detection result is used to indicate that the violation of the target data is in a confirmed state.
[0012] In an exemplary embodiment, when the cosine similarity is greater than a first threshold and less than a second threshold, it is determined that the probability of violation of the target data is less than or equal to a first preset probability, and after obtaining a first detection result, the method further includes: performing a weighted voting operation on the first detection result to obtain an operation result; and determining whether the target data is in violation based on the operation result.
[0013] According to another embodiment of the present application, a device for detecting data compliance based on a word vector model is provided, including: a first input model, used to input target data into a target word vector model to obtain a target word vector of the target data, wherein the target word vector model includes a word vector dictionary and an illegal word library vector space, the word vector dictionary is a data structure including words and corresponding vector representations, used to convert the target data into vector representations, and the illegal word library vector space is used to map illegal words to the vector space; a first calculation model, used to calculate the cosine similarity between the target word vector and the illegal word vector in the illegal word library vector space, wherein the cosine similarity is used to represent the degree of similarity between the target word vector and the illegal word vector in the illegal word library vector space; a first detection model, used to detect the compliance of the target data based on the cosine similarity.
[0014] According to another embodiment of the present application, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps of any of the methods described above are implemented.
[0015] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the steps of any of the methods described above are performed when the computer program is executed by a processor.
[0016] According to another embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of any of the methods described above when executing the computer program.
[0017] Through this application, the target data is input into the target word vector model to obtain the target word vector of the target data, wherein the target word vector model includes a word vector dictionary and an illegal word library vector space, the word vector dictionary is a data structure including words and corresponding vector representations, used to convert the target data into vector representations, and the illegal word library vector space is used to map illegal words to the vector space; the cosine similarity between the target word vector and the illegal word vector in the illegal word library vector space is calculated, wherein the cosine similarity is used to represent the degree of similarity between the target word vector and the illegal word vector in the illegal word library vector space; the compliance of the target data is detected based on the cosine similarity. In other words, this application detects data compliance through a word vector model, and detects data compliance through cosine similarity, which can effectively improve the detection accuracy, and solves the problem of low detection accuracy in the method for detecting data compliance in the related art. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a hardware structure block diagram of a mobile terminal of a method for detecting data compliance based on a word vector model according to an embodiment of the present application;
[0019] Figure 2 is a flowchart of a method for detecting data compliance based on a word vector model according to an embodiment of the present application;
[0020] Figure 3 is a flow chart of using a target model to detect data compliance according to a specific embodiment of the present application;
[0021] Figure 4 is a schematic diagram of a conversion word vector according to an embodiment of the present application;
[0022] Figure 5 is a flow chart of calculating cosine similarity according to an embodiment of the present application;
[0023] Figure 6 It is a structural block diagram of a device for detecting data compliance based on a word vector model according to an embodiment of the present application. DETAILED DESCRIPTION
[0024] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0026] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 11 is a hardware structure block diagram of a mobile terminal of a method for detecting data compliance based on a word vector model in an embodiment of the present application. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0027] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for detecting data compliance based on the word vector model in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.
[0028] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0029] In this embodiment, a method for detecting data compliance based on a word vector model is provided. Figure 2 is a flowchart of a method for detecting data compliance based on a word vector model according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:
[0030] Step S202, inputting the target data into a target word vector model to obtain a target word vector of the target data, wherein the target word vector model includes a word vector dictionary and an illegal word library vector space, the word vector dictionary is a data structure including words and corresponding vector representations, and is used to convert the target data into a vector representation, and the illegal word library vector space is used to map illegal words to a vector space;
[0031] Step S204, calculating the cosine similarity between the target word vector and the illegal word vector in the illegal word library vector space, wherein the cosine similarity is used to indicate the degree of similarity between the target word vector and the illegal word vector in the illegal word library vector space;
[0032] Step S206: detecting compliance of the target data based on the cosine similarity.
[0033] The execution subject of the above steps may be a terminal, a server, a specific processor provided in the terminal or the server, or a processor or processing device provided relatively independently from the terminal or the server, but is not limited thereto.
[0034] Through the above steps, the target data is input into the target word vector model to obtain the target word vector of the target data, wherein the target word vector model includes a word vector dictionary and an illegal word library vector space, the word vector dictionary is a data structure including words and corresponding vector representations, used to convert the target data into vector representations, and the illegal word library vector space is used to map illegal words to the vector space; the cosine similarity between the target word vector and the illegal word vector in the illegal word library vector space is calculated, wherein the cosine similarity is used to represent the similarity between the target word vector and the illegal word vector in the illegal word library vector space; the compliance of the target data is detected based on the cosine similarity. In other words, the present application detects data compliance through a word vector model, and detects data compliance through cosine similarity, which can effectively improve the detection accuracy, and solves the problem of low detection accuracy in the method for detecting data compliance in the related art.
[0035] Optionally, compliance testing refers to reviewing the target data to ensure that it complies with relevant laws and regulations, industry standards, company policies, and other regulations. Compliance testing usually involves reviewing the text content, format, grammar, wording, and other aspects of the target data to determine whether it complies with the specified requirements. Compliance testing can involve multiple aspects, for example, 1. Compliance with laws and regulations: Check whether the text complies with the provisions of relevant laws and regulations. 2. Compliance with industry standards: Check whether the text complies with the standard requirements of the industry to which it belongs. 3. Compliance with company policies: Check whether the text complies with the company's internal policy requirements. Compliance testing is very important to ensure the legality and standardization of the target data, which can effectively avoid illegal and irregular behaviors and standardize corporate behavior.
[0036] Optionally, by calculating the cosine similarity between the target word vector and the preset word vector, their similarity in the vector space can be evaluated. The value range of cosine similarity is between -1 and 1. The closer to 1, the more similar the two vectors are, and the closer to -1, the less similar the two vectors are. In this embodiment, by using cosine similarity to determine whether the data is compliant, it is possible to accurately determine whether the target data is compliant.
[0037] In an exemplary embodiment, the target word vector model is determined in the following manner: constructing a violation word library, wherein the violation word library includes multiple historical words that violate data compliance; constructing a corpus, wherein the corpus includes multiple historical data that have been tested for compliance; performing data processing on the multiple historical data in the corpus to obtain a target corpus; and determining the target word vector model using the violation word library and the target corpus.
[0038] Optionally, a violation word library can be constructed based on international treaties, domestic laws, regulations, other normative documents at home and abroad, industry standards, business practices, social ethics, and corporate charters, rules and regulations, etc. Alternatively, the words that violate data compliance of the enterprise and the industry are input into the word library to construct a violation word library.
[0039] Optionally, various methods can be used to collect enterprise and industry-related data sets to build a large enough corpus to help the target model better capture the co-occurrence relationship between words, improve the quality and accuracy of word vectors, and ensure the accuracy of data detection.
[0040] Optionally, multiple historical data in the corpus are processed to obtain a target corpus, including: traversing the multiple historical data in the corpus to perform regular processing on the multiple historical data using regular expressions, and performing word segmentation processing on the processed data to obtain a first corpus; using a stop word list to remove stop words in the first corpus to obtain a second corpus, wherein the stop word list is a set of words that are excluded and not analyzed and processed; performing part-of-speech tagging, word frequency statistics, and keyword extraction on the data in the second corpus to obtain a target corpus.
[0041] Jieba can be used for text segmentation. The Jieba library method can be used to traverse the text data and perform regularization and segmentation on each text. Jieba is a popular Chinese word segmentation Python library that supports three word segmentation modes: precise mode, full mode, and search engine mode. Using Jieba for text segmentation can better process and analyze Chinese text data.
[0042] Optionally, regular expressions can be used to perform regular processing on multiple historical data, and the processed data can be segmented. The steps are as follows:
[0043] 1. Regular processing: Use regular expressions to clean and process the text. For example, remove special characters, punctuation marks, numbers, etc. The re module in Python can be used for regular processing. For example, the re.sub() method can be used to replace the content that matches the regular expression.
[0044] 2. Word segmentation: Use a word segmentation tool to segment the text into individual words or phrases. Python's word segmentation tools can be used, such as the NLTK toolkit or the jieba segmentation library. For English text, spaces or punctuation marks can be directly used for word segmentation; for Chinese text, Chinese word segmentation tools are required. When performing regular processing and word segmentation, appropriate processing methods and tools need to be selected according to specific requirements and text characteristics to achieve effective processing and analysis of the text.
[0045] Optionally, a stop word list can be loaded to remove stop words. A stop word list refers to a set of words that are excluded from analysis and processing in natural language processing. These words are usually common, meaningless, or unimportant words, such as "de", "shi", "zai", "he", etc. The purpose of the stop word list is to improve the efficiency and accuracy of text processing, reduce noise and interference. Common stop word lists include English stop words and Chinese stop words, which can be customized and adjusted according to specific application fields and requirements.
[0046] In this embodiment, further processing such as part-of-speech tagging can also be performed on the word segmentation results according to actual needs. Read the file content line by line, remove the line breaks and add them to a list, and finally return a complete list of text data. This step includes operations such as loading data, regular processing, word segmentation, and removing stop words. By loading a custom dictionary and a stop word list, the text data is processed, and finally a list of segmented words is obtained, providing a basis for subsequent text analysis and processing.
[0047] In an exemplary embodiment, a word vector model is trained using a violation word library and a target corpus to obtain a target word vector model, including: inputting the violation word library into the word vector model to use the word vector model to convert the violation data in the violation word library into violation vectors, obtaining a target violation word library, where the word vector model is a word embedding model that maps words to a high-dimensional space; loading multiple word vectors from the target violation word library to construct a word vector dictionary; setting both the word vector dictionary and the target violation word library into the word vector model to obtain the target word vector model.
[0048] Optionally, the word vector model can be a Word2Vec model in the gensim library.
[0049] Optionally, the word vector dictionary is a data structure containing words and their corresponding vector representations. It is used to represent the position and semantic information of words in a vector space. The violation word library vector space is a method of mapping violation words into a vector space, which can be obtained through the word vector model.
[0050] Building the word vector dictionary and the violation word vector space can be achieved by the following steps:
[0051] S1, use pre-trained word vector models, such as Word2Vec, GloVe, etc., to obtain the vector representation of words.
[0052] S2, for illegal words, we can build an illegal word library by manually labeling and organizing them, and use the same word vector model to obtain their vector representations.
[0053] S3, stores the word vector dictionary and the violation word library vector space in a data structure, such as a dictionary or Numpy array in Python, for use in subsequent text analysis.
[0054] This embodiment constructs a word vector dictionary and an illegal word library vector space, so as to better utilize the semantic information of word vectors in text analysis tasks and more accurately identify and process illegal content.
[0055] In an exemplary embodiment, before inputting the target data into the target word vector model to obtain the target word vector of the target data, the method further includes: using a regular expression to perform regular processing on the acquired first data, and performing word segmentation processing on the processed first data to obtain second data; using a stop word list to remove stop words in the second data to obtain third data, wherein the stop word list is a set of words that are excluded and not analyzed and processed; performing part-of-speech tagging, word frequency statistics, and keyword extraction on the third data to obtain target data.
[0056] Optionally, this embodiment traverses the first data, and can use the cut method of the Jieba library to process the text, perform regularization and word segmentation on each text, and then load the stop word list to remove stop words. The word segmentation results can also be further processed by part-of-speech tagging according to actual needs. Then input it into the trained word2vec model to obtain the word vector of the text vocabulary in the sample to be tested. The specific process of regularization and word segmentation of the first data in this embodiment is the same as the above-mentioned processing of historical data, which will not be repeated here.
[0057] In an exemplary embodiment, calculating the cosine similarity between a target word vector and a violation word vector in a violation word library vector space includes: calculating the dot product and the norm between the target word vector and the violation word vector; calculating the cosine similarity using the ratio between the dot product and the norm.
[0058] Optionally, the comparison formula for the cosine similarity is as follows: Where A is used to represent the violation word vector in the constructed compliance word library, B is used to represent the target word vector, i is used to represent the serial number of the violation word vector in the violation word library vector space, and n is used to represent the number of calculations.
[0059] In an exemplary embodiment, detecting the compliance of target data based on the cosine similarity includes: when the cosine similarity is greater than a first threshold and less than a second threshold, determining that the violation probability of the target data is less than or equal to a first preset probability, and obtaining a first detection result, where the first detection result is used to indicate that the violation of the target data is in a pending determination state; when the cosine similarity is greater than the second threshold, determining that the violation probability of the target data is greater than the first preset probability, and obtaining a second detection result, where the second detection result is used to indicate that the violation of the target data is in a confirmed state.
[0060] Optionally, the first threshold is used to define the critical point between suspected violation and violation, and the second threshold is used to define the critical point between compliance and suspected violation. For example, if the first threshold is T1 and the second threshold is T2, then -1 < T2 < T1 < 1. T1 and T2 can be specified by relevant enterprise management personnel according to the relevant situation of the enterprise, and T1 and T2 can be changed according to the strictness of the enterprise's data compliance requirements and the workload of allocated manual review. For example, if an enterprise has high requirements for data compliance and sufficient personnel, T1 and T2 can be increased simultaneously to maximize the accuracy of data compliance detection. If an enterprise is short of manpower, T1 can be reduced to increase the machine workload and reduce the manual workload, which meets the needs of the enterprise.
[0061] Optionally, after determining that the violation probability of the target data is less than or equal to the first preset probability and obtaining the first detection result when the cosine similarity is greater than the first threshold and less than the second threshold, the above method further includes: performing a weighted voting operation on the first detection result to obtain an operation result; determining whether the target data is in violation based on the operation result.
[0062] Optionally, weighted voting is a truth inference method based on voting. It assigns different weights to the votes of workers according to indicators such as the reliability of workers or the difficulty of tasks, and then selects the label with the largest sum of weights as the truth. The formula for weighted voting is shown in Formula 2: Among them, Y is used to represent the output category of the fusion model, C is two possible categories, namely compliance and non-compliance, n is the number of evaluators, which is the total number of people participating in the crowdsourcing vote, wi is the weight of the i-th evaluator, yi is the output category of the i-th evaluator, and I is an indicator function, which is 1 when the conditions in the brackets are met, otherwise it is 0. In addition, some experts who understand data compliance specifications or relevant industry staff and professional data compliance reviewers are added to the crowdsourcing workers. The votes of compliance reviewers, compliance specification experts, industry staff, and ordinary workers are weighted in descending order, and finally the final voting results are weighted and summed to find out whether the label value is in violation. If it is in violation, it is marked.
[0063] The present application is described below in conjunction with specific embodiments:
[0064] This specific embodiment takes the detection of data compliance in power companies as an example to illustrate. By building a violation word library and vectorizing it, a training data set is established, a word vector model is trained and a word vector dictionary is built to achieve preprocessing of the data set text and training of the word vector model. Subsequently, the text data to be detected (target data) is preprocessed, compared with the word vectors in the violation word library, and a similarity threshold is set to automatically determine or manually mark suspected violation words, and finally the violation items are output. Figure 3 As shown in the figure, it is the process of training the target word vector model and using the target model to detect data compliance, which specifically includes the following steps:
[0065] S301, construct a violation word library: construct a violation word library based on international treaties, domestic laws, regulations and rules, other domestic and foreign normative documents, industry standards, business practices, social ethics, and corporate charters, rules and regulations, etc., input the words that violate data compliance of the enterprise and the industry into the word library to construct a violation word library.
[0066] S302, establish training data set: collect enterprise and industry related data sets, build prediction library, regularize data set text, perform pre-processing such as word segmentation, and establish training data set. Specifically include: first, use various methods to collect enterprise and industry related data sets, build a large enough corpus to help the model better capture the co-occurrence relationship between words, improve the quality and accuracy of word vectors, and ensure the accuracy of data detection. Use Jieba to perform text word segmentation, use the method of Jieba library to traverse text data, and perform regular processing and word segmentation on each text. Then load the stop word list and remove the stop words. Finally, the word segmentation result can be further processed such as part of speech tagging according to actual needs, read the file content line by line, remove the line break and add it to the list, and return a complete text data list. This step includes operations such as loading data, regular processing, word segmentation and removing stop words. By loading a custom dictionary and a stop word list, the text data is processed, and a text list after word segmentation is finally obtained, which provides a basis for subsequent text analysis and processing.
[0067] S303, training the word vector model and building a word vector dictionary. Figure 4 As shown, the illegal word library constructed in S301 and the enterprise and industry-related data sets collected in S302 are input into the word vector model, the word vector model is trained, and the trained word vector is saved in a file, and a word vector dictionary and an illegal word library vector space are constructed simultaneously. Optionally, the Word2Vec model in the gensim library is used to construct the Word2Vec model, specifically including: using the Word2Vec class of the gensim library to construct the word vector model, mainly in constructing the word vector model and the related dictionary. First, the Word2Vec model in the gensim library is used to train the word vector, and then the trained word vector is saved in a file. Then define the function build_vecs() function to load the word vector from the file and construct the word vector dictionary, which is then saved in a file. The word vector converted by the illegal word library is also saved in another file simultaneously.
[0068] S304, convert the data to be detected into word vector after word segmentation. The data to be detected is regularized, and the word segmentation and other processes are input into the target word vector model to obtain the word vector of the data to be detected (i.e. the target word vector of the target data). Optionally, Jieba is used to perform text segmentation, and the operation steps are the same as S302, specifically including: first the data to be detected is loaded, the text data is traversed, and then the text data is processed using the cut method of the Jieba library, and each text data is subjected to regular processing and word segmentation, and then the stop word list is loaded to remove stop words, and further processing such as part of speech tagging can also be performed to the word segmentation result according to actual needs. Then it is input into the trained word2vec model to obtain the word vector of the text vocabulary in the data to be detected.
[0069] S305. Compare the cosine similarity similarity between the word vectors in the data to be detected and the word vectors in the violation word library for compliance detection: Compare the cosine similarity similarity between the word vectors obtained from the data to be detected and the word vectors in the violation word library in S301. The comparison formula for cosine similarity is as follows: where A refers to the sensitive word vector in the constructed compliance word library, and B refers to the word vector of the word to be detected in the data to be detected.
[0070] As Figure 5 shown, the calculation process of cosine similarity includes the following steps: Set the cosine similarity compliance thresholds T1 and T2 (-1 < T2 < T1 < 1). Among them, T1 is the critical point for defining suspected violations and violations, and T2 is the critical point for defining compliance and suspected violations. If the word vector similarity in the sample >= T1, it is determined that the data is in violation, and this data is marked as a violation in the test sample, and the vocabulary in the matching violation word library is output. If T2 < similarity < T1 in the sample, the word is determined to be a suspected violation and handed over to crowdsourcing, and the results are weighted and voted on. Add some experts who understand data compliance specifications, relevant industry staff, and professional data compliance reviewers among the crowdsourcing workers. Assign weights to their votes in descending order for compliance reviewers, compliance specification experts, industry staff, and ordinary workers. Finally, whether the label value is in violation is obtained by weighted summation of the final voting results. If it is in violation, it is marked.
[0071] S306. Output the compliance detection result: After comparing the sample to be detected, if each item of data in the sample meets the compliance requirements, it is determined that the data is compliant and passes the compliance detection of the data. If there are violation items in the sample to be detected, the test sample with the data violation items already marked is output to facilitate the enterprise and users to check and correct in time to avoid compliance risks.
[0072] This specific embodiment can not only reduce the manual workload, reduce labor costs and enterprise costs, improve the automation level, intelligence level and work efficiency, but also help the enterprise and users to check and correct in time to avoid compliance risks.
[0073] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0074] In this embodiment, a device for detecting data compliance based on a word vector model is also provided, which is used to implement the above embodiments and preferred implementation modes, and will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0075] Figure 6 is a structural block diagram of a device for detecting data compliance based on a word vector model according to an embodiment of the present application, such as Figure 6 As shown, the device comprises:
[0076] A first input model 62 is used to input the target data into the target word vector model to obtain the target word vector of the target data, wherein the target word vector model includes a word vector dictionary and an illegal word library vector space, the word vector dictionary is a data structure including words and corresponding vector representations, and is used to convert the target data into a vector representation, and the illegal word library vector space is used to map illegal words to a vector space;
[0077] A first calculation model 64 is used to calculate the cosine similarity between the target word vector and the illegal word vector in the illegal word library vector space, wherein the cosine similarity is used to indicate the similarity between the target word vector and the illegal word vector in the illegal word library vector space;
[0078] The first detection model 66 is used to detect the compliance of the target data based on cosine similarity.
[0079] In an exemplary embodiment, the target word vector model is determined in the following manner: constructing a violation word library, wherein the violation word library includes multiple historical words that violate data compliance; constructing a corpus, wherein the corpus includes multiple historical data that have been tested for compliance; performing data processing on the multiple historical data in the corpus to obtain a target corpus; and determining the target word vector model using the violation word library and the target corpus.
[0080] In an exemplary embodiment, the above-mentioned device processes multiple historical data in a corpus in the following manner to obtain a target corpus: traverse multiple historical data in the corpus to perform regular processing on the multiple historical data using regular expressions, and perform word segmentation processing on the processed data to obtain a first corpus; use a stop word list to remove stop words in the first corpus to obtain a second corpus, wherein the stop word list is a set of words that are excluded and not analyzed and processed; perform part-of-speech tagging, word frequency statistics, and keyword extraction on the data in the second corpus to obtain a target corpus.
[0081] In an exemplary embodiment, the above-mentioned device uses the illegal vocabulary and the target corpus to train a word vector model to obtain a target word vector model in the following manner: input the illegal vocabulary into the word vector model to use the word vector model to convert the illegal data in the illegal vocabulary into illegal vectors to obtain a target illegal vocabulary, wherein the word vector model is a word embedding model that maps words to a high-dimensional space; load multiple word vectors from the target illegal vocabulary to construct a word vector dictionary; set both the word vector dictionary and the target illegal vocabulary into the word vector model to obtain a target word vector model.
[0082] In an exemplary embodiment, the above-mentioned device also includes: a first processing module, which is used to input the target data into the target word vector model, and before obtaining the target word vector of the target data, use a regular expression to perform regular processing on the acquired first data, and perform word segmentation processing on the processed first data to obtain second data; a first removal module, which is used to remove stop words in the second data using a stop word list to obtain third data, wherein the stop word list is a set of words that are excluded and not analyzed and processed; a first labeling module, which is used to perform part-of-speech tagging, word frequency statistics and keyword extraction on the third data to obtain target data.
[0083] In an exemplary embodiment, the first calculation module includes: a first calculation unit, which is used to calculate the dot product and modulus between the target word vector and the offending word vector; and a second calculation unit, which is used to calculate the cosine similarity using the ratio between the dot product and the modulus.
[0084] In an exemplary embodiment, the first detection module includes: a first determination unit, configured to determine that the probability of violation of the target data is less than or equal to a first preset probability when the cosine similarity is greater than a first threshold value and less than a second threshold value, and obtain a first detection result, wherein the first detection result is used to indicate that the violation of the target data is in a pending determination state;
[0085] The second determination unit is used to determine that the violation probability of the target data is greater than the first preset probability when the cosine similarity is greater than the second threshold, and obtain a second detection result, wherein the second detection result is used to indicate that the violation of the target data is in a confirmed state.
[0086] In an exemplary embodiment, the above-mentioned device also includes: a second processing module, which is used to determine that the violation probability of the target data is less than or equal to a first preset probability when the cosine similarity is greater than the first threshold and less than the second threshold, and after obtaining the first detection result, perform a weighted voting operation on the first detection result to obtain an operation result; a first determination module, which is used to determine whether the target data is in violation based on the operation result.
[0087] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0088] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.
[0089] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.
[0090] The embodiments of the present application also provide a computer program, which includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any one of the above method embodiments.
[0091] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.
[0092] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0093] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0094] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0095] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.
[0096] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.
[0097] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.
Claims
1. A method for detecting data compliance based on a word vector model, characterized in that: include: Input the target data into the target word vector model to obtain the target word vector of the target data, wherein the target word vector model includes a word vector dictionary and an illegal word library vector space, the word vector dictionary is a data structure including words and corresponding vector representations, and is used to convert the target data into a vector representation, and the illegal word library vector space is used to map illegal words to a vector space; Calculating the cosine similarity between the target word vector and the illegal word vector in the illegal word library vector space, wherein the cosine similarity is used to indicate the degree of similarity between the target word vector and the illegal word vector in the illegal word library vector space; The compliance of the target data is detected based on the cosine similarity.
2. The method according to claim 1, characterized in that The target word vector model is determined in the following way: Building a violation word library, wherein the violation word library includes a plurality of historical words that violate data compliance; Constructing a corpus, wherein the corpus includes a plurality of historical data that have been subjected to compliance testing; Performing data processing on the plurality of historical data in the corpus to obtain a target corpus; The target word vector model is determined using the offending word library and the target corpus.
3. The method according to claim 2, characterized in that Processing the plurality of historical data in the corpus to obtain a target corpus includes: Traversing the plurality of historical data in the corpus, performing regular processing on the plurality of historical data using regular expressions, and performing word segmentation processing on the processed data to obtain a first corpus; Using a stop word list to remove stop words in the first corpus to obtain a second corpus, wherein the stop word list is a set of words that are excluded and not analyzed and processed; Part-of-speech tagging, word frequency statistics, and keyword extraction are performed on the data in the second corpus to obtain the target corpus.
4. The method according to claim 2, characterized in that: Using the illegal word library and the target corpus to train a word vector model to obtain the target word vector model includes: Inputting the illegal word library into a word vector model, so as to convert the illegal data in the illegal word library into illegal vectors by using the word vector model, and obtain a target illegal word library, wherein the word vector model is a word embedding model that maps words into a high-dimensional space; Loading a plurality of word vectors from the target violation word library to construct the word vector dictionary; The word vector dictionary and the target illegal word library are both set in the word vector model to obtain the target word vector model.
5. The method according to claim 1, characterized in that Before inputting the target data into the target word vector model to obtain the target word vector of the target data, the method further includes: Performing regular processing on the acquired first data using a regular expression, and performing word segmentation processing on the processed first data to obtain second data; Using a stop word list to remove stop words in the second data to obtain third data, wherein the stop word list is a set of words that are excluded and not analyzed and processed; Part-of-speech tagging, word frequency statistics, and keyword extraction are performed on the third data to obtain the target data.
6. The method according to claim 1, characterized in that Calculating the cosine similarity between the target word vector and the illegal word vector in the illegal word library vector space includes: Calculate the dot product and modulus between the target word vector and the offending word vector; The cosine similarity is calculated using the ratio between the dot product and the modulus.
7. The method according to claim 1, characterized in that Detecting compliance of the target data based on the cosine similarity includes: When the cosine similarity is greater than the first threshold and less than the second threshold, determining that the probability of violation of the target data is less than or equal to the first preset probability, and obtaining a first detection result, wherein the first detection result is used to indicate that the violation of the target data is in a pending state; When the cosine similarity is greater than the second threshold, it is determined that the violation probability of the target data is greater than the first preset probability, and a second detection result is obtained, wherein the second detection result is used to indicate that the violation of the target data is in a confirmed state.
8. The method according to claim 7, characterized in that When the cosine similarity is greater than the first threshold and less than the second threshold, it is determined that the probability of violation of the target data is less than or equal to the first preset probability, and after obtaining the first detection result, the method further includes: Performing a weighted voting operation on the first detection result to obtain an operation result; It is determined whether the target data violates a rule based on the operation result.
9. A device for detecting data compliance based on a word vector model, characterized in that: include: A first input model is used to input target data into a target word vector model to obtain a target word vector of the target data, wherein the target word vector model includes a word vector dictionary and an illegal word library vector space, the word vector dictionary is a data structure including words and corresponding vector representations, and is used to convert the target data into a vector representation, and the illegal word library vector space is used to map illegal words to a vector space; A first calculation model is used to calculate the cosine similarity between the target word vector and the illegal word vector in the illegal word library vector space, wherein the cosine similarity is used to represent the similarity between the target word vector and the illegal word vector in the illegal word library vector space; The first detection model is used to detect the compliance of the target data based on the cosine similarity.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 8 when executed by a processor.
12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 8 are implemented.
