Hidden keyword extraction method, terminal device and storage medium

Abstract

The present invention relates to a method for extracting latent keywords, a terminal device, and a storage medium. The method comprises the following steps: S1: setting a classification dimension and different categories contained in each classification dimension, extracting text from forensic data according to each category under each classification dimension, and constructing text libraries under different categories; S2: performing word segmentation processing on the text in the text library under each category, and constructing the results of the word segmentation processing into phrase libraries under each category; S3: filtering the words in each phrase library; S4: calculating the similarity between each word in the phrase library and known keywords, and eliminating words with similarity less than a similarity threshold; S5: calculating the weight of each word in the phrase library in the forensic data, and eliminating words with weight less than a weight threshold; S6: obtaining latent keywords under each category based on the phrase library processed through the above steps. The present invention realizes the automatic mining of latent keywords involved in a case in massive forensic data.

Description

Hidden keyword extraction method, terminal device and storage medium Technical Field

[0001] The present invention relates to the technical field of evidence collection, and in particular to a hidden keyword extraction method, terminal equipment and storage medium. Background Art

[0002] With the rapid development of mobile Internet technology, mobile phone forensic data is appearing more and more in various types of case work. Analysis methods based on keywords involved in the case can often play a key supporting role in the case investigation process. Especially when there is no clear clue, the use of keywords can often quickly realize the excavation of similar case clues, make breakthroughs in the case, and greatly improve the work efficiency of the investigators.

[0003] In the use of existing keywords involved in cases, on the one hand, as the professional knowledge and anti-investigation awareness of the people involved in the case increase, they often use hidden keywords to carry out illegal activities. The frequently changing and endless new keywords involved in the case cannot be grasped in a timely manner, resulting in difficulty in discovering clues related to the case, which brings inconvenience to the investigation and crackdown of the case; on the other hand, due to the geographical characteristics of illegal activities, different types are concentrated in different regions. Therefore, the keyword library summarized by traditional analysis is difficult to adapt to the situation in various parts of the country and the development of the times. Summary of the Invention

[0004] In order to solve the above problems, the present invention proposes a hidden keyword extraction method, a terminal device and a storage medium.

[0005] The specific plan is as follows:

[0006] A method for extracting latent keywords comprises the following steps:

[0007] S1: Set the classification dimensions and the different categories contained in each classification dimension, extract text from the forensic data according to the categories under each classification dimension, and build a text library under different categories;

[0008] S2: Perform word segmentation on the texts in the text library under each category, and organize the results of the word segmentation into a phrase library under each category;

[0009] S3: Filter the words in each phrase library;

[0010] S4: Based on the phrase library processed in step S3, for each word in the phrase library, similarity is calculated between the word and the known keywords in the corresponding category of the phrase library, and words with similarity less than a similarity threshold are removed from the phrase library;

[0011] S5: Based on the phrase library processed in step S4, for each word in the phrase library, the weight of each word in the forensic data is calculated using the IF-IDF algorithm, and words with weights less than a weight threshold are removed from the phrase library;

[0012] S6: Obtain the implicit keywords under each category based on the phrase library processed in step S5.

[0013] Furthermore, the classification dimensions include case type, crime area, registered residence of the persons involved, ethnicity of the persons involved, application type and collection time.

[0014] Furthermore, the forensic data is communication data, and the source of the communication data is one or more of mobile phone text messages, instant messaging content, email, Weibo, Tieba and data law vocabulary.

[0015] Furthermore, before constructing the text library in step S1, the text used to construct the text library is preprocessed. The preprocessing includes: deduplication processing, invalid data removal processing, and converting semi-structured or structured data into structured data in text format.

[0016] Furthermore, the filtering process in step S3 includes the following steps:

[0017] S31: performing stop word filtering on the words in the phrase library to remove the stop words in the phrase library;

[0018] S32: Based on the phrase library processed in step S31, perform part-of-speech filtering on the words in the phrase library according to the part of speech of each word, and remove words corresponding to unnecessary parts of speech in the phrase library;

[0019] S33: Based on the phrase library processed in step S32, words in the phrase library that exist in the white-name word library are removed according to the white-name word library constructed for storing common keywords;

[0020] S34: Based on the phrase library processed in step S33, all the words in the phrase library are Gaussian distributed according to the word frequency of each word. According to the distribution interval corresponding to the implicit keyword, the words corresponding to the distribution interval are extracted from the Gaussian distribution results, and other words are removed from the phrase library.

[0021] Furthermore, the method for calculating the similarity in step S4 is: after calculating the word vector of each word using the word2vec algorithm, the distance between the word vectors of two words is used as the similarity between the two words.

[0022] Furthermore, step S6 is specifically as follows: using the phrase library processed by step S5 as the phrase library corresponding to the current time period, performing an intersection operation on it and the keyword library corresponding to the historical time period, and using the words in the difference between the phrase library corresponding to the current time period and the intersection operation result as the implicit keywords under the corresponding category of the phrase library.

[0023] A terminal device for extracting hidden keywords includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described above are implemented.

[0024] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described above in an embodiment of the present invention.

[0025] The present invention adopts the above technical solution to realize the automatic mining of hidden keywords involved in the case in massive forensic data. The new keywords involved in the case are continuously iterated and updated, and the keyword library is continuously accumulated. The method for discovering hidden keywords has a strong self-learning ability and is highly adaptable to future changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is a flow chart showing a first embodiment of the present invention. DETAILED DESCRIPTION

[0027] To further illustrate various embodiments, the present invention provides accompanying drawings. These drawings form part of the present disclosure and are primarily used to illustrate the embodiments and, in conjunction with the relevant description in the specification, to explain the operating principles of the embodiments. By referring to these drawings, those skilled in the art will be able to understand other possible implementations and the advantages of the present invention.

[0028] The present invention will now be further described with reference to the accompanying drawings and specific embodiments.

[0029] Example 1:

[0030] An embodiment of the present invention provides a method for extracting latent keywords, as shown in FIG1 , the method comprising the following steps:

[0031] S1: Set the classification dimensions and the different categories contained in each classification dimension, extract text from the forensic data according to the categories under each classification dimension, and build a text library under different categories.

[0032] The classification dimensions set in this embodiment include case type, crime location, registered residence of the person involved, ethnicity of the person involved, application type, and collection time. The categories under each classification dimension, such as the category corresponding to the application type, include WeChat, QQ, email, etc.

[0033] The evidence-taking data is selected as communication data, and the sources of communication data include mobile phone text messages, instant chat content, email, Weibo, Tieba, and data legal term libraries, etc.

[0034] Since the extracted text may contain duplicate and invalid content, and may also contain semi-structured or unstructured data such as voice, pictures, documents, etc., therefore, in this embodiment, before constructing the text library, it also includes preprocessing the text used to construct the text library. The preprocessing includes: deduplication processing, removing invalid data (such as empty text, pure emojis, links, red envelopes, system messages, etc.), and converting semi-structured or structured data into text-formatted structured data. The conversion process of structured data can be implemented using existing artificial intelligence algorithms and will not be elaborated here.

[0035] S2: Perform word segmentation on the text in the text libraries under each category, and form a phrase library under each category with the results of the word segmentation.

[0036] The word segmentation can be performed using existing artificial intelligence algorithms and will not be elaborated here.

[0037] S3: Filter the words in each phrase library.

[0038] The filtering process is used to narrow the scope of implicit keywords, that is, to exclude words that cannot be implicit keywords. In this embodiment, the filtering process includes the following steps:

[0039] S31: Perform stop word filtering on the words in the phrase library, and remove the stop words in the phrase library.

[0040] Stop words such as "de", "shi", "he", and the specific content can be stored in the way of pre-constructing a stop word library.

[0041] S32: Based on the phrase library processed in step S31, according to the词性 of each word, perform词性 filtering on the words in the phrase library, and remove the words corresponding to the词性 that are not needed in the phrase library.

[0042] For example, only retain words with词性 such as nouns, verbs, quantifiers, adjectives, etc.

[0043] S33: Based on the phrase library processed in step S32, according to the white list word library constructed for storing common keywords, remove the words in the phrase library that exist in the white list word library.

[0044] The white list word library can be constructed by performing word frequency statistics through the same type analysis method of historical evidence-taking data, combined with manual supplementation and data provided by third parties.

[0045] S34: Based on the phrase library processed in step S33, all the words in the phrase library are Gaussian distributed according to the word frequency of each word. According to the distribution interval corresponding to the implicit keyword, the words corresponding to the distribution interval are extracted from the Gaussian distribution results, and other words are removed from the phrase library.

[0046] Hidden keywords in cases, or what are traditionally referred to as "jargon" or "slang" involved in the case, are often low-frequency words in the entire communication content. In this embodiment, through verification of a large number of segmented word phrases, it was found that the frequency statistics of segmented word phrases in the communication content conform to a Gaussian distribution. Based on the characteristics of the Gaussian distribution, by substituting dimensions such as case type, crime area, registered residence of the person involved, ethnicity of the person involved, application type, and collection time into corresponding groups to calculate the distribution range of hidden keywords, most normal keywords can be excluded.

[0047] S4: Based on the phrase library processed in step S3, for each word in the phrase library, similarity is calculated between it and known keywords in the corresponding category of the phrase library, and words with similarity less than a similarity threshold are removed from the phrase library.

[0048] The similarity calculation method in this embodiment is: after calculating the word vector of each word using the word2vec algorithm, the distance between the word vectors of two words is used as the similarity between the two words. Known keywords of different categories can be manually collected and stored in the form of a known keyword library.

[0049] Those skilled in the art can set the similarity threshold as needed, and it is not limited here.

[0050] S5: Based on the phrase library processed in step S4, for each word in the phrase library, the weight of each word in the forensic data is calculated using the IF-IDF algorithm, and words in the phrase library whose weights do not meet the weight range are eliminated.

[0051] Combined with the proportion of each word in the entire phrase library, the IF-IDF algorithm is used to further calculate the weight of each word. By setting the weight threshold, the result range is further narrowed. Staff can further analyze and judge the credibility of keywords based on actual business scenarios.

[0052] The weight range can be set by those skilled in the art based on experience and test results, and is not limited here.

[0053] S6: Obtain the implicit keywords under each category based on the phrase library processed in step S5.

[0054] Since the keywords involved in the case will change with the characteristics and means of the illegal behavior and the anti-detection awareness of the people involved, the keywords have a strong timeliness. Using keyword set operations based on dynamic time windows to cross-match the hidden keywords involved in the case can maximize the elimination of interference from the time dimension.

[0055] In this embodiment, the phrase library processed in step S5 is used as the phrase library corresponding to the current time period. An intersection operation is performed on the phrase library with the keyword library corresponding to the previous time period. The words in the difference between the phrase library corresponding to the current time period and the intersection operation result are used as implicit keywords under the corresponding category of the phrase library. These implicit keywords are newly appeared in the current time period. In the next time period, the implicit keywords of the current time period can be added to the keyword library corresponding to the previous time period, and then the implicit keywords corresponding to the next time period can be extracted.

[0056] The embodiment of the present invention performs fine-grained segmentation of the communication content of forensic data into dimensions such as case type, crime area, place of residence, ethnicity, application type, and collection time, and automatically segments the grouped communication content. By combining multiple screening strategies and Gaussian distribution, it greatly reduces interference data and uses corresponding algorithms to implement weight calculation. It can process massive forensic data in real time and automatically discover hidden keywords involved in the case, without the need for human intervention in the entire processing link.

[0057] This embodiment of the present invention supports all case types and communication application types, and can adapt to phrases, special codes, and other factors without being constrained by any conditions. It is highly adaptable to the characteristics and patterns of crime in different regions. Newly discovered hidden keywords can be further incorporated into the case keyword library for continuous iterative updates and improvements, allowing for subsequent analysis and clue mining, thereby improving overall work efficiency.

[0058] Example 2:

[0059] The present invention also provides a terminal device for extracting hidden keywords, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiment of the first embodiment of the present invention are implemented.

[0060] Furthermore, as an executable solution, the latent keyword extraction terminal device can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The latent keyword extraction terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that the composition structure of the above-mentioned latent keyword extraction terminal device is merely an example of a latent keyword extraction terminal device and does not constitute a limitation on the latent keyword extraction terminal device. It may include more or fewer components than the above-mentioned components, or a combination of certain components, or different components. For example, the latent keyword extraction terminal device may also include input and output devices, network access devices, buses, etc., and the embodiments of the present invention do not limit this.

[0061] Furthermore, as an executable solution, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the latent keyword extraction terminal device, and utilizes various interfaces and lines to connect various parts of the entire latent keyword extraction terminal device.

[0062] The memory can be used to store the computer programs and / or modules. The processor implements the various functions of the hidden keyword extraction terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created based on the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0063] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method in the embodiment of the present invention are implemented.

[0064] If the module / unit integrated in the terminal device for extracting hidden keywords is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) and software distribution medium, etc.

[0065] Although the present invention has been particularly shown and described in conjunction with preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made to the present invention without departing from the spirit and scope of the invention as defined in the appended claims, and all such changes are within the scope of protection of the present invention.

Claims

1. A method for extracting latent keywords, characterized in that: The following steps are involved: S1: Set the classification dimensions and the different categories contained in each classification dimension, extract text from the forensic data according to the categories under each classification dimension, and build a text library under different categories; S2: Perform word segmentation on the texts in the text library under each category, and organize the results of the word segmentation into a phrase library under each category; S3: Filter the words in each phrase library. The filtering process includes the following steps: S31: performing stop word filtering on the words in the phrase library to remove the stop words in the phrase library; S32: Based on the phrase library processed in step S31, perform part-of-speech filtering on the words in the phrase library according to the part of speech of each word, and remove words corresponding to unnecessary parts of speech in the phrase library; S33: Based on the phrase library processed in step S32, words in the phrase library that exist in the white-name word library are removed according to the white-name word library constructed for storing common keywords; S34: Based on the phrase library processed in step S33, perform Gaussian distribution on all words in the phrase library according to the frequency of each word. According to the distribution interval corresponding to the implicit keyword, extract the words corresponding to the distribution interval from the Gaussian distribution result, and remove other words from the phrase library; S4: Based on the phrase library processed in step S3, for each word in the phrase library, similarity is calculated between the word and the known keywords in the corresponding category of the phrase library, and words with similarity less than a similarity threshold are removed from the phrase library; S5: Based on the phrase library processed in step S4, for each word in the phrase library, the weight of each word in the forensic data is calculated using the IF-IDF algorithm, and words in the phrase library that do not meet the weight range are removed; S6: Based on the phrase library processed by step S5, the implicit keywords under each category are obtained. Specifically, the phrase library processed by step S5 is used as the phrase library corresponding to the current time period, and an intersection operation is performed on it with the keyword library corresponding to the historical time period. The words in the difference between the phrase library corresponding to the current time period and the intersection operation result are used as the implicit keywords under the category corresponding to the phrase library.

2. The method for extracting latent keywords according to claim 1, wherein: The classification dimensions include case type, crime area, registered residence of the persons involved, ethnicity of the persons involved, application type and collection time.

3. The method for extracting latent keywords according to claim 1, wherein: The forensic data is communication data, and the source of the communication data is one or more of mobile phone text messages, instant chat content, email, Weibo, Tieba and data law vocabulary.

4. The method for extracting latent keywords according to claim 1, wherein: Before building the text library in step S1, the text used to build the text library is preprocessed. The preprocessing includes: deduplication processing, invalid data removal processing, and converting semi-structured or structured data into structured data in text format.

5. The method for extracting latent keywords according to claim 1, wherein: The method for calculating the similarity in step S4 is: after calculating the word vector of each word using the word2vec algorithm, the distance between the word vectors of two words is used as the similarity between the two words.

6. A terminal device for extracting hidden keywords, characterized by: The method comprises a processor, a memory, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 5 when executing the computer program.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images