Method and system for semantic analysis of chat records based on language model

Abstract

The invention discloses a method and system for semantic analysis of chat records based on a language model, and the method comprises the following steps: segmenting the chat records into dialogue blocks, designing an analysis template, and marking language analysis on the dialogue blocks to obtain a fine tuning training set and a fine tuning test set; after the data set is constructed, performing fine tuning training on the basic model by using fine tuning data in the fine tuning training set; and after domain fine tuning, fine tuning data in the fine tuning test set obtains an analysis result through the trained model, whether the analysis result is right or wrong is judged to obtain alignment data, the alignment data is supplemented into the fine tuning data set, and fine tuning training is performed again. According to the method and the system for carrying out semantic analysis on the chat records based on the language model, the problems that chat record analysis methods in the prior art are low in efficiency and important information is missed are solved.

Description

Technical Field

[0001] The present invention relates to a method and system for performing semantic analysis on chat records based on a language model, and belongs to the field of natural language processing. Background Art

[0002] In the electronic evidence collection of public security cases, the chat records of instant messaging tools are important data analysis objects. They not only contain electronic evidence of the current case, but also may contain other clues to the crime. At present, the analysis of these chat records is mainly carried out through keyword search and manual analysis, which is not only inefficient but also may miss important information. In recent years, general language models represented by ChatGPT have made a qualitative leap in language understanding and language generation tasks in complex scenarios. Subsequently, domain-based fine-tuning based on open source language models has achieved success in many fields such as medicine, law, and finance, providing technical reference for semantic analysis of chat records. The key to the present invention is to use open source language models to construct a model for semantic analysis of chat records. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a method and system for performing semantic analysis on chat records based on a language model, which solves the problem of low efficiency and omission of important information in the chat record analysis method in the prior art.

[0004] Technical solution: The method for semantic analysis of chat records based on a language model of the present invention comprises the following steps:

[0005] (1) Divide the chat records into dialogue blocks, and design an analysis template to annotate the language analysis on the dialogue blocks to obtain the fine-tuning training set and fine-tuning test set;

[0006] (2) Use the fine-tuning data in the fine-tuning training set to perform fine-tuning training on the base model;

[0007] (3) The fine-tuning data in the fine-tuning test set is analyzed using the trained model, and the correctness of the analysis results is judged to obtain the aligned data. The aligned data is then added to the fine-tuning data set, and fine-tuning training is performed again.

[0008] Furthermore, the step (1) specifically includes the following steps:

[0009] (11) selecting chat record segments covering common crime types from real chat records involved in criminal cases, and obtaining dialogue blocks after segmentation;

[0010] (12) Design clue inquiry questions based on clues of crime and related behavior;

[0011] (13) Design an analysis template to determine whether the dialogue block meets the retrieval conditions of the clue query question and annotate the dialogue block with the designed analysis template to perform language analysis to obtain a fine-tuning dataset.

[0012] Furthermore, the segmentation in step (11) is to pre-process the data to remove or compress texts that have little to do with semantic analysis, convert the CSV data format into a dialogue format, and segment the dialogue according to the rules to obtain dialogue blocks; the rules are to segment according to the chat interval time and the character length of the dialogue block.

[0013] Furthermore, the analysis template in step (13) specifically includes a restatement of the question, a summary of the chat content, a segmented analysis, a general analysis and a conclusion; the restatement of the question rewrites the goal of the question into a scenario that conforms to the clue extraction in electronic forensics; the chat content summary summarizes the chat content in order from top to bottom; the segmented analysis analyzes each summary in combination with the question; the general analysis conducts a comprehensive analysis of all rewritten questions, summaries and segmented analyses; and the conclusion determines whether the dialogue block meets the retrieval conditions of the question.

[0014] Among them, question restatement combines the user's input question with the electronic evidence collection scenario to generate a question that is more in line with the case; chat content summary gives the model the ability to understand chat records; segment analysis finds arguments from each segment; general analysis analyzes the overall situation; conclusion determines whether it meets the search conditions and outputs the results. When writing an analysis, the author is required to first redefine the goal of the question.

[0015] Furthermore, the step (3) specifically includes the following steps:

[0016] (31) Using the trained model to analyze the fine-tuning data of the fine-tuning test set to obtain analysis results;

[0017] (32) Manually judge the correctness of the analysis results to obtain aligned data;

[0018] (33) Designing clue query sentences for the collected aligned data, and adding the aligned data and the designed clue query sentences to the fine-tuning dataset;

[0019] (34) Use the fine-tuning data in the supplementary fine-tuning dataset to re-fine-tune the model.

[0020] After domain-specific fine-tuning, the model has learned the ability to analyze chat records, and through alignment training, the model's judgment scale is made close to that of forensic personnel.

[0021] The system for performing semantic analysis on chat records based on a language model of the present invention comprises:

[0022] The dataset construction module is used to divide the chat records into conversation blocks and design analysis templates to annotate language analysis on the conversation blocks to obtain fine-tuning training sets and fine-tuning test sets;

[0023] A fine-tuning training module is used to perform fine-tuning training on the base model using the fine-tuning data in the fine-tuning training set;

[0024] The alignment training module is used to fine-tune the fine-tuning data in the test set, obtain the analysis results through the trained model, judge whether the analysis results are correct or not to obtain the alignment data, add the alignment data to the fine-tuning data set, and perform fine-tuning training again.

[0025] Furthermore, the data set construction module specifically includes a dialogue block design unit, a question design unit and a writing and analysis unit; the dialogue block design unit screens chat record segments covering common crime types from real chat records involved in the case, and obtains dialogue blocks after segmentation; the question design unit designs clue query questions based on crime clues and clues to the behavior involved in the case; the writing and analysis unit designs an analysis template, determines whether the dialogue block meets the retrieval conditions of the clue query question and annotates the language analysis on the dialogue block according to the designed analysis template to obtain a fine-tuned data set.

[0026] Furthermore, the segmentation in the designed dialogue block unit is to pre-process the data to remove or compress the text that has little relationship with the semantic analysis, convert the CSV data format into a dialogue format, and segment the dialogue according to the rules to obtain dialogue blocks; the rules are to segment according to the chat interval time and the dialogue block character length.

[0027] Furthermore, the analysis template of the analysis writing unit specifically includes question restatement, chat content summary, segment analysis, overall analysis and conclusion; the question restatement rewrites the goal of the question into a scenario that conforms to the clue extraction in electronic forensics; the chat content summary summarizes the chat content in order from top to bottom; the segment analysis analyzes each summary in combination with the question; the overall analysis conducts a comprehensive analysis of all rewritten questions, summaries and segment analyses; and the conclusion determines whether the dialogue block meets the retrieval conditions of the question.

[0028] Furthermore, the alignment training module includes a model analysis unit, a manual evaluation unit, an alignment question unit and an alignment fine-tuning unit; the model analysis unit uses the trained model to analyze the fine-tuning data of the fine-tuning test set to obtain an analysis result; the manual evaluation unit manually judges the correctness of the analysis result to obtain the alignment data; the alignment question unit designs clue query questions for the collected alignment data, and supplements the alignment data and the designed clue query questions into the fine-tuning data set; the alignment fine-tuning unit uses the fine-tuning data in the supplemented fine-tuning data set to re-fine-tune the model.

[0029] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) It realizes semantic analysis of chat records, which helps to improve the work efficiency of staff; (2) The recall rate is high. The model used in this method can accurately identify obscure sentences in chat records such as jargon, slang, abbreviations, etc. through preliminary training. Moreover, after the chat records are divided into blocks for analysis, the analysis of each block is combined to finally draw a general conclusion without missing important information; (3) The precision rate is high. This method can perform accurate semantic analysis of chat records; (4) It is simple to use and fully natural language interaction, and does not require a deep understanding of the case or rich case handling experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the domain fine-tuning stage of the present invention;

[0031] Figure 2 Schematic representation of fine-tuning training hyperparameters of the present invention;

[0032] Figure 3 It is a schematic diagram of the alignment training phase of the present invention;

[0033] Figure 4 It is the overall workflow diagram of the model of the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0035] See also Figure 1 As shown, the method of semantic analysis of chat records based on language model in this embodiment selects chat record segments from real chat records involved in the case, covering 10 types of crimes and 30 chat record files. After segmentation, a total of 15,000 dialogue segments were obtained; when designing questions, the question forms were required to be scattered and diverse, and a total of 100 questions were designed. Question design is divided into criminal clue query in a general sense, and clue query for the behavior involved in the case, which only checks the specific behavior without involving the crime.

[0036] This embodiment also proposes an analysis template for chat records. According to this template, a model that meets the requirements is trained, as shown in Table 2:

[0037] Table 2 Model analysis process

[0038]

[0039]

[0040] Among them, the "question restatement" model combines the user's input questions with the electronic evidence collection scenario to generate questions that are more in line with the case; the "chat content summary" gives the model the ability to understand the chat records; the "segment analysis" finds arguments from each paragraph; the "overall analysis" analyzes the overall situation; the "conclusion" determines whether it meets the search conditions and outputs the results. After labeling, a total of 1,000 fine-tuning data were obtained. In addition, we also selected 5 separate chat records involved in the case and constructed 200 data of the same format according to the same labeling as a test set.

[0041] See also Figure 2 As shown in the figure, after the data set is constructed, the model is trained. We used the Chinese adaptation model of llama2-13b as the base model and trained it on 1,000 fine-tuning data.

[0042] See also Figure 3 As shown in the figure, after domain-based fine-tuning, the model has learned the ability to analyze chat records, and then through alignment training, the model's judgment scale is close to that of forensic personnel. The domain-based fine-tuned model is used to analyze questions and chat fragments to obtain analysis results. The analysis results are manually evaluated to obtain aligned data. A total of 10,000 aligned data are collected.

[0043] See also Figure 4 As shown in the figure, the overall workflow of the model is:

[0044] (1) Upload chat records. Users upload chat records obtained from electronic forensics software in CSV format, which contains at least the following fields: A (device owner), B (conversation party), sent content, sent time, sent content, and sent direction. The sending direction refers to whether the message is sent by the device owner or the conversation party.

[0045] (2) Data preprocessing and segmentation. The system first preprocesses the data, with the main purpose of removing or compressing text that has little relevance to semantic analysis. For example, all external links, images, sounds, and application codes are removed and replaced with: "A sent a video / image / voice / link to B." After processing, the length of the text will be significantly shortened, thereby reducing the computing cost of subsequent semantic analysis. Secondly, the CSV data format is converted into a dialogue format, and the dialogue is segmented according to the rules to obtain several dialogue blocks. The dialogue blocks can be regarded as separate documents and are also basic retrieval objects. After segmentation, each dialogue block contains continuous chat content between the owner and a certain dialogue party, and the length does not exceed 400 words. For example: "A: Hello. B: Hello. A: Do you want to play basketball tonight? B: Sure."

[0046] (3) The user asks a question and enters a search formula in natural language. The search formula can be a direct question about criminal clues, such as "Did the chat involve ** activities?" or a question about limited behaviors, such as "Did the chat discuss hotel accommodation? It must also be accompanied by money transactions such as transfers and red envelopes, and the amount must be no less than RMB 2,000."

[0047] (4) Analyze the chat records, input the user's search query and the pre-processed chat records, and output the analysis results that match the user's search query and the pre-processed chat records.

[0048] (5) The chat record semantic analysis model responds with a formatted text: "The conclusion is yes / no." If "the conclusion is yes," it means that the model believes that the conversation block matches the description of the search question, and the corresponding conversation block is returned to the user as a search result.

[0049] The test was conducted on a 200-item test data set, with accuracy as the main evaluation indicator. The results are shown in Table 4:

[0050] Table 4 Comparison of experimental results

[0051] Model Accuracy Random guessing 50％ Chinese-Alpaca-llama2-13b 51％ Chat GPT 55％ GPT-4 68％ Chat Clue (domain-based fine-tuning) 82％ Chat Clue (alignment training) 95％

[0052] Combined with our evaluation of the generated results, we can see from the above:

[0053] 1. Chinese-Alpaca-llama2-13b is the basic model of Chat Clue. It can hardly understand chat records or questions, and its accuracy is close to random guessing.

[0054] 2. Chat GPT has a slightly better ability to understand chat records, but the overall analysis has a certain logic, but the ability to understand questions is poor. The accuracy rate has been slightly improved.

[0055] 4. The domain-specific fine-tuning of Chat Clue has surpassed the existing language model and has better chat content understanding and question analysis capabilities. The analysis ideas are basically correct, but there are still many missed detections.

[0056] 5. After alignment training, Chat Clue can correctly analyze and identify most chat content with an accuracy rate of up to 95%, and the missed detection rate is greatly reduced.

Claims

1. A method for semantic analysis of chat records based on a language model, characterized in that: The following steps are involved: (1) Divide the chat records into dialogue blocks, and design an analysis template to annotate the language analysis on the dialogue blocks to obtain the fine-tuning training set and fine-tuning test set; (2) Use the fine-tuning data in the fine-tuning training set to perform fine-tuning training on the base model; (3) The fine-tuning data in the fine-tuning test set is analyzed using the trained model, and the correctness of the analysis results is judged to obtain the aligned data. The aligned data is then added to the fine-tuning data set, and fine-tuning training is performed again.

2. The method for performing semantic analysis on chat records based on a language model according to claim 1, characterized in that: The step (1) specifically comprises the following steps: (11) selecting chat record segments covering common crime types from real chat records involved in criminal cases, and obtaining dialogue blocks after segmentation; (12) Design clue inquiry questions based on clues of crime and related behavior; (13) Design an analysis template to determine whether the dialogue block meets the retrieval conditions of the clue query question and annotate the dialogue block with the designed analysis template to perform language analysis to obtain a fine-tuning dataset.

3. The method for semantic analysis of chat records based on a language model according to claim 2, characterized in that: The segmentation in step (11) is to pre-process the data to remove or compress texts that have little to do with semantic analysis, convert the CSV data format into a dialogue format, and segment the dialogue according to the rules to obtain dialogue blocks; the rules are to segment according to the chat interval time and the character length of the dialogue block.

4. The method for performing semantic analysis on chat records based on a language model according to claim 2, characterized in that: The analysis template in step (13) specifically includes a restatement of the question, a summary of the chat content, a segmented analysis, a general analysis and a conclusion; the restatement of the question rewrites the goal of the question into a scenario that conforms to the clue extraction in electronic forensics; the chat content summary summarizes the chat content in order from top to bottom; the segmented analysis analyzes each summary in combination with the question; the general analysis conducts a comprehensive analysis of all rewritten questions, summaries and segmented analyses; and the conclusion determines whether the dialogue block meets the retrieval conditions of the question.

5. The method for semantic analysis of chat records based on a language model according to claim 1, characterized in that: The step (3) specifically comprises the following steps: (31) Using the trained model to analyze the fine-tuning data of the fine-tuning test set to obtain analysis results; (32) Manually judging the correctness of the analysis results to obtain aligned data; (33) Designing clue query sentences for the collected aligned data, and adding the aligned data and the designed clue query sentences to the fine-tuning dataset; (34) Use the fine-tuning data in the supplementary fine-tuning dataset to re-fine-tune the model.

6. A system for semantic analysis of chat records based on a language model, characterized in that include: The dataset construction module is used to divide the chat records into conversation blocks and design analysis templates to annotate language analysis on the conversation blocks to obtain fine-tuning training sets and fine-tuning test sets; A fine-tuning training module is used to perform fine-tuning training on the base model using the fine-tuning data in the fine-tuning training set; The alignment training module is used to fine-tune the fine-tuning data in the test set, obtain the analysis results through the trained model, judge whether the analysis results are correct or not to obtain the alignment data, add the alignment data to the fine-tuning data set, and perform fine-tuning training again.

7. The system for performing semantic analysis on chat records based on a language model according to claim 6, characterized in that: The data set construction module specifically includes a dialogue block design unit, a question design unit and a writing and analysis unit; the dialogue block design unit screens chat record segments covering common crime types from real chat records involved in the case, and obtains dialogue blocks after segmentation; the question design unit designs clue query questions based on crime clues and clues to the behavior involved in the case; the writing and analysis unit designs an analysis template, determines whether the dialogue block meets the retrieval conditions of the clue query question, and annotates the language analysis on the dialogue block according to the designed analysis template to obtain a fine-tuning data set.

8. The system for performing semantic analysis on chat records based on a language model according to claim 7, characterized in that: The segmentation in the designed dialogue block unit is to pre-process the data to remove or compress the text that has little relationship with the semantic analysis, convert the CSV data format into a dialogue format, and segment the dialogue according to the rules to obtain dialogue blocks; the rules are to segment according to the chat interval time and the character length of the dialogue block.

9. The system for performing semantic analysis on chat records based on a language model according to claim 7, characterized in that: The analysis template of the writing analysis unit specifically includes question restatement, chat content summary, segment analysis, overall analysis and conclusion; the question restatement rewrites the goal of the question into a scenario that conforms to the clue extraction in electronic forensics; the chat content summary summarizes the chat content in order from top to bottom; the segment analysis analyzes each summary in combination with the question; the overall analysis conducts a comprehensive analysis of all rewritten questions, summaries and segment analyses; and the conclusion determines whether the dialogue block meets the retrieval conditions of the question.

10. The system for performing semantic analysis on chat records based on a language model according to claim 6, characterized in that: The alignment training module includes a model analysis unit, a manual evaluation unit, an alignment question unit and an alignment fine-tuning unit; the model analysis unit uses the trained model to analyze the fine-tuning data of the fine-tuning test set to obtain an analysis result; The manual evaluation unit manually judges the correctness of the analysis results to obtain the alignment data; the alignment question unit designs clue query questions for the collected alignment data, and adds the alignment data and the designed clue query questions to the fine-tuning data set; The alignment fine-tuning unit uses the fine-tuning data in the supplementary fine-tuning dataset to re-fine-tune the model.

Images