Electronic data forensic analysis method and system based on multiple modes, medium and equipment
Through multimodal data processing and integration, combined with language classification, automatic speech recognition and visual language model, the problem of insufficient utilization of audio and image data in traditional methods is solved, and more accurate and comprehensive chat content analysis is achieved, which improves the evidence. Reliability and analysis quality.
Patent Information
- Application Number
- CN202411051004.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-22
AI Technical Summary
Traditional chat content data analysis and evidence collection methods mainly rely on text information and cannot effectively utilize audio and image data, resulting in incomplete and inaccurate analysis results, and lack of utilization of the value of audio and image data in analysis.
Using multimodal-based electronic data forensic analysis method, audio and image data are converted into text content through language classifiers, automatic speech recognition models and visual language models, and these text contents are integrated with the original text data, using multimodal The large language model conducts in-depth analysis and extracts key information such as characters, time, and events.
Achieve a more comprehensive understanding of chat content, improve the accuracy and reliability of evidence, optimize the presentation method of information, improve the dimension and quality of data analysis, adapt to the analysis needs of different languages and dialects, and enhance long conversations Or the processing ability of complex chat leads.
Smart Images

Figure CN119004024A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of electronic data forensics, and in particular relates to a multimodal electronic data forensics analysis method, system, medium and equipment. Background Art
[0002] With the rapid development of information technology, the role of electronic data in legal proceedings has become increasingly prominent. As a form of electronic data, the authenticity, integrity and validity of chat content have an important impact on the judgment of the case. Traditional chat content data analysis and forensic analysis methods mainly rely on text information, and cannot effectively utilize data such as audio and images. They ignore the value of audio and image data in analysis, resulting in incomplete and inaccurate analysis results.
[0003] In view of the limitations of existing technologies, how to develop an electronic data forensics analysis method that can comprehensively analyze text, audio and image data has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention
[0004] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to provide a multimodal electronic data forensic analysis method, system, medium and equipment, which can more comprehensively understand the context of the chat content, comprehensively restore the facts of the case, improve the accuracy and reliability of evidence, and meet judicial needs.
[0005] In a first aspect, the present invention provides a multimodal electronic data forensic analysis method, the method comprising the following steps:
[0006] Acquiring electronic data, wherein the electronic data includes audio data, text data, and image data;
[0007] Classify the audio data based on a language classifier to obtain a language category of the audio data;
[0008] Selecting a corresponding automatic speech recognition model based on the language category to obtain the text content of the audio data;
[0009] Acquire text content of the image data based on a visual language model;
[0010] The text data, the text content of the audio data, and the text content of the image data are input into a multimodal large language model to obtain a forensic analysis result of the electronic data.
[0011] In an implementation of the first aspect, the training of the language classifier includes the following steps:
[0012] Label the audio data corresponding to different language categories and build corresponding speech databases for different language categories;
[0013] Selecting the CAM++ model as the basic architecture of the language classifier;
[0014] The language classifier is trained using a corresponding speech database.
[0015] In an implementation of the first aspect, selecting a corresponding automatic speech recognition model based on the language category to obtain the text content of the audio data includes the following steps:
[0016] Preprocessing the audio data to obtain preprocessed audio data;
[0017] The paraformer model is selected as the basic architecture of the automatic speech recognition model;
[0018] The preprocessed audio data is input into an automatic speech recognition model, and the automatic speech recognition model is trained by an end-to-end training method to obtain text content corresponding to the audio data.
[0019] In an implementation of the first aspect, acquiring the text content of the image data based on the Qwen-VL-Chat model specifically includes the following steps:
[0020] Segmenting and adjusting the resolution of the image data based on a visual encoder to obtain image features of the image data;
[0021] The image features are compressed into a sequence of fixed length based on a visual language adapter, and the text content corresponding to the image data is obtained in combination with a large language model.
[0022] In an implementation of the first aspect, inputting the text data, the text content corresponding to the audio data, and the text content of the image data into a multimodal large language model, and obtaining the forensic analysis result of the electronic data includes the following steps:
[0023] Integrate the text data, the text content of the audio data, and the text content of the image data to obtain integrated electronic data;
[0024] Based on the Qwen-72B-Chat model, the integrated electronic data is extracted by person, time, and event to obtain the forensic analysis results output in chronological order.
[0025] In an implementation of the first aspect, the method further includes processing the text data, the text content of the audio data, and the text content of the image data through continuous time recognition to obtain analysis results within a preset continuous time.
[0026] In an implementation manner of the first aspect, the method further includes performing semantic similarity evaluation on the text data, the text content of the audio data, and the text content of the image data to obtain corresponding content with high semantic similarity.
[0027] In a second aspect, the present invention provides a multimodal electronic data forensic analysis system, the system comprising a first acquisition module, a language classification module, a second acquisition module, a third acquisition module and a data analysis module;
[0028] The first acquisition module is used to acquire electronic data, and the electronic data includes audio data, text data and image data;
[0029] The language classification module is used to classify the audio data based on a language classifier to obtain the language category of the audio data;
[0030] The second acquisition module is used to select a corresponding automatic speech recognition model based on the language category to obtain the text content of the audio data;
[0031] The third acquisition module is used to acquire the text content of the image data based on the visual language model;
[0032] The data analysis module is used to input the text data, the text content of the audio data, and the text content of the image data into a multimodal large language model to obtain a forensic analysis result of the electronic data.
[0033] In a third aspect, the present invention provides an electronic device, the electronic device comprising: a processor and a memory;
[0034] The memory is used to store computer programs;
[0035] The processor is used to execute the computer program stored in the memory so that the electronic device performs the above-mentioned multimodal-based electronic data forensics analysis method.
[0036] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the above-mentioned multimodal-based electronic data forensics analysis method.
[0037] As described above, the multimodal electronic data forensic analysis method, system, medium and device described in the present invention have the following beneficial effects:
[0038] The multimodal electronic data forensic analysis method, system, medium and device described in the present invention have multimodal data processing and integration capabilities: the present invention processes audio and image data by integrating an automatic speech recognition model (such as the Paraformer architecture) and a visual language model (such as Qwen-VL-Chat), and converts them into text content, and then integrates these texts with the original text data. This integration of multimodal data provides rich and comprehensive materials for subsequent analysis. And using the Qwen-72B-Chat model to conduct in-depth analysis of the integrated data can effectively extract key information such as people, time, events, and output the results in chronological order. This method not only optimizes the presentation of information, but also improves the dimension and quality of data analysis.
[0039] The present invention can achieve the optimization capability of specific language recognition: for the specific processing of dialects, the self-trained Paraformer automatic speech recognition model significantly improves the accuracy of chat content analysis for specific regions or groups, which is particularly important for application environments with language diversity. By adopting the language classifier trained by the CAM++ model architecture, it provides reliable guarantee for the automated processing and analysis of different languages.
[0040] The present invention can achieve efficient information extraction and analysis: by considering the time interval and semantic similarity of messages to perform window division, it ensures that messages on the same topic can be correctly classified and analyzed. This method improves the ability to handle long conversations or complex chat threads and avoids topic fragmentation. The time interval and semantic similarity thresholds are dynamically adjusted according to actual application feedback, allowing the system to adapt to changing user needs and behavior patterns. This flexibility and adaptability is one of the important features of the present technical solution. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 Shown is a flow chart of a multi-modal electronic data forensics analysis method in one embodiment of the present invention;
[0042] Figure 2 Shown is a system flow chart of a multi-modal electronic data forensics analysis method in one embodiment of the present invention;
[0043] Figure 3 Shown is a paraformer model training diagram in one embodiment of the present invention;
[0044] Figure 4 Shown is a flow chart of obtaining forensic analysis results based on the Qwen-72B-Chat model of the present invention;
[0045] Figure 5 Shown is a schematic diagram of chat record evidence in one embodiment of the present invention;
[0046] Figure 6 Shown is a schematic diagram of the structure of a multi-modal electronic data forensics analysis system in one embodiment of the present invention;
[0047] Figure 7 It is a schematic structural diagram of an electronic device of the present invention in one embodiment. DETAILED DESCRIPTION
[0048] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0049] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0050] Digital Forensics is a specialized technology and scientific method used to collect, secure, and analyze data from electronic devices to support legal investigations, criminal investigations, or internal audits. The goal of digital forensics is to ensure the integrity and credibility of data so that it can be used as evidence in court or other legal proceedings. Electronic data evidence may include data in computer storage devices, network traffic, content in mobile devices, information in IoT devices, etc. Fixing evidence in chat content is part of digital forensics, especially when it involves legal proceedings or investigations. Extract key information from chat records, such as interlocutors, timestamps, message content, etc. Specialized forensic techniques are needed to extract deep metadata and analyze the extracted information to discover potential evidence or clues.
[0051] The technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0052] like Figure 1 and Figure 2 As shown, in one embodiment, the multimodal electronic data forensic analysis method of the present invention includes steps S11 to S15.
[0053] Step S11: Acquire electronic data, where the electronic data includes audio data, text data, and image data.
[0054] Specifically, taking WeChat chat content as an example, a segment of WeChat chat content is obtained, and the chat content includes audio data-recording, text data-text, and image data-picture.
[0055] Step S12: classify the audio data based on a language classifier to obtain a language category of the audio data, where the category includes Chinese, foreign language, and dialect.
[0056] The training of the language classifier comprises the following steps:
[0057] Label the audio data corresponding to different language categories and build corresponding speech databases for different language categories;
[0058] Selecting the CAM++ model as the basic architecture of the language classifier;
[0059] The language classifier is trained using a corresponding speech database.
[0060] Specifically, using the CAM++ model architecture, a classifier is trained that can distinguish these four languages.
[0061] In one embodiment, a large amount of speech data is prepared, and the speech data is preprocessed to filter out audio data with relatively large noise to avoid affecting the training effect; the data is annotated using a labeling method of model pre-labeling and manual labeling to effectively reduce the cost of manual labeling; the language classification data is sorted into four categories in total: Minnan dialect, Cantonese, English and Mandarin, with about 100,000 entries in each category; and the output is the speech path and the corresponding language.
[0062] The CAM++ model is a recognition model based on a densely connected time-delay neural network. The entire model consists of two parts, a residual convolutional network as the front end and a time-delay neural network structure as the backbone. Compared to some mainstream models, such as ResNet34 (deep residual network) and ECAPA-TDNN (a neural network based on TDNN in speaker recognition), CAM++ has more accurate recognition performance and faster reasoning speed.
[0063] The main part of the model adopts a densely connected time-delay neural network (D-TDNN). The input of each layer is concatenated from the output of all previous layers. This hierarchical feature reuse and one-dimensional convolution of the time-delay network can significantly improve the computational efficiency of the network. At the same time, each layer of the D-TDNN is embedded with a lightweight context-aware mask (CAM) module, which extracts context information of different scales through multi-granularity pooling operations. The generated mask can remove irrelevant noise in the features and retain key information. The CAM module extracts context information of different scales through global and segment-level pooling operations, and the generated mask can remove irrelevant noise in the features. TDNN-CAM forms a unified modeling of local-segment-level-global features, and the network can learn richer speaker information in the features. The front-end module of CAM++ is a lightweight residual convolutional network that uses two-dimensional convolution in the time-frequency dimension. Compared with one-dimensional convolution, two-dimensional convolution has a smaller receptive field and can capture more local and detailed frequency domain information. The front-end module is a 2D convolutional structure, which is used to extract more local and detailed time-frequency features. The backbone module uses dense connections and reuses hierarchical features to improve computational efficiency. At the same time, it can also be robust to speaker-specific frequency pattern shifts that may exist in the input features.
[0064] Combine multiple models or different parts of models to improve overall performance or adapt to specific tasks. In the present invention, the language classifier is integrated with the ASR model, and these two components are combined into a unified system, so that the system can automatically identify the language of the input speech and select the corresponding ASR model for speech recognition based on the recognition result. Specifically, the integration process includes the following steps: the user inputs a speech; the system first uses the language classifier to classify the input speech and determine its language; based on the language classification result, the system selects the corresponding ASR model; the selected ASR model is used to perform speech recognition on the input speech to generate text content; the system outputs the recognized text content. Through this integration, the system can automatically process the input speech and generate accurate text output, which is suitable for the processing needs of various languages and dialects.
[0065] Step S13: Selecting a corresponding automatic speech recognition model based on the language category to obtain the text content of the audio data.
[0066] Selecting a corresponding automatic speech recognition model based on the language category to obtain the text content of the audio data comprises the following steps:
[0067] Preprocessing the audio data to obtain preprocessed audio data;
[0068] The paraformer model is selected as the basic architecture of the automatic speech recognition model;
[0069] The preprocessed audio data is input into an automatic speech recognition model, and the automatic speech recognition model is trained by an end-to-end training method to obtain text content corresponding to the audio data.
[0070] In one embodiment, dialects use a specific self-trained Paraformer model, and Mandarin and foreign languages use the open source Paraformer model of the ModelScope community to obtain the corresponding text content. Figure 3 As shown in the figure, the specific steps of the self-training automatic speech recognition model for dialects are introduced as follows, taking Minnan dialect as an example:
[0071] S31. Data preprocessing: Perform denoising, normalization and other preprocessing operations on the collected Minnan audio data to improve data quality.
[0072] S32. Build an automatic speech recognition model: Select the paraformer model as the basic architecture. The model consists of five parts: Encoder, Predictor, Sampler, Decoder and Loss function.
[0073] Encoder can adopt different network structures, such as self-attention, conformer, SAN-M (self-attention network-multimodal), etc. In the automatic speech recognition (ASR) system, the encoder is a key component, which is responsible for converting the original audio signal into a feature vector that the model can understand and process.
[0074] Predictor is a two-layer FFN (Feedforward Neural Network) that is used to predict the number of target words in speech and extract the acoustic feature vectors corresponding to the target words.
[0075] Sampler is a sampling module based on GLM (Glancing language model). It generates feature vectors containing semantic information according to the input acoustic feature vector and the number of target characters, and then cooperates with the bidirectional Decoder to enhance the model's ability to model context.
[0076] The decoder structure is a bidirectional modeling and is responsible for converting the model's predicted sequence into text output. The decoder receives the feature vector sequence from the encoder or sampler as input and uses these feature vector sequences to generate the corresponding text sequence, that is, the recognized speech content.
[0077] The loss function is a measure of the difference between the model's predicted output and the true label. Its function is to guide the model's learning process and help the model adjust its weights to reduce prediction errors.
[0078] In traditional ASR training, cross-entropy loss is usually used as a loss function, which is used to measure the difference between the predicted probability distribution and the true distribution. In ASR, it is used to evaluate the degree of match between the text probability distribution predicted by the model at each time step and the true text sequence. On this basis, the present invention also introduces MWER (Minimum Word Error Rate) discriminative training, focusing on the word error rate in the entire transcription process, and designs a loss function containing negative samples so that the model is more severely punished when the prediction is wrong. During the ASR training process, some incorrect transcription results are selected as negative samples. These negative samples are incorrect predictions generated by the model during the training process, which are contrasted with the correct transcription results. By introducing incorrect transcription results as negative samples and designing a loss function containing these negative samples, the model can better learn the correct transcription pattern, reduce the number of word errors in the entire transcription process, and thus improve the overall performance of the system. In addition, the accuracy of the predictor module in predicting the number of target characters and extracting acoustic feature vectors is also within the optimization range. The average absolute value of the difference between the predicted value and the actual value is calculated by MAE (Mean Absolute Error), which significantly improves the accuracy of the predictor module in predicting the number of target characters and extracting acoustic feature vectors. It can be seen that the present invention adopts a multi-objective optimization strategy in the training process, aiming to improve the performance of the model through different loss functions.
[0079] S33. Model pre-labeling: Use model pre-labeling and manual labeling methods to label data, effectively reducing the cost of manual labeling.
[0080] S34, training process: The processed speech and the text corresponding to the annotated speech data are trained on the GPU cluster, and the cross entropy loss function and Adam optimizer are used to update the model weights through the back propagation algorithm.
[0081] S35, Model deployment: Deploy the trained model to the server and provide real-time speech recognition services through the API interface.
[0082] Step S14: Acquire text content of the image data based on a visual language model.
[0083] Specifically, the text content corresponding to the image data is obtained based on the Qwen-VL-Chat model, including the following steps:
[0084] Segmenting and adjusting the resolution of the image data based on a visual encoder to obtain image features of the image data;
[0085] The image features are compressed into a sequence of fixed length based on a visual language adapter, and the text content corresponding to the image data is obtained in combination with a large language model.
[0086] The overall network architecture of Qwen-VL-Chat consists of three components: a large language model, a visual encoder, and a position-aware visual language adapter. Among them, the large language model is the basic component of the model, which is initialized with pre-trained weights; the visual encoder adopts the Vision Transformer (ViT) structure, and adjusts the input image to a specific resolution during training and inference; the position-aware visual language adapter alleviates the efficiency problems caused by long sequences by compressing image features into sequences of fixed length, and takes into account the importance of positioning information. This model aims to solve the problems of visual language understanding and generation in natural language processing, including tasks such as text generation, visual question answering, visual reasoning, reference positioning, and guided generation. Through multi-task pre-training of a large number of data sets and tasks, efficient visual language understanding and generation can be achieved, which can be applied to a variety of practical scenarios. Specifically:
[0087] The visual encoder uses the Vision Transformer (ViT) structure, which is a Transformer architecture designed specifically for images. ViT divides the image into fixed-size blocks, which are then used as input to the Transformer model. This approach allows the model to process image data directly without the need for a complex feature extraction process. During training and inference, the input images are adjusted to a specific resolution to ensure that the model can handle images of different sizes.
[0088] The position-aware visual-language adapter alleviates the efficiency issues caused by long sequences by compressing image features into fixed-length sequences. The adapter takes into account the importance of localization information, that is, the importance of different regions in the image for understanding the content of the entire image. In this way, the model is able to better understand the spatial relationships in the image and combine them with language information.
[0089] Step S15: input the text data, the text content corresponding to the audio data, and the text content of the image data into a multimodal large language model to obtain analysis results of the electronic data.
[0090] Specifically, Figure 4 As shown, the text data, the text content corresponding to the audio data, and the text content of the image data are input into a multimodal large language model, and obtaining the forensic analysis result of the electronic data includes the following steps:
[0091] Integrate the text data, the text content corresponding to the audio data, and the text content of the image data to obtain integrated electronic data;
[0092] Based on the Qwen-72B-Chat model, the integrated electronic data is extracted in terms of person, time and event to obtain analysis results output in chronological order.
[0093] Qwen-72B-Chat adopts a design based on the Transformer architecture and optimizes data, tokenization, architecture, etc. First, in terms of data, various types and languages of data sources are used, and various technical means are used to process and filter them to build a large-scale pre-training dataset containing tens of trillions of tokens. Secondly, in terms of tokenization, byte pair encoding (BPE) is used as the word segmentation method, and some specific strategies are combined, such as splitting numbers into single digits, to improve model performance. Finally, in terms of architecture, the Transformer architecture is modified and optimized, including selecting appropriate embedding and output projection methods, using RoPE as position encoding, removing bias items in most layers, and other measures to further improve the model effect.
[0094] In one embodiment, the acquired WeChat chat content is integrated to form complete chat content data, and Qwen-72B-Chat is used to extract characters, time, and events from the integrated data, so that character behaviors and events are output in chronological order. The model inference parameters and prompts are continuously adjusted to achieve the best analysis effect. The entire module is integrated into the forensic software, and the analysis results are displayed to the user in an easy-to-understand format.
[0095] Furthermore, the method also includes processing the text data, the text content corresponding to the audio data, and the text content of the image data through continuous time recognition technology to obtain a forensic analysis result within a preset continuous time.
[0096] Furthermore, the method also includes performing semantic similarity evaluation on the text data, the text content corresponding to the audio data, and the text content of the image data, and setting a preset threshold, obtaining and merging corresponding content with semantic similarity higher than the preset threshold.
[0097] Due to the limitation of LLM context window length, the chat content needs to be divided into windows. In order to ensure that the chat content of the same topic is not divided into multiple windows, the window is divided using a method based on continuous time and semantic similarity. The core of this method is to ensure that related messages can be included in the same processing window, so as to better understand and analyze the chat content.
[0098] In one embodiment, if the time interval is used as the basis for division, in a chat or instant messaging application, the timestamp between messages provides an intuitive basis for window division. If the sending time of two messages does not exceed 2 hours, they are considered to be continuous and should be divided into the same window. This simple rule based on time can effectively group most related messages together. Although the timestamp is an effective division tool, relying solely on time may mistakenly divide messages belonging to the same topic but with a slightly longer time interval into different windows. For example, a chat topic may be mentioned again two hours later. In this case, dividing only by time will break the continuity of the topic. Based on this, after acquiring electronic data, the system will first pre-classify according to the timestamp of each message. By calculating the time difference between two adjacent messages, the system can preliminarily determine which messages may belong to the same conversation or discussion topic. Set a time difference threshold (such as 2 hours) to determine whether there is sufficient time continuity between messages. If the time difference between two messages is less than or equal to the set threshold, they will be preliminarily classified as the same discussion thread. On the basis of ensuring the time continuity of the message, the system will further analyze the message content to optimize the data integration process. For example, for messages that are close in time but have very different contents, the system will decide whether to merge them into the same window based on semantic analysis.
[0099] In one embodiment, by converting messages into semantic vectors using the bge-large-zh-v1.5 model, the cosine similarity between different messages can be calculated. If the semantic similarity of the messages in the two windows is high, it indicates that they may belong to the same topic and should be merged into one window for processing. By introducing semantic similarity considerations, the system can more accurately identify and process the continuation and conversion of topics in actual conversations. This method helps to avoid the problem of related messages being incorrectly separated due to a large time span. Based on this, semantic similarity evaluation and dynamic window adjustment include the following steps:
[0100] S51. Use large language models for semantic encoding: Use large language models such as Qwen-72B-Chat to convert the text content of text data, transcribed audio data, and image data into high-dimensional semantic vectors.
[0101] S52. Calculate semantic similarity: By comparing the semantic vectors of different messages or message combinations, the system calculates their cosine similarity. This step is to quantify the strength of semantic association between messages. Setting the cosine similarity threshold to 0.5 is a reasonable value based on experiments and experience. This value can avoid excessive merging of irrelevant content while maintaining topic continuity.
[0102] S53, dynamically adjust window boundaries: Based on the calculated semantic similarity, the system can dynamically adjust the window boundaries. If the semantic similarity of messages contained in two temporally discontinuous windows is higher than a preset threshold, the two windows may be merged to reflect a more realistic communication process and topic continuation.
[0103] Based on feedback from actual application scenarios, the time interval and similarity threshold can be dynamically adjusted to optimize the effect of window division. This flexibility is the key to adapting to changing user needs and behavior patterns. The electronic data analysis method based on continuous time and semantic similarity ensures the coherence of chat content and the integrity of topics through multi-dimensional considerations. Although this method is technically complex, it significantly improves the accuracy of data processing and user experience. By continuously optimizing algorithms and adjusting parameters, the performance and adaptability of the system can be further improved.
[0104] In one embodiment, taking WeChat chat records as an example, first, the Qwen-VL-Chat model is used to convert all the pictures in the chat records into text descriptions; secondly, all the voices in the chat records are converted into text through the ASR model. Before converting to text, the voice language will be classified. Different languages will use different fine-tuned models for voice recognition and convert to text. Finally, after all modal data are converted into text data, the Qwen-72B-Chat model is used for analysis. The chat content can be understood by natural language such as chat topic, whether it is pornographic, summary, keyword extraction, etc., and questions and answers can be conducted based on the chat content. Due to the limitation of the LLM context window length, the chat content needs to be divided into windows. In order to ensure that the chat content of the same topic will not be divided into multiple windows as much as possible, the window is divided using a method based on continuous time and semantic similarity.
[0105] If the time interval between the sending of the previous and next two messages does not exceed 2 hours, it is considered that the two messages should be divided into the same window. Sometimes, two messages with an interval of more than two hours may belong to the same topic. Therefore, it is impossible to handle this scenario by only merging windows by time. Therefore, a window merging method based on semantic similarity is added. Using the bge-large-zh-v1.5 model, the last 5 messages of the previous window and the first five messages of the current window are converted into semantic vectors, and the cosine similarity between the two is calculated. The smaller the cosine similarity value, the more similar the two are, and the more likely they belong to the same window. Finally, in the implementation, we set the similarity threshold to 0.5. If the cosine similarity is less than 0.5, the two windows are merged, otherwise, they are not merged. Then, by limiting the length, it is ensured that the merged window does not exceed the context length of LLM.
[0106] In one embodiment, in cases involving pornography, Qwen-VL-Chat is used to identify pictures in the mobile phone of the inspection material, determine whether the pictures are pornographic, and convert the pictures into text, which can effectively reduce the time for case handlers to analyze the inspection materials. The chat records of the social software of the inspection material mobile phone are analyzed, and all pictures and voices are converted into text descriptions. The Qwen-72B-Chat model is used for dialogue. The chat record content between the owner and any friend can be discussed. For example, you can ask: What did Zhang San and Li Si talk about? Is the chat content between them pornographic? If there is pornography, the original records of the pornographic chat between the two will be listed and provided to the case handlers for evidence fixation. After the multimodal data is unified into text, important case information can be extracted from the text for any case.
[0107] In one embodiment, forensic evidence collection often requires analyzing the suspect's communication records, which may include text messages, audio calls, and pictures. How to accurately and quickly extract useful information from these multimodal data is the key to forensic analysis.
[0108] like Figure 5 As shown:
[0109] First, data collection and preprocessing are performed: the suspect’s WeChat chat records, including text messages, voice messages, and pictures, are collected. The audio data is processed using preprocessing techniques such as denoising and normalization to improve the accuracy of subsequent voice recognition.
[0110] Secondly, perform language category detection and automatic speech recognition: Apply a classifier based on the CAM++ model architecture to detect the language of the voice message to determine whether it is a standard language, dialect, or foreign language. Based on the language detection results, use the corresponding Paraformer model (standard language or specific dialect model) to transcribe the audio data into text. For example, the Minnan dialect voice uses a specially trained Minnan dialect Paraformer model.
[0111] Secondly, the image data is textualized: the Qwen-VL-Chat visual language model is used to convert the pictures in the chat records into text descriptions, such as identifying the text information and scene descriptions in the pictures.
[0112] Next, data integration and in-depth analysis are performed: the transcribed audio text, image description text and original text message are integrated. The Qwen-72B-Chat model is used to perform in-depth analysis on the integrated data to extract people, time, events and related behavioral dynamics.
[0113] Next, we divide and optimize the windows: We apply window division technology based on continuous time and semantic similarity to ensure that the content of the same topic is aggregated together to avoid fragmenting related discussions. We dynamically adjust the window division by calculating the time interval and semantic similarity between messages to adapt to different chat rhythms and topic transitions.
[0114] Finally, the analysis results are displayed in chronological order. By providing additional functions such as keyword extraction and pornographic detection, the development of the incident can be clearly understood. The forensic analysis results are as follows: 2020-12-01 21:18:19, Zhang San was going to Jiadian Fresh Exquisite Life Supermarket (1st Floor, Xincheng New World Club, Laodong East Road, Yuhua District, Changsha City, Hunan Province) and 2020-12-01 23:44:59, Zhang San was going to Greenland Center-T2 Office Building (Greenland Center, No. 319, Furong Middle Road, Kaifu District, Changsha City, Hunan Province), and waited for Wang Wu's results in room xxx.
[0115] Through targeted language recognition and optimized automatic speech recognition models, the accuracy of text transcription has been significantly improved. In addition, the integration of data from different modalities and unified analysis using a large language model have greatly improved the efficiency of data processing. At the same time, the dynamic window division technology ensures the coherence of the topic, making the analysis results more in line with the actual chat content.
[0116] The protection scope of the multimodal electronic data forensics analysis method described in the embodiment of the present invention is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present invention are included in the protection scope of the present invention.
[0117] An embodiment of the present invention further provides a multimodal electronic data forensics analysis system, which can implement the multimodal electronic data forensics analysis method described in the present invention. However, the implementation device of the multimodal electronic data forensics analysis system described in the present invention includes but is not limited to the structure of the multimodal electronic data forensics analysis system listed in this embodiment. All structural deformations and replacements of the prior art made according to the principles of the present invention are included in the protection scope of the present invention.
[0118] like Figure 6 As shown, in one embodiment, the multimodal electronic data forensics analysis system of the present invention includes a first acquisition module 61 , a language classification module 62 , a second acquisition module 63 , a third acquisition module 64 and a data analysis module 65 .
[0119] The first acquisition module 61 is used to acquire electronic data, and the electronic data includes audio data, text data and image data;
[0120] The language classification module 62 is connected to the first acquisition module 61, and the language classification module 62 is used to classify the audio data based on an integrated language classifier to obtain the language category of the audio data, wherein the category includes Chinese, foreign language and dialect;
[0121] The second acquisition module 63 is connected to the language classification module 62, and the second acquisition module 63 is used to acquire the text content corresponding to the audio data based on the automatic speech recognition model;
[0122] The third acquisition module 64 is connected to the second acquisition module 63, and the third acquisition module 64 is used to acquire the text content of the image data based on the visual language model;
[0123] The data analysis module 65 is connected to the third acquisition module 64, and the data analysis module 65 is used to input the text data, the text content corresponding to the audio data, and the text content of the image data into a multimodal large language model to obtain the analysis results of the electronic data.
[0124] In the several embodiments provided by the present invention, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules / units is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules or units, which can be electrical, mechanical or other forms.
[0125] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention. For example, the functional modules / units in the various embodiments of the present invention may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.
[0126] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0127] The embodiment of the present invention also provides a computer-readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by instructing the processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.
[0128] An embodiment of the present invention further provides an electronic device, which includes a processor and a memory.
[0129] The memory is used to store computer programs.
[0130] The memory includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.
[0131] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device performs the above-mentioned multi-modal-based electronic data forensics analysis method.
[0132] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
[0133] like Figure 7As shown, the electronic device of the present invention is in the form of a general computing device. The components of the electronic device may include but are not limited to: one or more processors or processing units 71, a memory 72, and a bus 73 connecting different system components (including the memory 72 and the processing unit 71).
[0134] Bus 73 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.
[0135] Electronic devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.
[0136] The memory 72 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 721 and / or cache memory 722. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 723 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 73 via one or more data medium interfaces. The memory 72 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.
[0137] A program / utility 724 having a set (at least one) of program modules 7241 may be stored, for example, in the memory 72, such program modules 7241 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 7241 generally perform the functions and / or methods of the embodiments described herein.
[0138] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, displays, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via input / output (I / O) interface 74. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 75. Figure 7 As shown, the network adapter 75 communicates with other modules of the electronic device via the bus 73. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0139] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.
Claims
1. A multimodal electronic data forensic analysis method, It is characterized in that The method comprises the following steps: Acquiring electronic data, wherein the electronic data includes audio data, text data, and image data; Classify the audio data based on a language classifier to obtain a language category of the audio data; Selecting a corresponding automatic speech recognition model based on the language category to obtain the text content of the audio data; Acquire text content of the image data based on a visual language model; The text data, the text content of the audio data, and the text content of the image data are input into a multimodal large language model to obtain a forensic analysis result of the electronic data.
2. The multimodal electronic data forensics analysis method according to claim 1, Features: The training of the language classifier comprises the following steps: Label the audio data corresponding to different language categories and build corresponding speech databases for different language categories; Selecting the CAM++ model as the basic architecture of the language classifier; The language classifier is trained using a corresponding speech database.
3. The multimodal electronic data forensics analysis method according to claim 1, Features: Selecting a corresponding automatic speech recognition model based on the language category to obtain the text content of the audio data comprises the following steps: Preprocessing the audio data to obtain preprocessed audio data; The paraformer model is selected as the basic architecture of the automatic speech recognition model; The preprocessed audio data is input into an automatic speech recognition model, and the automatic speech recognition model is trained by an end-to-end training method to obtain text content corresponding to the audio data.
4. The multimodal electronic data forensics analysis method according to claim 1, Features: Acquiring the text content of the image data based on the visual language model specifically includes the following steps: Segmenting and adjusting the resolution of the image data based on a visual encoder to obtain image features of the image data; Compressing the image features into a fixed-length sequence based on a visual language adapter, and acquiring text content corresponding to the image data in combination with a large language model; Wherein, the visual language model is the Qwen-VL-Chat model.
5. The multimodal electronic data forensics analysis method according to claim 1, Features: Inputting the text data, the text content corresponding to the audio data, and the text content of the image data into a multimodal large language model to obtain the forensic analysis result of the electronic data comprises the following steps: Integrate the text data, the text content of the audio data, and the text content of the image data to obtain integrated electronic data; Based on the Qwen-72B-Chat model, the integrated electronic data is extracted for people, time, and events to obtain forensic analysis results output in chronological order.
6. The multimodal electronic data forensics analysis method according to claim 1, Features: The method further includes processing the text data, the text content of the audio data, and the text content of the image data through continuous time recognition to obtain analysis results within a preset continuous time.
7. The multimodal electronic data forensics analysis method according to claim 1, Features: The method further includes performing semantic similarity evaluation on the text data, the text content of the audio data, and the text content of the image data to obtain corresponding content with high semantic similarity.
8. A multi-modal electronic data forensic analysis system, It is characterized in that The system includes a first acquisition module, a language classification module, a second acquisition module, a third acquisition module and a data analysis module; The first acquisition module is used to acquire electronic data, and the electronic data includes audio data, text data and image data; The language classification module is used to classify the audio data based on a language classifier to obtain the language category of the audio data; The second acquisition module is used to select a corresponding automatic speech recognition model based on the language category to obtain the text content of the audio data; The third acquisition module is used to acquire the text content of the image data based on the visual language model; The data analysis module is used to input the text data, the text content of the audio data, and the text content of the image data into a multimodal large language model to obtain a forensic analysis result of the electronic data.
9. An electronic device, It is characterized in that The electronic device comprises: a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory so that the electronic device executes the multimodal electronic data forensics analysis method described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by an electronic device, the multimodal electronic data forensics analysis method described in any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Traffic accident investigation document photo analysis method based on large model
CN121789150A
