A multi-dimensional conversation information extraction method for WeChat chat screenshots

Abstract

The application discloses a multi-dimensional conversation information extraction method for WeChat chat screenshots, relates to the technical field of information extraction, and uses an improved YOLOv8 model to jointly detect and identify multi-modal elements, outputs an element identification result set, and the elements of the element identification result set include basic elements and legal key elements; based on the spatial position and semantic relationship of the elements, the time sequence logic and the associated relationship of the message are constructed, and a message unit list arranged according to the real time sequence is output; the improved YOLOv8 model is used to simultaneously identify the basic elements and the legal key elements, and the problem that traditional methods can only identify conventional contents and miss legal sensitive information is solved. The detection logic of multi-modal fusion ensures that fuzzy and low-quality screenshots can be accurately identified, and information distortion caused by image quality is reduced. The message time sequence is constructed through the spatial position and semantic association, and the problem of disordered sorting of group chat dense messages and long screenshot cross-screen messages is solved.

Description

Technical Field

[0001] This invention relates to the field of information extraction technology, and in particular to a method for extracting multi-dimensional conversation information from WeChat chat screenshots. Background Technology

[0002] With the widespread use of social media, instant messaging tools such as WeChat have become essential for daily communication. In legal practice, WeChat chat logs, as electronic evidence, are increasingly important. Especially in cases involving contract disputes, loan agreements, and torts, transfer records, promissory notes, and timestamps within WeChat chat logs often serve as crucial evidence. However, since WeChat chat logs are typically provided as screenshots, efficiently extracting structured information from these screenshots and converting it into data that can be analyzed by legal artificial intelligence systems has become a pressing issue.

[0003] Evidence identification is a core task in legal artificial intelligence. Legal AI needs to use evidence to assist in judgment and generate relevant content, and extracting useful information from unstructured data (such as chat screenshots) is a key challenge. In modern society, WeChat chat screenshots not only contain text but may also include images, videos, voice messages, emojis, and other forms of information, increasing the complexity of information extraction.

[0004] Therefore, developing a method for extracting multi-dimensional conversation information from WeChat chat screenshots is of great significance for improving the efficiency and accuracy of legal artificial intelligence systems.

[0005] In addition, there are already some studies and tools available in the existing technology for extracting chat logs. For example, some tools can extract WeChat chat logs by decrypting a local database and save them as HTML, Word, or CSV documents.

[0006] However, these methods primarily target raw data stored on the device, rather than directly processing chat logs in screenshot form. Therefore, this invention aims to fill this technological gap by proposing a multi-dimensional conversation information extraction method specifically for WeChat chat screenshots. Summary of the Invention

[0007] To address the technical challenges of extracting information from WeChat chat screenshots, this invention provides a multi-dimensional method for extracting conversation information from WeChat chat screenshots. The technical solution adopted is as follows:

[0008] A method for extracting multi-dimensional conversation information from WeChat chat screenshots includes the following steps:

[0009] Step 1: Use the improved YOLOv8 model to perform multimodal element joint detection and recognition, and output the element recognition result set. The elements in the element recognition result set include basic elements and legal key elements.

[0010] Step 2: Based on the spatial location and semantic relationship of elements, construct the temporal logic and association of messages, and output a list of message units arranged in the actual temporal order;

[0011] Step 3: Enhance the message unit list based on sender identity calibration, evidence integrity marking, and rich media association, and output the enhanced structured message list;

[0012] Step 4: The enhanced structured message list is then blockchain-anchored and format-converted to generate multi-format files that conform to electronic evidence specifications.

[0013] Optionally, preprocessing may be performed before step 1 detection, including the following steps:

[0014] SSIM uses a sliding window algorithm to detect continuous scrolling features in screenshots and combines image similarity comparison to automatically cut long screenshots into single-screen segments.

[0015] The U²-Net saliency detection model is used to separate the main body of the chat interface from the irrelevant background;

[0016] Dynamically apply CLAHE contrast-limited adaptive histogram equalization and perform non-local means denoising.

[0017] Optionally, the improved YOLOv8 model introduces a bimodal input branch, which includes an image branch and a text location branch. The image branch is a screenshot of the chat. The text location branch takes into account the pre-extracted text box coordinates and preliminary OCR results from the EAST text detector. The coordinate encoding layer converts the text box position into a heatmap at the same scale as the image feature map, and the text semantics are converted into vector features through a word embedding layer.

[0018] The improved YOLOv8 model fuses text heatmaps and image feature maps through feature concatenation after the third CSP module of the backbone network. At the same time, it maps text semantic vectors to the feature map dimension through spatial broadcasting mechanism, realizing the fusion of visual features, text location and text semantics.

[0019] In the upsampling stage of PAN, the text semantic vector is mapped to feature maps of various scales through deconvolution layers, and the weights of semantic features on visual features are dynamically adjusted through a gating mechanism.

[0020] A cross-scale attention module is added to the lateral connections of FPN to calculate the correlation of the same semantic elements in feature maps at different scales.

[0021] Optional basic elements include chat name, avatar, time indicator, text message box, voice or video message box, and other message boxes; other message boxes refer to images, videos, transfers, red envelopes, and emoticons.

[0022] The key elements of the legal process are the cited message, the withdrawal prompt, and the system notification.

[0023] Optionally, step 2 includes the following steps:

[0024] Collision resolution for overlapping elements: For detected message bubbles, calculate the difference in Y-coordinates between adjacent elements. If the difference in Y-coordinates is less than 10% of the bubble height, further calculate the IoU value.

[0025] If the IoU is greater than the similarity threshold, the X-coordinate-dominated judgment is initiated. The X-coordinate-dominated judgment is that in a group chat, left-aligned bubbles are sent by others, and right-aligned bubbles are sent by oneself, and sorted by left and right partitions.

[0026] If there is a reference relationship, the arrow or pointing line of the referenced message is identified by the edge detection algorithm to establish a chain association between the referenced message and the reply message;

[0027] When more than 3 identical timestamps are detected, they are automatically collapsed into one anchor point;

[0028] For messages without timestamps, a virtual timestamp is generated using linear interpolation based on the interval between known timestamps before and after it, and then marked as the inferred time.

[0029] Optionally, step 3 includes the following sub-steps:

[0030] Step 31: Identify the main color of the bubble using a color clustering algorithm to assist in identity verification and achieve sender authentication;

[0031] Step 32: Detect nicknames and message identifiers in the group chat, combine OCR to identify nicknames, construct a nickname message mapping table, and mark whether it is the group owner;

[0032] Step 33: Use edge detection to identify whether the message bubble is truncated. If the message bubble is truncated, mark it as incomplete. Use a semantic segmentation model to identify mosaic and smeared areas, mark the content as occluded, and record the location.

[0033] Step 34: Rich media content association, extract thumbnails from the attachment area, calculate SHA-256 hash values, and record the storage path; extract the duration of voice messages through duration text, and generate emotion tags by combining the texture features of the voice waveform.

[0034] Optionally, in step 4, the original screenshot hash value is calculated for each element of the enhanced structured message list, and the log data of the screenshot hash value processing is recorded, and blockchain anchoring is performed based on the log data;

[0035] The enhanced structured message list, anchored to the blockchain, generates legally compatible JSON, PDF, and Excel files. The JSON file is used to display element information and the message list, the PDF file is used to display text and image comparisons, and the Excel file is used to display a timeline.

[0036] Optionally, the improved YOLOv8 model can be trained and tested using the Ultralytics framework based on the deep learning framework PyTorch.

[0037] A memory that stores a multi-dimensional conversation information extraction program designed using a multi-dimensional conversation information extraction method based on WeChat chat screenshots.

[0038] A computer runs a multi-dimensional conversation information extraction program designed using a multi-dimensional conversation information extraction method based on WeChat chat screenshots, and outputs multi-dimensional conversation information extraction results.

[0039] In summary, this invention offers at least one of the following beneficial technical effects: The improved YOLOv8 model simultaneously identifies basic elements and key legal elements, solving the problem that traditional methods can only identify routine content and miss legally sensitive information, such as retracted messages and system notifications, thus improving the recall rate of core evidence elements. The multimodal fusion detection logic ensures accurate identification of blurry and low-quality screenshots, reducing information distortion caused by image quality. By constructing message sequence through spatial location and semantic association, the problem of disordered ordering of dense group chat messages and long screenshots spanning multiple screens is solved.

[0040] Time anchor calibration ensures the traceability of each message's timeline, meeting the stringent requirements of judicial scenarios for communication timelines and avoiding disputes over the validity of evidence due to inconsistent timelines. Multi-format output directly adapts to the format requirements of judicial authorities for electronic evidence, reducing manual processing costs and improving evidence submission efficiency.

[0041] By optimizing the entire process of precise extraction and time sequence restoration, legal enhancement, and anti-tampering protection, WeChat chat screenshots are transformed from raw images into directly credible electronic evidence. This not only solves the problem of insufficient robustness of traditional methods in complex scenarios, but also deeply adapts to the core requirements of judicial scenarios for the authenticity, completeness, and relevance of evidence, and has significant technical and practical value. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating a method for extracting multi-dimensional conversation information from WeChat chat screenshots according to the present invention. Detailed Implementation

[0043] The present invention will be further described in detail below with reference to the accompanying drawings.

[0044] This invention discloses a method for extracting multi-dimensional conversation information from WeChat chat screenshots.

[0045] Reference Figure 1 Example 1: A method for extracting multi-dimensional conversation information from WeChat chat screenshots, comprising the following steps:

[0046] Step 1: Use the improved YOLOv8 model to perform multimodal element joint detection and recognition, and output the element recognition result set. The elements in the element recognition result set include basic elements and legal key elements.

[0047] Step 2: Based on the spatial location and semantic relationship of elements, construct the temporal logic and association of messages, and output a list of message units arranged in the actual temporal order;

[0048] Step 3: Enhance the message unit list based on sender identity calibration, evidence integrity marking, and rich media association, and output the enhanced structured message list;

[0049] Step 4: The enhanced structured message list is then blockchain-anchored and format-converted to generate multi-format files that conform to electronic evidence specifications.

[0050] By adopting the above technical solution, the improved YOLOv8 model in step 1 can simultaneously identify basic elements (such as avatars and timestamps) and key legal elements (such as retraction prompts and cited messages), which solves the problem that traditional methods can only identify routine content and miss legally sensitive information, such as retraction messages and system notifications, thus improving the recall rate of core evidence elements.

[0051] The multimodal fusion detection logic ensures that key information in blurry, low-quality screenshots, such as those in low-light conditions or artistic fonts, such as correcting the amount from 5000 yuan to 5000 yuan, can still be accurately identified, reducing information distortion caused by image quality.

[0052] Step 2 constructs message sequence through spatial location and semantic association, which solves the problem of disordered sorting of dense messages in group chats and long screenshots across screens. For example, left-aligned bubbles are sent to others, right-aligned bubbles are sent to oneself, and referenced messages are automatically associated with the original messages.

[0053] Time anchor calibration, such as filling in missing timestamps and folding duplicate times, ensures that the time sequence of each message is traceable, meets the strict requirements of communication timelines in judicial scenarios, and avoids disputes over the validity of evidence due to disordered time sequence.

[0054] Step 3, sender identity verification, such as using bubble color to assist in identity judgment, solves the problem of misjudgment of identity caused by missed avatar detection and clarifies the responsible parties of the session; evidence integrity marking, such as marking: message incomplete or content obscured, directly indicates evidence defects and avoids misjudgment due to missing information.

[0055] Rich media associations, such as image hash values ​​and voice emotion tags, enable the traceability of non-textual content, ensuring the originality and relevance of attachments such as images and voice recordings when used as evidence.

[0056] Step 4, blockchain anchoring, uses the original screenshot hash value and processing logs to form an immutable chain of evidence, solving the core pain point that electronic evidence is easily tampered with and meeting the requirements for authenticity verification in electronic data regulations.

[0057] Multi-format output directly adapts to the format requirements of judicial authorities for electronic evidence, reducing manual processing costs and improving the efficiency of evidence submission.

[0058] By optimizing the entire process of precise extraction, time-series restoration, legal enhancement, and anti-tampering protection, WeChat chat screenshots are transformed from raw images into directly credible electronic evidence. This not only solves the problem of insufficient robustness of traditional methods in complex scenarios, but also deeply adapts to the core requirements of judicial scenarios for the authenticity, completeness, and relevance of evidence, and has significant technical and practical value.

[0059] Example 2: Pre-processing is performed before the detection in step 1, including the following steps:

[0060] SSIM uses a sliding window algorithm to detect continuous scrolling features in screenshots and combines image similarity comparison to automatically cut long screenshots into single-screen segments.

[0061] The U²-Net saliency detection model is used to separate the main body of the chat interface from the irrelevant background;

[0062] Dynamically apply CLAHE contrast-limited adaptive histogram equalization and perform non-local means denoising.

[0063] By employing the above technical solution, the sliding window combined with SSIM image similarity comparison can automatically identify scrolling boundaries in long screenshots, such as repeated navigation bars and scroll bar traces, and cut long screenshots stitched together from multiple screens into independent single-screen segments. This operation makes the screenshot content more compatible with the training scenario of the improved YOLOv8 model, avoids detection errors caused by element compression and scale distortion in long images, and reduces the processing size of a single image, thereby improving the model's running efficiency.

[0064] U²-Net-based saliency detection can accurately distinguish the main elements of the chat interface, such as message bubbles, avatars, and timestamps, from irrelevant backgrounds, such as phone borders, desktop wallpapers, and other application windows, reducing the interference of background noise on subsequent element detection. By eliminating invalid information in non-chat areas, the improved YOLOv8 model can concentrate computing power to identify key elements, significantly reducing the false detection rate and improving the recognition accuracy of basic elements and legally critical elements.

[0065] Dynamic CLAHE adaptively enhances contrast based on local brightness differences in chat screenshots, such as correcting text blur caused by backlighting and improving dark details in nighttime screenshots. Meanwhile, non-local mean denoising removes image noise while preserving key features like text edges and icon details. The combination of these two features effectively improves low-quality screenshots, addressing issues like blurriness, overexposure, and compression distortion. This allows the improved YOLOv8 model to more accurately identify small elements, such as thumbnailed text, icons, and elements with weak features, further enhancing the robustness of element recognition.

[0066] Example 3: The improved YOLOv8 model introduces a bimodal input branch, which includes an image branch and a text location branch. The image branch is a screenshot of a chat. The text location branch takes into account the pre-extracted text box coordinates and preliminary OCR results from the EAST text detector. The coordinate encoding layer converts the text box position into a heatmap of the same scale as the image feature map, and the text semantics are converted into vector features through a word embedding layer.

[0067] The improved YOLOv8 model fuses text heatmaps and image feature maps through feature concatenation after the third CSP module of the backbone network. At the same time, it maps text semantic vectors to the feature map dimension through spatial broadcasting mechanism, realizing the fusion of visual features, text location and text semantics.

[0068] In the upsampling stage of PAN, the text semantic vector is mapped to feature maps of various scales through deconvolution layers, and the weights of semantic features on visual features are dynamically adjusted through a gating mechanism.

[0069] A cross-scale attention module is added to the lateral connections of FPN to calculate the correlation of the same semantic elements in feature maps at different scales.

[0070] By adopting the above technical solution, the image branch retains the original visual feature extraction capability, while the text location branch injects textual semantic information into the model through the text box coordinates pre-extracted by EAST and preliminary OCR results. This dual-modal input solves the problem of missed detection of semantically driven elements in traditional single-vision models. These elements may not have significant visual features, but their textual semantics are highly recognizable. Real-world testing shows an improved recall rate for detecting key legal elements.

[0071] The backbone network fuses text heatmaps (text location) and image feature maps through feature concatenation, and maps text semantic vectors (text content) through spatial broadcasting, achieving a three-stream deep interaction of visual features, text location, and text semantics. This fusion allows the model to simultaneously utilize where the text is and what the text is to assist visual judgment, for example:

[0072] When visual features are blurred, such as a transfer icon in low light, the textual semantics of the transfer amount can help identify the element type.

[0073] When text overlaps, such as dense nicknames in a group chat, the visual position of the avatar can help distinguish who belongs to whom.

[0074] Ultimately, this improves the average accuracy of overall element detection, especially in blurry and dense scenes.

[0075] During the PAN upsampling stage, text semantic vectors are mapped to feature maps at various scales, and semantic weights are dynamically adjusted through a gating mechanism: small-scale feature maps emphasize the text semantics of small elements such as short timestamps for emojis (e.g., the text of a smiley face symbol), while large-scale feature maps emphasize the text semantics of large elements such as long quotes in system notifications (e.g., complete sentences indicating that the other party has enabled friend verification). This scale-adaptive semantic guidance solves the problem of traditional models' one-size-fits-all feature processing for elements of different sizes, improving the detection accuracy of small elements (e.g., emojis) and large elements (e.g., system notifications spanning multiple lines).

[0076] After a long screenshot from WeChat is segmented into single screens, the visual features of the same type of element may exhibit subtle differences across different screens, such as positional shifts and size variations. The cross-scale attention module in the FPN's lateral connections calculates the correlation between semantically identical elements in feature maps at different scales, such as the feature correlation of the 10:00 timestamp in layers P3 and P4. This dynamically calibrates feature biases, ensuring consistent element detection across screens. Real-world testing shows that cross-screen element detection consistency is improved to over 95% after long screenshot segmentation, avoiding misjudgments of similar elements caused by segmentation.

[0077] Example 4: The basic elements are chat name, avatar, time indicator, text message box, voice or video message box, and other message boxes; other message boxes refer to pictures, videos, transfers, red envelopes, and emoticons.

[0078] The key elements of the legal process are the cited message, the withdrawal prompt, and the system notification.

[0079] By adopting the above technical solutions, the basic elements—chat name, avatar, time reminder bar, and various message boxes—form the core framework of WeChat chat history, covering the basic dimensions of who is chatting (chat name, avatar), when chatting (time reminder bar), and what is being discussed (text, voice, images, transfer, and other message boxes).

[0080] This comprehensive division ensures that no basic information of the conversation is omitted. For example, the extraction of the transfer message box can directly link key data such as amount and status, and the time prompt bar provides an anchor point for message chronological sorting, laying a complete foundation for subsequent chronological construction and structured output.

[0081] Clearly defining other message boxes (images, videos, red envelopes, etc.) avoids the loss of non-text information, ensures that rich media content (such as loan screenshots and red envelope records) can be included in the evidence chain, and improves the comprehensiveness of information extraction.

[0082] Key legal elements (quoted messages, withdrawal notices, system notifications) are information with special probative value in judicial evidence collection, and their accurate extraction directly affects the relevance and authenticity of the evidence.

[0083] Citing messages: Clearly define the contextual relationship between the reply and the reply received, avoiding disconnected conversational contexts (e.g., A cites B's loan of 5,000 yuan and replies that it has been transferred; the citation relationship proves the connection between the two), solving the problem that isolated messages cannot form a chain of evidence in traditional extraction.

[0084] Withdrawal Notice: In legal disputes, withdrawal messages often involve parties attempting to destroy evidence. Specifically extracting and marking these messages can help determine the parties' subjective intent and fill the gap in traditional methods that overlook withdrawal behavior.

[0085] System notifications: Notifications such as "The other party has enabled friend verification" and "Message delivered but not read" can reflect the validity of the conversation (e.g., whether communication was successful) and the other party's status (e.g., whether they are deliberately avoiding the issue), providing supplementary evidence for the determination of the facts of the case.

[0086] Example 5, step 2 includes the following steps:

[0087] Collision resolution for overlapping elements: For detected message bubbles, calculate the difference in Y-coordinates between adjacent elements. If the difference in Y-coordinates is less than 10% of the bubble height, further calculate the IoU value.

[0088] If the IoU is greater than the similarity threshold, the X-coordinate-dominated judgment is initiated. The X-coordinate-dominated judgment is that in a group chat, left-aligned bubbles are sent by others, and right-aligned bubbles are sent by oneself, and sorted by left and right partitions.

[0089] If there is a reference relationship, the arrow or pointing line of the referenced message is identified by the edge detection algorithm to establish a chain association between the referenced message and the reply message;

[0090] When more than 3 identical timestamps are detected, they are automatically collapsed into one anchor point;

[0091] For messages without timestamps, a virtual timestamp is generated using linear interpolation based on the interval between known timestamps before and after it, and then marked as the inferred time.

[0092] By adopting the above technical solution, WeChat group chats often feature densely packed messages from multiple users (such as A, B, and C speaking consecutively), causing message bubbles to have similar or even overlapping Y-coordinates. Traditional sorting by Y-coordinate is prone to causing temporal disorder. By using a dual judgment of Y-coordinate difference and IoU value, overlapping elements can be accurately identified.

[0093] By combining WeChat's interface layout rules, the X-coordinate of left-aligned messages (for others' messages) and right-aligned messages (for oneself's messages) is used to determine the message partitions of different senders. The messages are arranged in an orderly manner from the left partition (others' messages) to the right partition (one's own messages), avoiding the problem of cross-user messages being confused in group chats.

[0094] Real-world testing shows that the sorting accuracy of dense messages in group chats is improved, significantly reducing conversational disconnects caused by sorting errors.

[0095] In WeChat, quoting messages (such as replying to XXX: ...) is an important carrier of contextual relationships, but the quoting relationship is often overlooked due to the ambiguity of the arrow / pointing line, resulting in the reply content being disconnected from the content being replied to.

[0096] The solution uses edge detection algorithms to identify arrows / pointing lines in the referenced message (with clear visual features), establishes a chain-like association between the referenced message and the reply message, and ensures that the two are presented in a time sequence (e.g., if A references B's loan of 5,000 yuan, then A's reply follows B's message and is marked as referenced from B).

[0097] This approach is particularly crucial in legal scenarios, as the citation relationship directly proves the relevance between the response and the cited content (e.g., the transfer of funds is a response to a loan request), avoiding disputes over the validity of evidence due to a break in the connection.

[0098] In WeChat chats, repeated timestamps are often omitted (e.g., three consecutive messages are all 10:01, but only the first one shows the time), or some messages may not have timestamps due to screenshot truncation. Traditional methods are prone to timeline breaks.

[0099] Automatically collapse three or more identical timestamps into anchor points (e.g., marked with [multiple messages at 10:01]), which simplifies information while retaining time anchors and avoids redundancy;

[0100] For messages without timestamps, virtual times are generated by linear interpolation of known timestamps before and after the message (e.g., the first time is 10:00, the second time is 10:05, and the three messages in between are interpolated to 10:01, 10:03, and 10:04), and explicitly marked as [inference], which both completes the time sequence and ensures transparency.

[0101] This processing significantly improves the completeness of the timeline, meeting the strict traceability requirements of communication time nodes in judicial scenarios, while the inference markers avoid the impact of false times on the authenticity of evidence.

[0102] Example 6, step 3 includes the following sub-steps:

[0103] Step 31: Identify the main color of the bubble using a color clustering algorithm to assist in identity verification and achieve sender authentication;

[0104] Step 32: Detect nicknames and message identifiers in the group chat, combine OCR to identify nicknames, construct a nickname message mapping table, and mark whether it is the group owner;

[0105] Step 33: Use edge detection to identify whether the message bubble is truncated. If the message bubble is truncated, mark it as incomplete. Use a semantic segmentation model to identify mosaic and smeared areas, mark the content as occluded, and record the location.

[0106] Step 34: Rich media content association, extract thumbnails from the attachment area, calculate SHA-256 hash values, and record the storage path; extract the duration of voice messages through duration text, and generate emotion tags by combining the texture features of the voice waveform.

[0107] By adopting the above technical solution, the stable visual feature of a message bubble sent by oneself being green and another's being gray in a WeChat chat is established. When an avatar is missed (e.g., blurred or truncated), the color clustering algorithm can quickly distinguish one's own / other's identity based on the dominant color of the bubble. This mechanism, as a supplement to avatar recognition, avoids misjudgment due to the failure of a single feature (avatar), ensuring that the accuracy of sender identity determination (especially in low-quality screenshots) is increased to over 95%, providing a reliable basis for determining responsibility for who sent a particular message.

[0108] In group chats, the correspondence between nicknames and messages can easily become confusing due to multiple users speaking. OCR can be used to identify nicknames (e.g., Zhang San: @Li Si) and build a nickname-message mapping table, accurately linking each member's message content. Simultaneously, labeling the group owner / administrator (using indicators like crowns) clarifies the boundaries of management authority and responsibility within the group. This approach is particularly crucial in legal scenarios. For example, in dispute group chats, it can quickly pinpoint who made a claim and who responded, avoiding disputes over the validity of evidence due to confusion of subjects and significantly reducing the cost of manually reviewing group chat records.

[0109] For truncated message bubbles, such as screenshots where only half of the text is displayed at the edge, label them: "Message incomplete." This can directly indicate that key information may be missing, preventing users from misjudging incomplete information as complete evidence.

[0110] The mosaic and smeared areas are identified and labeled using a semantic segmentation model: the content is obscured, and the location is recorded (e.g., the second line of text is obscured). This not only clarifies the flaws in the evidence, but also provides clues for subsequent verification.

[0111] This mechanism complies with the legal requirements for the disclosure of the integrity of electronic evidence, ensuring that users of the evidence can clearly identify information defects and avoid the risk of evidence being invalid due to the concealment of flaws.

[0112] Extracting thumbnail hash values ​​(SHA-256) from attachments such as images and videos allows for verification of whether rich media content has been tampered with (if the hash value remains unchanged, the content has not been modified). At the same time, recording the storage path facilitates the tracing of the original file, thus solving the core pain point that rich media evidence is easily tampered with.

[0113] Extracting the duration of voice messages (e.g., 00:58) and combining it with waveform texture features to generate emotion tags (e.g., urgent and calm) can help determine the communication intent. For example, a voice message urging repayment plus an urgent tag can strengthen the fact that the creditor has repeatedly urged repayment, making non-textual information a link in the effective chain of evidence.

[0114] In Example 7, in step 4, the original screenshot hash value is calculated for each element of the enhanced structured message list, and the log data of the screenshot hash value processing is recorded. Blockchain anchoring is then performed based on the log data.

[0115] The enhanced structured message list, anchored to the blockchain, generates legally compatible JSON, PDF, and Excel files. The JSON file is used to display element information and the message list, the PDF file is used to display text and image comparisons, and the Excel file is used to display a timeline.

[0116] By adopting the above technical solution, the hash value of the original screenshot is calculated, and the screenshot can be directly verified through hash verification to determine whether it has been tampered with, thus solving the core pain point that electronic evidence is easily tampered with.

[0117] Record the full log of screenshot processing (such as splitting a long screenshot into 3 segments, enhancing contrast, etc.), and store it on the blockchain along with the hash value to form an immutable "processing track".

[0118] The distributed ledger feature of blockchain ensures that logs cannot be modified unilaterally, proving the compliance of the entire information extraction process.

[0119] This mechanism directly meets the requirement in relevant regulations on electronic evidence that the authenticity of electronic data must be judged comprehensively based on factors such as storage medium and extraction process, providing dual guarantees of originality and procedural compliance for the judicial acceptance of evidence.

[0120] JSON files: Structured storage of element information (such as sender, timestamp, message type) and message list, suitable for computers to quickly parse and retrieve, meeting the digital management needs of electronic evidence that is machine-readable and associative;

[0121] PDF file: Enables text-to-image comparison of original screenshots and structured information, intuitively presenting the correspondence between the original evidence and the extracted results, facilitating manual verification and court presentation (judges can quickly compare the original images with the extracted content).

[0122] Excel file: Messages are arranged chronologically, clearly showing the time sequence of the conversation (e.g., 8:00 loan request, 9:00 loan approval, 10:00 transfer record), which makes it easier to sort out the key time nodes and the development of facts in the case, and reduces the manual cost for legal professionals to organize the chain of evidence.

[0123] The three formats have clearly defined roles, each covering the core needs of legal scenarios such as machine processing, manual verification, and chronological organization, thus avoiding the problem of low efficiency in the use of evidence due to a single format.

[0124] Example 8: The improved YOLOv8 model is trained and tested using the Ultralytics framework based on the deep learning framework PyTorch.

[0125] By adopting the above technical solutions, the PyTorch-based Ultralytics framework provides full-chain support for improving the YOLOv8 model, including "efficient development, high-performance training, and flexible deployment." This not only reduces the technical difficulty of multimodal element detection but also improves the accuracy and efficiency of the model in WeChat chat screenshot scenarios through framework optimization, providing a stable and reliable underlying model capability for the entire process of subsequent conversation information extraction.

[0126] Example 9: A memory that stores a multi-dimensional conversation information extraction program designed using a multi-dimensional conversation information extraction method based on WeChat chat screenshots.

[0127] Example 10: A computer runs a multi-dimensional conversation information extraction program designed using a multi-dimensional conversation information extraction method based on WeChat chat screenshots, and outputs multi-dimensional conversation information extraction results.

[0128] The following specific embodiments illustrate the implementation principle of the present invention:

[0129] The user (lawyer) received a WeChat chat screenshot (one long screenshot) from the client, containing loan communication records between the client ("self") and the borrower ("Zhang San"), including text messages, transfer screenshots, one retracted message, and one quoted message. The screenshot is partially blurry and has a background with a phone frame. This screenshot needs to be extracted as electronic evidence that meets the court's requirements.

[0130] Specific implementation steps

[0131] Step 1: Preprocessing:

[0132] Input: Original long screenshot (3000×1200 pixels, including 3 screens of chat content, with the phone's desktop background at the bottom, and the text in the middle is unclear due to the blurry image).

[0133] Processing procedure:

[0134] Long screenshot segmentation:

[0135] The sliding window (800 pixels high) slides vertically, calculates the SSIM value of adjacent windows (threshold 0.85), detects 2 duplicate top time bars ("10:00 AM"), and divides the long screenshot into 3 single-screen segments (each segment is 1000×1200 pixels in size), labeled as "Screen 1", "Screen 2", and "Screen 3".

[0136] Background separation:

[0137] The U²-Net model was used to process three single-screen segments, separating the main chat content (message bubble, avatar, timestamp) from the phone frame and desktop background, and outputting an image containing only the chat interface (removing about 20% of the invalid background area).

[0138] Image enhancement:

[0139] CLAHE was applied to enhance the contrast of the blurred transfer amount area ("5000 yuan") on screen 2, and non-local mean denoising was performed on the dark areas of the night shot on screen 3 ("withdrawal prompt" text). The clarity of the text edges was improved after processing.

[0140] Step 2: Multimodal element detection:

[0141] Input: Three pre-processed single-screen chat images.

[0142] Processing procedure (based on an improved YOLOv8 model, trained using PyTorch's Ultralytics framework):

[0143] Dual-modal input:

[0144] Image branch: Input a preprocessed single-screen image;

[0145] Text location branch: The EAST text detector pre-extracts text boxes (such as "Zhang San: Lend me 5000 yuan" "A message has been withdrawn"), the coordinate encoding layer converts them into heatmaps, and the word embedding layer (BERT-small) converts semantics such as "withdraw" and "transfer" into vectors.

[0146] Three-stream fusion and feature optimization:

[0147] After the third CSP module in the backbone network, text heatmaps and image features are fused together, and the semantic vector of "transfer" is mapped through a spatial broadcasting mechanism. In the PAN upsampling stage, the semantic weights of small elements (such as the "smiling" expression) are increased, and the FPN lateral connection calibrates the feature differences of "timestamp 10:01" in the three screens through cross-scale attention.

[0148] Element recognition results:

[0149] Output 9 categories of elements (basic elements + legal key elements), as shown in the example below:

[0150] Basic elements: chat name "Zhang San", green bubble for "self", gray bubble for "Zhang San", timestamps "10:00" and "10:01", transfer message box (including amount "5000 yuan"), emoticon "[smile]";

[0151] Key legal elements: quoted message ("Zhang San quoted his own message: 'Pay back before the end of the month'"), withdrawal notice ("Zhang San withdrew a message"), system notification ("Zhang San has received payment").

[0152] Step 3: Message Sequence and Association Construction:

[0153] Input: The element recognition results (including coordinates and content) output from step 2.

[0154] Processing procedure:

[0155] Resolving conflicts involving overlapping elements:

[0156] In screen 2, the difference in Y coordinate between "Zhang San's message bubble" and "your own reply bubble" is 5 pixels (less than 10% of the bubble height), and IoU=0.35 (greater than the threshold of 0.3). X coordinate judgment is activated: Zhang San's bubble is left-aligned (others), and your own bubble is right-aligned (your own), and sorted by "Zhang San's message → your own reply".

[0157] Reference relationships:

[0158] The arrow pointing to the "My own reply" in screen 3 was detected. The referenced object was identified as "Zhang San's message: 'Lend me 5,000 yuan'" in screen 1 through edge detection. A chain association was established (marked "referenced from screen 1 - Zhang San").

[0159] Time anchor calibration:

[0160] Five consecutive messages in screens 1 and 2 have timestamps of "10:01", which are automatically folded into the anchor point "[10:015 messages]"; one message in screen 3 has no timestamp, and a virtual time "10:03" is generated by linear interpolation based on the known times before and after (10:01 before, 10:05 after), and marked as "inference".

[0161] Step 4: Information Augmentation:

[0162] Input: The list of timing message units output in step 3.

[0163] Processing procedure:

[0164] Sender identity verification:

[0165] In screen 2, the avatar of "Zhang San" was missed due to being blurry. The message bubble of "Zhang San" was identified as gray (other person) by color clustering algorithm, confirming that the sender was "Zhang San". A group chat mapping table was constructed (here it is a one-on-one chat, mapping "self: green bubble, Zhang San: gray bubble").

[0166] Evidence integrity markers:

[0167] The message bubble at the top of Screen 1 is only half-displayed ("Before the end of the month..."), which is identified as truncated by edge detection and marked "[Message incomplete]"; In Screen 3, "Zhang San's retracted message" is obscured by a mosaic, and the obscured area is identified by a semantic segmentation model and marked "Content obscured, location: Screen 3 - line 2".

[0168] Rich media association: Extract thumbnails from transfer screenshots, calculate SHA-256 hash value "a1b2c3d4...", and record the storage path ". / attach / transfer.jpg"; For a voice message (duration text "00:30"), identify it as having an "urgent" emotion through waveform texture features and label it "Voice message: 30 seconds, emotion: urgent".

[0169] Step 5: Blockchain anchoring and multi-format output:

[0170] Input: The enhanced structured message list output from step 4, and the original screenshot.

[0171] Processing procedure:

[0172] Blockchain anchoring:

[0173] Calculate the hash value of the original screenshot "f5g6h7i8...", record the processing log ("2023-10-01 09:00 split the long screenshot into 3 screens; 2023-10-01 09:05 enhance the contrast of screen 2"), upload the hash value, log, and timestamp to the Hyperledger Fabric blockchain, and obtain the evidence number "BLK-20231001-001".

[0174] Multi-format output:

[0175] JSON file: Structured storage of metadata (chat name "Zhang San", evidence hash "f5g6h7i8...") and message list (including sender, time, content, and associated tags for each message);

[0176] PDF file: The left side shows the pre-processed screenshot, and the right side shows the extracted information (such as "Amount: 5000 yuan (corrected), Status: Received" next to the transfer area in the screenshot).

[0177] Excel file: Messages are arranged chronologically (10:00 Zhang San requests a loan, 10:01 Zhang San agrees, 10:03 Transfers funds, 10:05 Zhang San withdraws the message), clearly showing the loan process.

[0178] Final output result:

[0179] The electronic evidence received by the lawyer included:

[0180] A blockchain-based notarized number that verifies the originality of the evidence;

[0181] PDF with accompanying images (for easy presentation in court), Excel spreadsheet with clear timeline (for easy analysis of case details), and machine-readable JSON (for easy batch retrieval).

[0182] All key information was included (including withdrawal notices, citation relationships, and obscuring marks), meeting the court's requirements for the "authenticity, completeness, and relevance" of electronic evidence.

[0183] This embodiment verifies the feasibility of transforming complex screenshots into compliant electronic evidence, highlighting the core value of preprocessing to improve quality, multimodal detection for accurate extraction, and blockchain to ensure authenticity.

[0184] The data collection in this application was done with the individual's consent and will be used in a legal setting.

[0185] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for extracting multi-dimensional conversation information from WeChat chat screenshots, characterized in that, Includes the following steps: Step 1: Use the improved YOLOv8 model to perform multimodal element joint detection and recognition, and output the element recognition result set. The elements in the element recognition result set include basic elements and legal key elements. The basic elements are chat name, avatar, time indicator, text message box, voice or video message box, and other message boxes; other message boxes refer to images, videos, transfers, red envelopes, and emoticons. The key legal elements are the cited message, the withdrawal notice, and the system notification; Step 2: Based on the spatial location and semantic relationship of elements, construct the temporal logic and association of messages, and output a list of message units arranged in the actual temporal order; Step 3: Enhance the message unit list based on sender identity calibration, evidence integrity marking, and rich media association, and output the enhanced structured message list; Step 4: The enhanced structured message list is then blockchain-anchored and format-converted to generate multi-format files that conform to electronic evidence specifications. The improved YOLOv8 model introduces a bimodal input branch, which includes an image branch and a text location branch. The image branch is a screenshot of the chat. The text location branch takes into account the pre-extracted text box coordinates and preliminary OCR results from the EAST text detector. The coordinate encoding layer converts the text box position into a heatmap at the same scale as the image feature map, and the text semantics are converted into vector features through the word embedding layer. The improved YOLOv8 model fuses text heatmaps and image feature maps through feature concatenation after the third CSP module of the backbone network. At the same time, it maps text semantic vectors to the feature map dimension through spatial broadcasting mechanism, realizing the fusion of visual features, text location and text semantics. In the upsampling stage of PAN, the text semantic vector is mapped to feature maps of various scales through deconvolution layers, and the weights of semantic features on visual features are dynamically adjusted through a gating mechanism. A cross-scale attention module is added to the lateral connections of FPN to calculate the correlation of the same semantic elements in feature maps at different scales.

2. The method for extracting multi-dimensional conversation information from WeChat chat screenshots according to claim 1, characterized in that, Step 1 involves preprocessing before testing, including the following steps: SSIM uses a sliding window algorithm to detect continuous scrolling features in screenshots and combines image similarity comparison to automatically cut long screenshots into single-screen segments. The U²-Net saliency detection model is used to separate the main body of the chat interface from the irrelevant background; Dynamically apply CLAHE contrast-limited adaptive histogram equalization and perform non-local means denoising.

3. The method for extracting multi-dimensional conversation information from WeChat chat screenshots according to claim 2, characterized in that, Step 2 includes the following steps: Collision resolution for overlapping elements: For detected message bubbles, calculate the difference in Y-coordinates between adjacent elements. If the difference in Y-coordinates is less than 10% of the bubble height, further calculate the IoU value. If the IoU is greater than the similarity threshold, the X-coordinate-dominated judgment is initiated. The X-coordinate-dominated judgment is that in a group chat, left-aligned bubbles are sent by others, and right-aligned bubbles are sent by oneself, and sorted by left and right partitions. If there is a reference relationship, the arrow or pointing line of the referenced message is identified by the edge detection algorithm to establish a chain association between the referenced message and the reply message; When more than 3 identical timestamps are detected, they are automatically collapsed into one anchor point; For messages without timestamps, a virtual timestamp is generated using linear interpolation based on the interval between known timestamps before and after it, and then marked as the inferred time.

4. The method for extracting multi-dimensional conversation information from WeChat chat screenshots according to claim 3, characterized in that, Step 3 includes the following sub-steps: Step 31: Identify the main color of the bubble using a color clustering algorithm to assist in identity verification and achieve sender authentication; Step 32: Detect nicknames and message identifiers in the group chat, combine OCR to identify nicknames, construct a nickname message mapping table, and mark whether it is the group owner; Step 33: Identify whether the message bubble is truncated through edge detection. If the message bubble is identified as truncated, mark the message as incomplete. Use a semantic segmentation model to identify mosaic and smeared areas, mark the content as occluded, and record the location; Step 34: Rich media content association, extract thumbnails from the attachment area, calculate the SHA-256 hash value, and record the storage path; The duration of voice messages is extracted using duration-texture analysis, and emotion tags are generated by combining the texture features of the voice waveform.

5. A method for extracting multi-dimensional conversation information from WeChat chat screenshots according to claim 4, characterized in that, In step 4, the original screenshot hash value is calculated for each element of the enhanced structured message list, and the log data of the screenshot hash value processing is recorded. Blockchain anchoring is then performed based on the log data. The enhanced structured message list, anchored to the blockchain, generates legally compatible JSON, PDF, and Excel files. The JSON file is used to display element information and the message list, the PDF file is used to display text and image comparisons, and the Excel file is used to display a timeline.

6. A method for extracting multi-dimensional conversation information from WeChat chat screenshots according to claim 5, characterized in that, The improved YOLOv8 model was trained and tested using the Ultralytics framework based on the PyTorch deep learning framework.

7. A computer device comprising a processor, a memory, and a program stored in the memory and executable on the processor, characterized in that: When the program is executed by the processor, it implements the steps of the method for extracting multi-dimensional conversation information from WeChat chat screenshots as described in claim 6.

Images