Intelligent screen projection evidence obtaining method

Abstract

The invention discloses an intelligent screen projection evidence obtaining method which specifically comprises the following steps: carrying out screen capture on a chat interface by utilizing a screen capture API interface of screen projection software, and carrying out block labeling on a captured chat content picture by utilizing labeling software; selecting the width and height of an actual operation interface area of the mobile phone application by using a mouse, reversely controlling an API (Application Program Interface) of a mobile phone screen through screen projection software to control the mobile phone application to automatically turn pages, intercepting a screen projection picture and storing the screen projection picture; carrying out block target detection and block slicing on the screen projection picture by utilizing a picture block target detection and block slicing module to obtain blocks containing text contents; the AI OCR text content recognition module is used for carrying out AI recognition on the blocks containing the text content, and the affiliation relation between the successfully recognized text content and the sliced blocks is stored; and displaying the screenshot and projection picture and the identified text content in a linkage manner, calculating a data structure of the APP screenshot in combination with the coordinate position of each block of the APP screenshot, and storing the data structure in an evidence obtaining case database table to complete intelligent screen projection evidence obtaining.

Description

Technical Field

[0001] This invention relates to the field of electronic data forensics and security, and mainly to an intelligent screen projection forensics method based on labeled and identified chat data. Background Technology

[0002] The field of electronic data forensics is developing rapidly, driven by technological advancements, legal improvements, and expanded applications. However, it also faces challenges such as strong encryption, privacy protection, and technological updates. The following is a summary of the current state of development in electronic data forensics technology: With the widespread adoption of mobile devices, smartphone forensics has become a key area, involving numerous electronic data forensics techniques such as phone unlocking, privilege escalation, data extraction, application reverse engineering and parsing, data format analysis, and encrypted data decryption.

[0003] The current state of electronic data forensics technology is characterized by technological advancements, deepening applications, and a coexistence of challenges. However, the technological challenges remain significant: with the increased frequency of mobile phone system and application updates, the difficulty of unlocking phones, escalating phone privileges, and decrypting encrypted data (especially end-to-end encryption) has increased, significantly increasing the difficulty of evidence collection. Privacy protection and legal compliance issues are particularly prominent in cross-border forensics. The rapid development of artificial intelligence and machine learning technologies has opened up entirely new technological routes for electronic data forensics. Effective use of artificial intelligence can greatly promote intelligent forensics. For example, deep learning can be used to achieve multi-target detection of mobile phone screen content, and AI OCR can be used to recognize text content on the screen, integrating and innovating a completely new electronic data forensics technology—intelligent screen projection forensics technology. Overall, electronic data forensics technology is continuously breaking boundaries through the integration of multiple technologies and scenario adaptation.

[0004] Traditional electronic data forensics techniques, especially those targeting smartphones, typically involve unlocking the phone, escalating privileges, extracting data, reverse engineering and parsing applications, analyzing data formats, and decrypting encrypted data. The high cost and long development cycle of these techniques are current limitations. Furthermore, the timeliness requirements for on-site investigations and audits are extremely high, and current conventional electronic data forensics techniques are increasingly insufficient to meet these needs. A review of domestic patents and online resources revealed no products or patents that integrate mobile screen projection technology, AI multi-target detection, target image slicing recognition, and AI OCR recognition technology into electronic data forensics. Summary of the Invention

[0005] To address the above shortcomings, according to the first aspect of this invention, an intelligent screen projection forensics method based on labeled and recognized chat data is proposed. This method comprehensively utilizes APP screenshot sample labeling and model training technology, mobile screen projection and automatic page-turning screenshot technology, multi-target recognition model technology, artificial intelligence multi-target detection technology, target image slicing technology, and AI OCR text recognition technology to achieve intelligent screen projection forensics. Even in situations where mobile phone privilege escalation is difficult, application reverse engineering and parsing are difficult, encrypted data decryption is difficult, the format of interface display data and application storage data is unknown, and the timeliness requirements for forensics and auditing are high, the WYSIWYG intelligent screen projection forensics method can still complete electronic data forensics work. The specific steps are as follows: S1. Use the screenshot API interface of the screen mirroring software to take screenshots of the chat interface, capture a certain number of chat content images, and use annotation software to annotate the chat content images into blocks; use the CUDA model, PyTorch framework and YOLO algorithm to train the chat image sample model to obtain the YOLO object detection model best.pt; S2. Use the mouse to select the width and height of the actual running interface area of ​​the mobile application (AppErea). , ),in, This indicates the width of the actual running interface area. This indicates the height of the actual running interface area. The API of the screen mirroring software controls the mobile phone screen to automatically turn pages, capture the screen mirroring image and save it. S3. The image block object detection and block slicing module runs in a thread and is loaded in the YOLO object detection model best.pt. It performs block object detection and block slicing on the projected image to obtain blocks containing text content. S4. The AI ​​OCR text content recognition module is loaded into the AI ​​OCR English and Chinese simplified and traditional character recognition model. It performs AI recognition on the blocks containing text content and saves the attribution relationship between the successfully recognized text content and the slice blocks. S5. Link the screenshot projection image with the recognized text content, combine the coordinate positions of each block of the APP screenshot, calculate the data structure of the APP screenshot, and store it in the evidence collection case database table to complete the intelligent projection evidence collection.

[0006] Furthermore, step S1 also includes training a chat image sample model using a CUDA model, the PyTorch framework, and the YOLO algorithm. Specific steps include: Write YOLO model code using PyTorch, defining the network structure and loss function; PyTorch calls CUDA to accelerate model training using GPU; Export the trained YOLO model as TorchScript or ONNX format and run inference on a CUDA-supported GPU.

[0007] Step S1 further includes: Modify the model configuration file yolov8.yaml to ensure that the number of detection categories required for the dataset matches the number of block types, and modify the path of the data loading configuration file; Train the model to obtain the object detection model file best.pt; Using the already trained model file best.pt, the image block object detection and block slicing module is used to verify whether the chat context blocks in the sample images are correctly identified. If there is a deviation, the block annotation needs to be readjusted and the model needs to be retrained. If the chat context blocks are detected correctly, the training ends.

[0008] Furthermore, the specific steps for controlling the automatic page turning of mobile applications via the API of the screen mirroring software to reverse control of the mobile phone screen are as follows: Select the location of the latest message on the mobile application screen and record the entire screen projection area (ScreenErea). , ) and the initial top-left corner coordinates BottomPosition( , Capture and save the screen-casting image using the screen-casting software's screenshot API; By calling the API to control the phone screen's automatic swiping for page turning, the horizontal axis remains unchanged while the vertical axis slides downwards. -20 pixels in height, the screen mirroring image is captured and saved through the screen mirroring software's screenshot API, and each screen mirroring image has a unique name and ID.

[0009] Furthermore, the image block object detection and block slicing module runs in a thread and is loaded into the YOLO object detection model best.pt, specifically including: The image block target detection and block slicing module uses a block detection model to analyze and predict images. The module records each target block and block type, and assigns a unique name and ID to each target block slice. Save the slice image of each target block, including its center point coordinates, width, and height, and denote it as SliceChunk. , , , ), used for AI OCR to recognize text content; Calculate and sort the bottom left corner coordinates of the target block. Based on the block type and bottom left corner coordinates of the slice, obtain the attribution relationship between the chat context slice and the chat member avatar slice. Save the block type of the target slice and the attribution relationship of the chat context slice into a list of slice block objects. For each screenshot image, after block object detection and block slicing are completed, blocks containing text content are used as input data for the AI ​​OCR text content recognition module, based on the block type.

[0010] Furthermore, the AI ​​OCR text content recognition module also includes the following steps: Starting from the first-level block, recursively identify the text content of the next-level blocks and temporarily save it; Based on the block ownership relationship, the identified block content is combined to form a content set with semantic properties; Save the content collection in chronological order of chat times.

[0011] Furthermore, the linked display of the screenshot and the recognized text content specifically includes: The system performs context splicing and deduplication based on the overlap of context blocks and the repetition of context content. First, the recognition results are accurately sorted according to the screenshot time, and advanced AI algorithms, such as Natural Language Processing (NLP) algorithms, are used to scan each text segment. In the context splicing stage, when the system recognizes a chat member's avatar but not their nickname, it filters from the context cache based on the overlap of avatars and the repetition of content. At the same time, the recognition results are deeply analyzed to identify and remove duplicate content.

[0012] According to a second aspect of the present invention, an intelligent screen projection evidence collection system based on labeled and identified chat data is proposed, comprising the following modules: The sample annotation and model training module is configured to use the screenshot API interface of the screen-sharing software to capture screenshots of the chat interface, capture a certain number of chat content images, and use annotation software to perform block annotation on the chat content images; and use the CUDA model, PyTorch framework and YOLO algorithm to train the chat image sample model to obtain the YOLO object detection model best.pt. The image block target detection and block slicing module is configured for mobile screen projection and automatic page-turning screenshot module. It is configured to use the mouse to select the width and height of the actual running interface area of ​​the mobile application, control the mobile application to automatically turn pages through the API of the screen projection software to control the mobile screen, capture the projection image and save it. The image block object detection and block slicing module runs in a thread and is loaded in the YOLO object detection model best.pt. It performs block object detection and block slicing on the projected image to obtain blocks containing text content. The AI ​​OCR text content recognition module is configured to be loaded into the AI ​​OCR English and Chinese simplified and traditional character recognition models, to perform AI recognition on the blocks containing text content, and to save the attribution relationship between the successfully recognized text content and the slice blocks. The linked display function module is configured to link the screenshot projection image with the recognized text content. By combining the coordinate positions of each block of the APP screenshot, the data structure of the APP screenshot is calculated and stored in the evidence collection case database table to complete intelligent screen projection evidence collection.

[0013] According to a third aspect of the present invention, a computer program product is provided, on which one or more computer programs are stored, which, when executed by a computer processor, implement the method described above.

[0014] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects: This invention discloses an intelligent screen projection forensics technology method and system. This method utilizes mobile screen projection technology, multi-object detection technology, YOLO multi-object detection model training technology, AI OCR text recognition module, screenshot image block slicing technology, and coordinate position-based inference of contextual attribution, ensuring the authenticity of original evidence screenshots, improving the efficiency of smartphone application forensics, and significantly reducing the technical difficulty of electronic data forensics. Furthermore, the system integrates screenshot image text content linkage and recognition result search functions, which can intuitively, accurately, and efficiently display the forensic results, truly achieving "what you see is what you get," effectively ensuring the authenticity, correctness, and completeness of the forensic content. It brings an innovative solution to the field of electronic data forensics and is an innovative application combining mobile screen projection and AI technology. Attached Figure Description

[0015] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0016] Figure 1 A flowchart illustrating an intelligent screen projection forensics method based on labeled and identified chat data, according to an embodiment of the present invention, is shown.

[0017] Figure 2 A schematic diagram of a process for extracting text from an original screenshot according to an embodiment of the present invention is shown.

[0018] Figure 3 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application embodiments. Detailed Implementation

[0019] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] Figure 1 A flowchart illustrating an intelligent screen-projection forensics method based on labeled and identified chat data according to an embodiment of the present invention is shown, as follows: Figure 1 As shown: S1. Use the screenshot API of the screen mirroring software to capture screenshots of the chat interface, obtaining a certain number of chat content images. Use annotation software (such as: Meiya Pico APP image annotation software, labelimg, X-AnyLabeling, or other annotation software) to perform block annotation on the chat content images; the sample annotation and model pre-training process is as follows: S1.1 First, define the block type name based on the context of the group chat, such as: A. Title bar area B. Title / Name Block C. Title bar information block D. Chat member avatar block E. Chat member nickname block F. Chat content blocks of chat members G. Chat content sub-block of chat members H. Chat member's chat content reference block I. Chat member's chat content sending time sub-block J. ... blocks S1.2, Annotate group chat screenshot samples, and use annotation tools to annotate the block types of the chat context. Use annotation software to annotate the blocks on the chat content image samples (title bar block, title name, title avatar, title bar information, chat member avatar block, chat member nickname block, chat member chat content block, chat member reply time block, etc.).

[0022] S1.3. Train a group chat image sample model using CUDA, PyTorch framework, and YOLO algorithm. Specific steps include: Write YOLO model code in PyTorch, define the network structure and loss function, including calculating confidence and class probability, extracting features through the backbone network, fusing multi-scale features using the feature pyramid network, generating prediction results through the YOLO output layer, and finally applying anchor boxes to adjust the prediction. PyTorch calls CUDA to accelerate model training using GPUs; the core purpose of calling CUDA in PyTorch is to leverage the parallel computing capabilities of NVIDIA GPUs to significantly accelerate the training and inference process of deep learning models. Export the trained YOLO model as TorchScript or ONNX format and run inference on a CUDA-supported GPU.

[0023] S1.4 Modify the model configuration file yolov8.yaml to make the number of detection categories required for the dataset consistent with the number of block types, and modify the path of the data loading configuration file; Train the model to obtain the object detection model file best.pt; Using the already trained model file best.pt, the image block object detection and block slicing module is used to verify whether the chat context blocks in the sample images are correctly identified. If there is a deviation, the block annotation needs to be readjusted and the model needs to be retrained. If the chat context blocks are detected correctly, the training ends.

[0024] S2. Use the mouse to select the width and height of the actual running interface area of ​​the mobile application (AppErea). , ),in, This indicates the width of the actual running interface area. This indicates the height of the actual running interface area. The API of the screen mirroring software controls the mobile phone screen to automatically turn pages, capture the screen mirroring image and save it. S2.1 When the mobile application interface is first projected onto the computer, select the width and height of the actual running interface area of ​​the mobile application using the mouse. (AppErea) , The purpose is to provide a more accurate reference for using screen mirroring software to control mobile phone screen page turning. The technical process of using screen mirroring software to control the automatic page turning of mobile applications via API is as follows: S2.2 Select the location of the latest message on the mobile application interface and record the entire screen projection area (ScreenErea). , ) and the initial top-left corner coordinates BottomPosition( , Capture and save the screen-casting image using the screen-casting software's screenshot API; S2.3. By calling the API, the phone screen's automatic page-turning function is controlled. The page-turning speed can be set to one page down per second, keeping the horizontal axis constant while the vertical axis slides downwards. -20 pixels in height, minus 20 pixels, to avoid the text being cut into two parts by block slicing, causing the loss of text content during AI OCR recognition. This can be increased or decreased according to actual needs; furthermore, screenshot images are captured and saved through the screen-casting software's screenshot API, and the hash value of each image is calculated and assigned a unique name and ID. After each page of images is saved, the images are pushed to the image block object detection and block slicing.

[0025] The method of controlling the automatic swiping of the phone screen to turn pages by calling the API further includes constructing virtual touch signals, specifically including: S2.3.1 Analyze the projection signal stream, obtain continuous frame images, and establish an interactive channel that can receive and inject virtual touch signals.

[0026] S2.3.2 Perform visual analysis on the acquired frame images, identify the latest message area of ​​the application interface through OCR and object detection, and extract the coordinates of the area as the page turning anchor point.

[0027] S2.3.3 Based on the anchor point position, construct a virtual touch signal equivalent to a real finger swipe. The signal includes the starting coordinates, path trajectory, and speed curve. Trigger a page-turning event through the called API. Inject the virtual touch signal into the mobile terminal through a simulated signal method to drive the interface to perform a page-turning operation.

[0028] S2.3.4 After the page is turned, capture a new frame image, compare it with the frame image before the page was turned, calculate the difference, and determine whether the page content is complete and valid.

[0029] S2.3.5 If the image similarity is too high or the content is missing, a compensatory sliding signal injection is triggered, and the subsequent page turning step size (page turning speed) is dynamically adjusted.

[0030] By injecting virtual touch signals directly into the interaction link layer, the signal injection method offers greater compatibility and is suitable for various software and system environments. Furthermore, the signal injection scheme can simulate the sliding trajectory and speed curve of a real finger, resulting in more flexible and natural control, and even enabling non-linear sliding, thus enhancing the realism and precision of the operation. Combined with image feedback, it can also automatically detect page-turning effects and adjust the sliding step size to avoid overlap or page skipping, improving accuracy.

[0031] S3. The image block object detection and block slicing module runs in a thread and is loaded into the YOLO object detection model best.pt. It performs block object detection and block slicing on the projected image to obtain blocks containing text content. The specific process includes: S3.1 The image block target detection and block slicing module uses a block detection model to analyze and predict images. The module records each target block and block type, and assigns a unique name and ID to each target block slice. S3.2 Save the slice image of each target block, and save the center point coordinates, width, and height, denoted as SliceChunk. , , , ), used for AI OCR to recognize text content; S3.3 Calculate and sort the bottom left corner coordinates of the target block. Based on the block type and bottom left corner coordinates of the slice, obtain the attribution relationship between the chat context slice and the chat member avatar slice. Save the block type of the target slice and the attribution relationship of the chat context slice into a list of slice block objects. S3.4 After completing the block object detection and block slicing of each screenshot image, the blocks containing text content are used as input data for the AI ​​OCR text content recognition module according to the block type.

[0032] In this embodiment, step S3.3 further includes the following algorithm process: S3.3.1 Calculate the coordinates of the bottom left corner and sort the blocks according to the coordinates: The original coordinates (x, y) of the block are the coordinates of the top left corner, and the y value of the bottom left corner should be reduced by the height of the block. Sort by the y-coordinate of the bottom left corner (from bottom to top); If the y-coordinates are the same, then sort them according to the x-coordinate of the bottom left corner (sorted from left to right).

[0033] S3.3.2 Calculate the distance between blocks and the overlapping area: The relationship between two blocks is determined by calculating the overlapping area between them. If there is an overlapping area, they are more likely to belong to the same member. The Euclidean distance is used to calculate the distance between the bottom left corners of two blocks to determine if they are close in relative position. The formula is: s = ,in,( , )and( , () represent the coordinates of two points at different locations; The algorithm calculates the difference in block size as one of the features of the membership relationship. The specific process is as follows: the features of each pair of blocks are stored in a list, and all features are eventually converted into a NumPy array, including the geometric features between each pair of blocks (such as distance, overlapping area, and size difference). The `classify_relationships` function is called to train a classifier using the generated features, and the classifier is returned. Each pair of blocks in the list is iterated through; if the two blocks are of different types, their features are calculated, focusing only on the features between blocks of different types. The trained classifier is then used to predict whether the features of the current pair of blocks belong to the same member. The `classify_relationships` function is a machine learning method that can train and return a classifier based on feature data.

[0034] S3.3.3 Establishing an attribution relationship model: A weighted KNN model is trained using the K-Nearest Neighbors algorithm. The model predicts the relationship between chat context and avatars based on the aforementioned features. Specifically, Euclidean distance, overlap area, and size difference are used as inputs to the weighted KNN model. The K value is adaptively selected through cross-validation and local density estimation. The training formula is as follows: ,in, This represents the overall influence of the i-th neighbor sample on the prediction result of the current sample, i.e., the importance weight. This represents the distance between the current sample and its i-th neighbor. Represents distance weight, distance The smaller the value, the larger the distance to the weight. It's a very small number, so avoid having a denominator of 0. This represents the decay weight over the time interval. This represents the time difference between neighboring samples and the current sample. The exponential decay function means that newer data has a greater impact, while older data has a smaller impact. This represents the time decay coefficient, which controls the decay rate. This represents the similarity between the images. If two blocks have a high similarity, the weight of that neighbor will be amplified; if they are completely dissimilar, the weight will approach 0. After training, the relationship between new block pairs is predicted, the features of the new block pairs are calculated, and the trained weighted KNN model is used to predict whether they belong to the same member.

[0035] After each screenshot image completes block object detection and block slicing, blocks containing text content are pushed to the AI ​​OCR text content recognition module for text recognition based on the block type.

[0036] S4. The AI ​​OCR text content recognition module is loaded into the AI ​​OCR English and Chinese simplified and traditional character recognition models. It performs AI recognition on the blocks containing text content and saves the attribution relationship between the successfully recognized text content and the slice blocks; specifically including: S4.1 Starting from the first-level block, recursively identify the text content of the next-level blocks and temporarily save it; S4.2 Based on the block ownership relationship, combine the block identification content to form a content set with semantic properties; S4.3 Save the content collection in order of chat time.

[0037] By understanding the hierarchical relationships between blocks, textual information belonging to the same category or related to each other can be grouped together, helping the system understand the semantic relationships of the block content. For example, textual information on the same topic or in the same dialogue can be grouped together instead of being stored in isolation.

[0038] When saving content, it is arranged in chronological order of the chat to ensure the timeliness of information. In chat or conversational data, chronological order is crucial, as it determines the context and coherence of the conversation. By arranging content chronologically, the system can better understand the progress of the conversation, changes in emotions, or the flow of information.

[0039] S5. Link the screenshot projection image with the recognized text content, combine the coordinate positions of each block in the APP screenshot, calculate the data structure of the APP screenshot, and store it in the evidence collection case database table to complete intelligent screen projection evidence collection. Specifically, this also includes: S5.1. Contextual stitching and deduplication are performed based on the overlap of context blocks and the repetition of contextual content. This function first accurately sorts the recognition results according to the screenshot time, laying an orderly foundation for subsequent analysis and processing. Then, advanced AI algorithms, such as Natural Language Processing (NLP) algorithms, are used to conduct a comprehensive and detailed scan of each text segment. In the context stitching stage, when the system encounters situations such as recognizing a chat member's avatar but not their nickname, it will filter from the context cache based on the overlap of avatars and the repetition of content.

[0040] In this embodiment, the ContextMerger algorithm is applied to the context reconstruction scenario of group chat message screenshots, involving multiple batches of screenshots uploaded by users on different devices.

[0041] The input consists of group chat screenshots from multiple devices. The data structure for each screenshot is as follows: timestamp: Screenshot time (e.g., "2025-09-08 21:35:12"); avatar: The byte stream of the user's avatar image; text_content: The text parsed from the screenshot; sender_id: A unique identifier for the message sender; source_id: Screenshot file identifier; device_id: Identifier of the device from which the screenshot originated.

[0042] Timestamps from different devices may have an error of a few seconds. This embodiment adds a time alignment module to sort_by_timestamp: Correct the timestamp by calibrating the network clock deviation uploaded by the device; A sender_id constraint is introduced during sorting to ensure that consecutive messages from the same user are aggregated first. The time alignment module resolves the issue of conversation out-of-order delivery caused by cross-device time drift.

[0043] In the _merge_contexts method, this embodiment uses hierarchical concatenation: If adjacent messages have the same sender_id and the interval is less than 30 seconds, they will be forcibly merged. Furthermore, multimodal fusion similarity calculation is used: pHash is used to calculate profile picture similarity; Text similarity is calculated using TF-IDF and semantic vectors (BERT Embedding); The temporal distribution characteristics of the Gaussian kernel function are introduced to measure the temporal density between messages; Overall similarity: ; in, This represents the normalization function, ensuring that the similarity is within the interval [0,1]. Indicates the similarity of profile pictures. Indicates text similarity. Indicates the message time interval. Indicates the similarity in time allocation. , , , and Indicates an adjustable parameter. and This represents the nonlinear exponential parameter, emphasizing the contribution of the modes. This represents the decay weight over the time interval. This represents the time decay coefficient, which reduces the merging probability over long time intervals. This indicates the interaction between profile pictures and text. When profile pictures and text are similar, they tend to be merged. Items representing the interaction between text and time; consecutive and similar messages are more likely to belong to the same context.

[0044] If the number of participants in a group chat is greater than 5, lower the merging threshold (to avoid excessive splitting); If it is a private chat, increase the text similarity weight β; Furthermore, the _combine_fragments function is called. If screenshots with different device_ids contain the same sender_id and similar text_content (similarity > 0.8), they are considered to be redundant messages from multiple devices and are merged into one fragment. Each context fragment stores a list of sources, supporting cross-device tracing and resolving the redundancy issue of different devices repeatedly uploading the same chat.

[0045] Based on the merging engine _deduplicate, semantic hashing is introduced: Approximate hash values ​​are generated by using text vectors to perform dimensionality reduction (LSH); By using fuzzy matching, redundant messages with "different punctuation" or "slight OCR errors" are removed, significantly improving deduplication robustness.

[0046] Finally, the output is a context sequence, with each record containing: the context start time, the user IDs involved in the context, the set, the complete conversation pieced together in order, the list of original screenshot IDs, and the set of source device IDs.

[0047] For example, output sample: { "timestamp": "2025-09-08 21:35:12", "participants": ["user_A", "user_B"], "text_content": "user_A: What time is the meeting tonight? user_B: Around 8 o'clock, is that convenient for everyone?" "sources": ["img_021", "img_022", "img_045"], "devices": ["iphone_001", "android_007"] } In summary, this algorithm introduces multimodal fusion, utilizing avatar image features, text semantic features, and temporal distribution features to improve the accuracy of context splicing; at the same time, it introduces semantic hashing, enabling high-quality deduplication even under OCR errors and slight text changes.

[0048] Specifically, the system selects the nickname corresponding to the avatar with the highest overlap (meeting a preset minimum repetition requirement) and assigns it to the avatar for which no nickname was identified. This cleverly achieves effective contextual splicing, greatly improving the coherence and completeness of the content. Simultaneously, to ensure accuracy, this function utilizes powerful text analysis algorithms to deeply analyze the recognition results, accurately identifying and removing duplicate content. This deduplication process is not simply deleting duplicate text; it intelligently determines which content is truly redundant based on semantic and contextual understanding, effectively avoiding the accidental deletion of key information and ensuring the accuracy and conciseness of the final presented content. The processed screenshots are automatically cached by the system for later retrieval and viewing, providing efficient and reliable data support for the entire evidence collection process.

[0049] S5.2 Recognition Result Search Function: The search function can quickly assist evidence collectors in locating relevant chat subjects by accurately setting keywords based on chat content, nicknames, and time, thereby efficiently locking in key information and facilitating evidence extraction and subsequent data analysis.

[0050] The hardware and software resources required for this invention include: a mobile phone, a computer, a USB data cable or wireless screen projection protocol, screen projection software, an APP image annotation software, a YOLO model training module, a YOLO object detection module, and an AI OCR text recognition module, etc. The intelligent screen projection evidence collection system runs on the computer. This invention mainly includes five technical method functional modules: 1. Sample annotation and model training; 2. Mobile phone screen projection and automatic page-turning screenshot; 3. Screenshot image block object detection and block slicing; 4. AI OCR text content recognition; 5. Linked display of screenshot images and recognized text content. Based on the above-mentioned resources, this invention discloses an intelligent screen projection evidence collection system based on annotated and recognized chat data, characterized by including the following modules: The sample annotation and model training module is configured to use the screenshot API interface of the screen-sharing software to capture screenshots of the chat interface, capture a certain number of chat content images, and use annotation software to perform block annotation on the chat content images; and use the CUDA model, PyTorch framework and YOLO algorithm to train the chat image sample model to obtain the YOLO object detection model best.pt. The mobile screen mirroring and automatic page-turning screenshot module is configured to use the mouse to select the width and height of the actual running interface area of ​​the mobile application, and control the mobile application to automatically turn pages through the API of the screen mirroring software to control the mobile screen, capture the screen mirroring image and save it. The image block object detection and block slicing module runs in a thread and is loaded in the YOLO object detection model best.pt. It performs block object detection and block slicing on the projected image to obtain blocks containing text content. The AI ​​OCR text content recognition module is configured to be loaded into the AI ​​OCR English and Chinese simplified and traditional character recognition models, to perform AI recognition on the blocks containing text content, and to save the attribution relationship between the successfully recognized text content and the slice blocks. The linked display function module is configured to link the screenshot projection image with the recognized text content. By combining the coordinate positions of each block of the APP screenshot, the data structure of the APP screenshot is calculated and stored in the evidence collection case database table to complete intelligent screen projection evidence collection.

[0051] According to a specific embodiment of the present invention, Figure 2 The diagram illustrates a process for text extraction based on the original screenshot according to an embodiment of the present invention, such as... Figure 2 As shown: The original image was a screenshot taken by projecting the phone screen onto the computer screen. The hash value of the image was calculated and stored in the evidence case database table.

[0052] The target area block is the area object on the UI where the original data of the APP is presented, and it is also recognizable by the human eye.

[0053] YOLO inference regions are regions identified by the APP's recognition model. Sub-regions are then inferred by combining the coordinates of the identified regions, thus achieving the recognition of various regions.

[0054] OCR text recognition works by segmenting the YOLO inference area into image blocks with attribute markers (group name, avatar, nickname, chat content text, time, etc.) and then performing OCR text recognition.

[0055] Finally, by combining the coordinates of each block in the APP screenshot, the data structure of the APP screenshot is deduced and calculated, and stored in the evidence collection case database table, thus completing the intelligent screen projection evidence collection.

[0056] The method and system of this invention are also applicable to the structured recognition of computer APP interface data captured by computer screenshots and cameras.

[0057] The following is for reference. Figure 3 It shows a schematic diagram of the structure of a computer system 300 suitable for implementing electronic devices according to embodiments of the present application. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0058] like Figure 3 As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 302 or programs loaded from storage section 308 into random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the system 300. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0059] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a liquid crystal display (LCD) and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card and a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. A removable medium 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 310 as needed so that computer programs read from it can be installed into storage section 308 as needed.

[0060] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0061] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0062] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0063] The modules described in the embodiments of this application can be implemented in software or in hardware.

[0064] On the other hand, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: use the screenshot API interface of the screen mirroring software to take screenshots of the chat interface, capturing a certain number of chat content images; use annotation software to annotate the chat content images into blocks; and use a mouse to select the width and height of the actual running interface area of ​​the mobile application (AppErea). , ),in, This indicates the width of the actual running interface area. The height of the actual running interface area is represented by the API of the screen mirroring software, which controls the mobile application to automatically flip pages, capture and save the screen mirroring image; the image block object detection and block slicing module runs in a thread and is loaded in the YOLO object detection model best.pt, and performs block object detection and block slicing on the screen mirroring image to obtain blocks containing text content; the AI ​​OCR text content recognition module is loaded in the AI ​​OCR English and Chinese simplified and traditional character recognition model, and performs AI recognition on the blocks containing text content, and saves the attribution relationship between the successfully recognized text content and the sliced ​​blocks; the screen mirroring image and the recognized text content are linked and displayed, and the data structure of the APP screenshot is calculated by combining the coordinate position of each block of the APP screenshot, and stored in the evidence case database table to complete the intelligent screen mirroring evidence collection.

[0065] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for intelligent screen projection for evidence collection based on labeled and recognized chat data, characterized in that, include: S1. Use the screenshot API interface of the screen mirroring software to take screenshots of the chat interface, capture a certain number of chat content images, and use annotation software to annotate the chat content images into blocks; use the CUDA model, PyTorch framework and YOLO algorithm to train the chat image sample model to obtain the YOLO object detection model best.pt; S2. Use the mouse to select the width and height of the actual running interface area of ​​the mobile application. ,in, This indicates the width of the actual running interface area. This indicates the height of the actual running interface area. The API of the screen mirroring software controls the mobile phone screen to automatically turn pages, capture the screen mirroring image and save it. S3. The image block object detection and block slicing module runs in a thread and is loaded in the YOLO object detection model best.pt. It performs block object detection and block slicing on the projected image to obtain blocks containing text content. S4. The AI ​​OCR text content recognition module is loaded into the AI ​​OCR English and Chinese simplified and traditional character recognition model. It performs AI recognition on the blocks containing text content and saves the attribution relationship between the successfully recognized text content and the slice blocks. S5. Link the screenshot projection image with the recognized text content, combine the coordinate positions of each block of the APP screenshot, calculate the data structure of the APP screenshot, and store it in the evidence collection case database table to complete the intelligent projection evidence collection.

2. The intelligent screen projection evidence collection method based on labeled and recognized chat data according to claim 1, characterized in that, Also includes: Modify the chat image sample model configuration file yolov8.yaml to make the number of categories to be detected in the dataset consistent with the number of block types, and modify the path of the data loading configuration file; Train the model to obtain the object detection model file best.pt; Using the already trained model file best.pt, the image block object detection and block slicing module is used to verify whether the chat context blocks in the sample images are correctly identified. If there is a deviation, the block annotation needs to be readjusted and the model needs to be retrained. If the chat context blocks are detected correctly, the training ends.

3. The intelligent screen projection evidence collection method based on labeled and recognized chat data according to claim 1, characterized in that, The image block object detection and block slicing module runs in a thread and is loaded into the YOLO object detection model best.pt. Specifically, it also includes: The image block target detection and block slicing module uses a block detection model to analyze and predict images. The module records each target block and block type, and assigns a unique name and ID to each target block slice. Save the slice image of each target block, including the coordinates of the image's center point, width, and height, denoted as SliceChunk. , , , ), used for AI OCR to recognize text content; Calculate and sort the bottom left corner coordinates of the target block. Based on the block type and bottom left corner coordinates of the slice, obtain the attribution relationship between the chat context slice and the chat member avatar slice. Save the block type of the target slice and the attribution relationship of the chat context slice into a list of slice block objects. For each screenshot image, after block object detection and block slicing are completed, blocks containing text content are used as input data for the AI ​​OCR text content recognition module, based on the block type.

4. The intelligent screen projection evidence collection method based on labeled and recognized chat data according to claim 1, characterized in that, The AI ​​OCR text content recognition module also includes the following steps: Starting from the first-level block, recursively identify the text content of the next-level blocks and temporarily save it; Based on the block ownership relationship, the identified block content is combined to form a content set with semantic properties; Save the content collection in chronological order of chat times.

5. The intelligent screen projection evidence collection method based on labeled and recognized chat data according to claim 1, characterized in that, The linked display of screenshot images and recognized text content specifically includes: context splicing and deduplication based on the overlap of context blocks and the repetition of context content; firstly, the recognition results are accurately sorted according to the screenshot time, and natural language processing (NLP) algorithms are used to scan each text segment; in the context splicing stage, when the system recognizes a chat member's avatar but not their nickname, it will filter from the context cache based on the overlap of avatars and the repetition of content; at the same time, the recognition results are deeply analyzed to identify and remove duplicate content.

6. The intelligent screen projection evidence collection method based on labeled and recognized chat data according to claim 1, characterized in that, The API control of the mobile application to automatically turn pages via screen mirroring software includes calling the API to control the mobile screen to automatically swipe the screen, specifically including: The projected signal stream is analyzed to obtain continuous frame images, and an interactive channel that can receive and inject virtual touch signals is established. Visual analysis is performed on the acquired frame images. The latest message area of ​​the application interface is identified through OCR and object detection, and the coordinates of this area are extracted as page turning anchor points. Based on the position of the anchor point, a virtual touch signal equivalent to a real finger swipe is constructed. The signal includes the starting coordinates, path trajectory, and speed curve. The page turning event is triggered through the called API. The virtual touch signal is injected into the mobile terminal through a simulated signal method to drive the interface to perform the page turning operation. After the page is turned, a new frame image is captured and compared with the frame image before the page is turned. The difference is calculated to determine whether the page-turned content is complete and valid. If the image similarity is too high or the content is missing, a compensatory sliding signal injection is triggered, and the subsequent page-turning step size is dynamically adjusted.

7. The intelligent screen projection evidence collection method based on labeled and recognized chat data according to claim 1, characterized in that, Step S3 further includes establishing an attribution relationship model, which is trained using the K-nearest neighbor algorithm.

8. The intelligent screen projection evidence collection method based on labeled and recognized chat data according to claim 7, characterized in that, The K-Nearest Neighbors algorithm is used to train a weighted KNN model. The specific algorithm process is as follows: The parameter features, including Euclidean distance, overlap area, and size difference, are used as input to the weighted KNN model; the K value is adaptively selected through cross-validation and local density estimation; the training formula is... ,in, This represents the combined influence of the i-th neighbor sample on the prediction result of the current sample. This represents the distance between the current sample and its i-th neighbor. Indicates distance weight, For numbers that approach 0 but are not equal to 0, This represents the decay weight over the time interval. Indicates time difference, Indicates the time decay coefficient. Indicates the similarity between profile pictures.

9. A smart screen projection evidence collection system based on labeled and recognized chat data, characterized in that, Includes the following modules: The sample annotation and model training module is configured to use the screenshot API interface of the screen-sharing software to capture screenshots of the chat interface, capture a certain number of chat content images, and use annotation software to perform block annotation on the chat content images; and use the CUDA model, PyTorch framework and YOLO algorithm to train the chat image sample model to obtain the YOLO object detection model best.pt. The mobile screen mirroring and automatic page-turning screenshot module is configured to use the mouse to select the width and height of the actual running interface area of ​​the mobile application, and control the mobile application to automatically turn pages through the API of the screen mirroring software to control the mobile screen, capture the screen mirroring image and save it. The image block object detection and block slicing module runs in a thread and is loaded in the YOLO object detection model best.pt. It performs block object detection and block slicing on the projected image to obtain blocks containing text content. The AI ​​OCR text content recognition module is configured to be loaded into the AI ​​OCR English and Chinese simplified and traditional character recognition models, to perform AI recognition on the blocks containing text content, and to save the attribution relationship between the successfully recognized text content and the slice blocks. The linked display function module is configured to link the screenshot projection image with the recognized text content. By combining the coordinate positions of each block of the APP screenshot, the data structure of the APP screenshot is calculated and stored in the evidence collection case database table to complete intelligent screen projection evidence collection.

Images