Mobile terminal evidence obtaining method and device based on OCR (Optical Character Recognition) and serial port simulation operation

Abstract

The invention discloses a mobile terminal evidence obtaining method and device based on OCR recognition and serial port simulation operation, and the method comprises the steps: obtaining a current interface image, and carrying out the OCR recognition, and obtaining an OCR recognition result; and combining the recognized text with the cue word, inputting the combined text and cue word into a large language model, determining a final control interface to which the current interface image belongs, a final interface state, a target control position and a final next-step action plan according to a reasoning result and an OCR recognition result, and generating a current serial port simulation operation instruction. On the basis of the current serial port simulation operation instruction, the mapping model and the OCR recognition result, executing a corresponding interaction action on a final control interface to which the current interface image belongs, and obtaining a next interface image; and repeating the above steps until the target process is completed by executing interaction actions in sequence, completing evidence collection related settings based on the target process, and performing data collection to obtain an evidence collection result. According to the invention, automatic and high-accuracy evidence obtaining can be realized.

Description

Technical Field

[0001] This invention relates to the field of evidence collection, specifically to a mobile terminal evidence collection method and apparatus based on OCR recognition and serial port simulation operation. Background Technology

[0002] In the field of digital forensics, data from mobile devices, especially smartphones, has become a crucial source of electronic evidence. To extract this data, it is typically necessary to manually enable developer options, USB debugging mode, or configure a Wi-Fi forensic network on the mobile device for data access. These operations heavily rely on human experience and standardized procedures, which are both labor-intensive and carry a significant probability of error. In complex or remote scenarios, technicians cannot quickly operate the target device, further limiting the efficiency of mobile forensics.

[0003] While some remote control and automation scripting technologies exist, they generally suffer from the following shortcomings:

[0004] 1. The interface relies on fixed logic, which cannot adapt to the differences between different brands and systems;

[0005] 2. Lacks the ability to boot and simulate operations on non-rooted phones;

[0006] 3. It is difficult to achieve a truly unattended and continuous evidence collection process.

[0007] The aforementioned process relies heavily on manual operation, which is not only inefficient but also difficult to standardize and automate in a batch case testing environment, failing to meet the needs of unattended evidence collection scenarios. Therefore, there is an urgent need for a new solution that combines image recognition, intelligent judgment, and physical layer control to overcome traditional technical bottlenecks and achieve an efficient, secure, and automated mobile phone evidence collection process. Summary of the Invention

[0008] The purpose of this application is to propose a mobile terminal forensics method and device based on OCR recognition and serial port simulation operation to address the aforementioned technical problems.

[0009] In a first aspect, the present invention provides a mobile terminal forensics method based on OCR recognition and serial port simulation operation, comprising the following steps:

[0010] S1, Construct a mapping model between the preset moving distance based on serial port simulation operation and the actual moving distance of the corresponding control point in the interface displayed on the screen of the mobile terminal;

[0011] S2, acquire the current interface image and perform OCR recognition to obtain the OCR recognition result, which includes the text and its corresponding coordinates;

[0012] S3 combines the text and prompt words and inputs them into the large language model, outputting the reasoning results. The reasoning results include the control interface to which the current interface image belongs, the interface state, and the next action plan. Based on the reasoning results and OCR recognition results, the final control interface to which the current interface image belongs, the final interface state, the position of the target control, and the final next action plan are determined.

[0013] S4. Based on the final control interface to which the current interface image belongs, the final interface state, the position of the target control, and the final next action plan, generate the current serial port simulation operation command. Based on the current serial port simulation operation command, the mapping model, and the OCR recognition result, execute the corresponding interactive action on the final control interface to which the current interface image belongs to switch to the next interface and obtain the next interface image.

[0014] S5, take the next interface image as the current interface image and repeat steps S2-S4 until the target process is completed by executing interactive actions in sequence. Based on the target process, complete the evidence collection-related settings and collect data to obtain the evidence collection results.

[0015] Preferably, step S1 specifically includes:

[0016] The system controls the corresponding control point on the screen of the mobile terminal to move a preset distance along a specified direction by means of preset serial port simulated operation commands.

[0017] The interface images of the corresponding control point in the screen of the mobile terminal are obtained before and after the process of moving a preset distance in a specified direction. The optical flow analysis or image difference analysis is performed in combination with OCR recognition technology to detect the coordinate changes of the control point in the interface images before and after the process is executed.

[0018] The coordinate difference is identified by the coordinate change of the control point in the interface image before and after the process is executed, so as to calculate the actual movement distance corresponding to the interface displayed on the screen of the mobile terminal at the preset movement distance.

[0019] Repeat the above steps under multiple specified directions and preset movement distances to establish a mapping model between the preset movement distance based on serial port simulation operation and the actual movement distance of the corresponding control point in the interface displayed on the mobile terminal screen.

[0020] Preferably, the prompts include the currently executing forensic task, a preset interface keyword library, interface differences between different mobile terminals, and preset state flows; the control interface includes the main menu, settings, the interface corresponding to the about device or developer options; the next action plan includes clicking a button, swiping up, swiping down, or entering a password.

[0021] Preferably, based on the reasoning results and OCR recognition results, the final control interface to which the current interface image belongs, the final interface state, the position of the target control, and the final next action plan are determined, specifically including:

[0022] The text in the OCR recognition result is compared with the preset interface keyword library to determine the control interface to which the current interface image of the OCR recognition belongs. It is then determined whether the control interface to which the current interface image of the OCR recognition belongs is consistent with the control interface to which the current interface image of the inference result belongs. If so, the control interface to which the current interface image belongs, the interface state, and the next action plan of the current interface image in the inference result are taken as the final control interface, the final interface state, and the final next action plan of the current interface image. The position of the target control is obtained based on the coordinates in the OCR recognition result. Otherwise, an abnormal recovery process is triggered. The abnormal recovery process includes repeating step S2 to re-perform OCR recognition on the current interface image, adjusting the sliding direction of the current step action plan, or returning to the previous interface and repeating steps S2-S3.

[0023] Preferably, the target process includes enabling USB debugging operation, WiFi cloning forensics, or direct connection forensics. The operation coordinates contained in the serial port simulation operation instructions are obtained by coordinate transformation based on the mapping model and the OCR recognition results. The action instructions contained in the serial port simulation operation instructions are determined based on the final interface state and the final next action plan.

[0024] Preferably, step S4 further includes:

[0025] Determine whether the final control interface to which the current interface image belongs is the target interface. The target interface includes the interface or main screen interface to be entered after executing the previous serial port simulation operation command. If so, the generated current serial port simulation operation command is the serial port simulation operation command that controls the execution of the corresponding interactive action on the final control interface to which the current interface image belongs. Otherwise, the generated current serial port simulation operation command is the interface displayed on the screen of the mobile terminal by sliding and adjusting it. Repeat the above determination process until the final control interface to which the current interface image belongs is the target interface.

[0026] Secondly, the present invention provides a mobile terminal forensics device based on OCR recognition and serial port simulation operation, comprising:

[0027] The model building module is configured to build a mapping model between a preset moving distance based on serial port simulation operation and the actual moving distance of the corresponding control point in the interface displayed on the screen of the mobile terminal.

[0028] The OCR recognition module is configured to acquire the current interface image and perform OCR recognition to obtain the OCR recognition result, which includes the text and its corresponding coordinates.

[0029] The reasoning module is configured to combine text and prompt words and input them into a large language model, and output reasoning results. The reasoning results include the control interface to which the current interface image belongs and its next action plan. Based on the reasoning results and OCR recognition results, the final control interface to which the current interface image belongs, the final interface state, the position of the target control and its final next action plan are determined.

[0030] The execution module is configured to generate the current serial port simulation operation command based on the final control interface to which the current interface image belongs, the final interface state, the position of the target control, and the final next action plan. Based on the current serial port simulation operation command, the mapping model, and the OCR recognition result, it executes the corresponding interactive action on the final control interface to which the current interface image belongs to switch to the next interface and obtain the next interface image.

[0031] The repeat module is configured to use the next interface image as the current interface image and repeat the OCR recognition module to the execution module until the target process is completed by sequentially executing interactive actions. Based on the target process, the relevant settings for evidence collection are completed and data is collected to obtain the evidence collection results.

[0032] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0033] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0034] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) The mobile terminal forensics method based on OCR recognition and serial port simulation operation proposed in this invention is based on OCR recognition technology and large language model. It has high recognition accuracy for interface images and interactive actions, supports mobile terminals of various brands, has extremely high adaptability, and has strong recognition ability for different resolutions and different language systems. The control logic is flexible, and basic operation can be achieved in screen lock and network-free environments through serial port operation.

[0037] (2) The mobile terminal evidence collection method based on OCR recognition and serial port simulation operation proposed in this invention supports the automatic execution of the entire process and can complete the debugging and start process operation without manual intervention; it can also work with evidence collection software to automatically initiate mobile phone cloning requests and confirmation processes, minimize manual participation, and improve the stability and integrity of evidence collection.

[0038] (3) The mobile terminal evidence collection method based on OCR recognition and serial port simulation operation proposed in this invention can realize an evidence collection method that can automatically identify and control mobile phones without human intervention, significantly improving the level of evidence collection automation. It has good promotion value and market prospects in automatic evidence collection and law enforcement operations. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating a mobile terminal forensics method based on OCR recognition and serial port simulation, as an embodiment of this application.

[0041] Figure 2 This is a flowchart illustrating a mobile terminal forensics method based on OCR recognition and serial port simulation, as an embodiment of this application.

[0042] Figure 3 This is a schematic diagram illustrating the process of simulating the USB debugging operation in a mobile terminal forensics method based on OCR recognition and serial port simulation, which is an embodiment of this application.

[0043] Figure 4 This is a schematic diagram of the WiFi cloning forensics process of a mobile terminal forensics method based on OCR recognition and serial port simulation operation, which is an embodiment of this application.

[0044] Figure 5 This is a schematic diagram of the interface of the mobile terminal forensics method based on OCR recognition and serial port simulation operation, which is an embodiment of this application, applied to the software of an intelligent forensics keyboard system.

[0045] Figure 6 This is a schematic diagram of a mobile terminal forensics device based on OCR recognition and serial port simulation operation, which is an embodiment of this application.

[0046] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0048] Figure 1 This application illustrates an embodiment of a mobile terminal forensics method based on OCR recognition and serial port simulation, comprising the following steps:

[0049] S1. Construct a mapping model between the preset movement distance based on serial port simulation operation and the actual movement distance of the corresponding control point in the interface displayed on the mobile terminal screen.

[0050] In a specific embodiment, step S1 specifically includes:

[0051] The system controls the corresponding control point on the screen of the mobile terminal to move a preset distance along a specified direction by means of preset serial port simulated operation commands.

[0052] The interface images of the corresponding control point in the screen of the mobile terminal are obtained before and after the process of moving a preset distance in a specified direction. The optical flow analysis or image difference analysis is performed in combination with OCR recognition technology to detect the coordinate changes of the control point in the interface images before and after the process is executed.

[0053] The coordinate difference is identified by the coordinate change of the control point in the interface image before and after the process is executed, so as to calculate the actual movement distance corresponding to the interface displayed on the screen of the mobile terminal at the preset movement distance.

[0054] Repeat the above steps under multiple specified directions and preset movement distances to establish a mapping model between the preset movement distance based on serial port simulation operation and the actual movement distance of the corresponding control point in the interface displayed on the mobile terminal screen.

[0055] Specifically, the mobile terminal in the embodiments of this application takes a mobile phone as an example, but it can also be a tablet computer or other devices. In order to improve the serial port control accuracy and screen adaptation capability, the embodiments of this application introduce a mapping model to dynamically estimate the actual screen response of the serial port simulated sliding or mouse movement operation on the target mobile phone.

[0056] The mapping model is constructed through the following process:

[0057] 1) Send a fixed-length serial port simulation operation command of mouse sliding or clicking to the target mobile phone, so that the mouse pointer on the target mobile phone moves one unit step in a specified direction (such as vertical), and the one unit step is used as the preset movement distance, and the mouse pointer is used as the control point on the mobile phone screen interface.

[0058] 2) Use a camera to capture the current mobile phone screen interface image, and combine it with OCR recognition technology to perform optical flow analysis or image difference to detect the change in the position of the mouse pointer after it moves;

[0059] 3) Based on the coordinate difference of the mouse pointer identified in the previous and next images, calculate the actual displacement of the interface corresponding to the "unit step" at the current screen resolution.

[0060] 4) Repeated measurements were performed in multiple directions and at preset moving distances to establish a mapping model between the preset moving distance and the actual moving distance of the serial port simulated operation command of the target mobile phone.

[0061] By introducing this mapping model, the embodiments of this application can significantly improve the stability and adaptability of serial port analog control operations on different devices. This mapping model is mainly used in the following three scenarios:

[0062] In step S4, during the generation of serial port simulated operation commands, a mapping model is used to precisely determine the sliding distance and click position of the control point. For example, when swiping down to find the "About phone" screen in the settings interface, the sliding distance is unknown, but the relative position of the target text is known, i.e., the coordinates in the OCR recognition result. Here, a mapping model is needed to calculate how many preset movement distances of the serial port simulated operation command are required to move the screen interface by the actual distance. Otherwise, situations may occur such as sliding too short to find the target, sliding too long to overshoot the target, or execution failure due to differences in sliding response between different phones. Simultaneously, the OCR recognition result returns pixel coordinates, but the serial port simulated operation command can only send relative movement amounts. Therefore, the coordinates must be converted into serial port simulated operation commands through the mapping model, converting the coordinate difference of the OCR recognition result into the actual number of OTG mouse movement serial port simulated operation commands that need to be sent. Without this adaptation, the control cannot be clicked correctly.

[0063] During the execution of the serial port simulation operation command in step S4, this mapping model is used to keep the phone on the target interface. In the USB debugging or WiFi forensics process, it checks whether the phone is currently on the specified interface. If not, it performs appropriate swipes and clicks to return to the correct interface. At this time, it is also necessary to obtain the control positions based on the OCR recognition results and generate precise swipe and click commands according to the mapping model. Without the mapping model, the swipe distance will be inaccurate, and the button will not be found.

[0064] In the anomaly recovery process in step S4, when recognition fails or click offset occurs, remedial measures such as "swipe up one screen" or "return to the top of the current page" will be executed. All these operations rely on the mapping model to ensure that the correction actions are accurate and controllable.

[0065] S2, acquire the current interface image and perform OCR recognition to obtain the OCR recognition result, which includes the text and its corresponding coordinates.

[0066] For details, please refer to Figure 2 By continuously capturing high-definition image frames of the interface displayed on the target mobile phone screen using a camera device (such as a document camera), an interface image can be obtained. OCR recognition technology is then used to recognize text in the interface image, extracting the corresponding text and coordinates. The text and coordinates are then combined to generate a structured OCR recognition result. In one example, the OCR recognition technology can use the RapidOCR model, which features real-time processing, adaptation to different screen resolutions, and support for multilingual recognition. Other models or algorithms can be used in other embodiments.

[0067] S3 combines the text and prompts and inputs them into the large language model, outputting the reasoning results. The reasoning results include the control interface to which the current interface image belongs and its next action plan. Based on the reasoning results and OCR recognition results, the final control interface to which the current interface image belongs, the final interface state, the position of the target control, and its final next action plan are determined.

[0068] In a specific embodiment, the final control interface, final interface state, target control position, and final next action plan of the current interface image are determined based on the inference result and the OCR recognition result. Specifically, this includes:

[0069] The text in the OCR recognition result is compared with the preset interface keyword library to determine the control interface to which the current interface image of the OCR recognition belongs. It is then determined whether the control interface to which the current interface image of the OCR recognition belongs is consistent with the control interface to which the current interface image of the inference result belongs. If so, the control interface to which the current interface image belongs, the interface state, and the next action plan of the current interface image in the inference result are taken as the final control interface, the final interface state, and the final next action plan of the current interface image. The position of the target control is obtained based on the coordinates in the OCR recognition result. Otherwise, an abnormal recovery process is triggered. The abnormal recovery process includes repeating step S2 to re-perform OCR recognition on the current interface image, adjusting the sliding direction of the current step action plan, or returning to the previous interface and repeating steps S2-S3.

[0070] In a specific embodiment, the prompt words include the currently executed forensic task, a preset interface keyword library, interface differences between different mobile terminals, and a preset state flow; the control interface includes the main menu, settings, the interface corresponding to the about device or developer options; the next action plan includes clicking a button, swiping up, swiping down, or entering a password.

[0071] Specifically, the embodiments of this application are based on a Large Language Model (LLM). In addition to the text in the OCR recognition result, prompts are introduced. These prompts include the currently executed forensic task, a preset interface keyword library, specific interface differences between different mobile phone brands, and a preset state flow. The preset interface keyword library contains all keywords appearing in preset interfaces throughout the target flow required to complete the forensic function, and these keywords form a mapping relationship with the preset interfaces. An example of a preset state flow is as follows: USB debugging enable flow: "Settings → About phone → Tap the build number repeatedly → Return to Settings → Developer options → Enable USB debugging". This preset state flow serves as a prompt in the context of LLM inference, enabling the Large Language Model to determine which stage of the preset state flow it is in based on the current interface image and to infer subsequent operation steps. The text in the OCR recognition result is combined with the prompts to form the complete input content of the Large Language Model. LLM performs natural language parsing and semantic reasoning on the input content to determine the control interface (such as the main menu, settings, about phone, developer options, etc.) and interface state (such as whether the USB debugging switch is visible, whether it is necessary to continuously click the "build number", whether it is necessary to enter "developer options", etc.), and generates the next action plan (such as clicking a button, swiping up / down, entering a password, etc.). To improve the accuracy of the judgment, the embodiments of this application will also compare the OCR recognition result with a preset interface keyword library to help verify the inference result of LLM, locate the position of the target control, and decide whether to perform a swipe search. By comparing the text in the OCR recognition result with the preset interface keyword library, the control interface to which the current interface image recognized by OCR belongs can be determined. As a comparison result, if the comparison result is inconsistent with the control interface to which the current interface image belongs output by LLM, an abnormal recovery process is automatically triggered, such as re-performing OCR recognition, adjusting the swipe direction, or returning to the previous interface. While interfaces may differ across brands and systems, the embodiments of this application are based on a universal preset state flow template and can be adapted through a preset interface keyword library and LLM-enhanced parsing capabilities, eliminating the need to write separate flows for each brand of mobile phone. The large language model of the embodiments of this application supports the following functions: semantic recognition of interface states to address system and language differences; action planning for multi-step operations, which can automatically adjust control strategies in cases of recognition failure, interface changes, or different language systems, ensuring continuous and stable execution of the overall forensics process.

[0072] In a specific embodiment, step S4 further includes:

[0073] After executing the previous serial port simulation operation command to complete one of the target interactive actions in the target process or to enter the main screen, determine whether the final control interface to which the current interface image belongs is the interface or main screen interface to be entered after executing the previous serial port simulation operation command. If so, the generated current serial port simulation operation command is the serial port simulation operation command for controlling the next interactive action to complete one of the target interactive actions in the target process. Otherwise, the generated current serial port simulation operation command is the interface displayed on the screen of the mobile terminal by sliding and adjusting the screen, and the above judgment process is repeated until the final control interface to which the current interface image belongs is the interface to be entered after executing the previous serial port simulation operation command.

[0074] Specifically, to avoid errors in serial port simulation operation commands, the embodiments of this application, after the previous serial port simulation operation command is executed or after entering the main screen in the initial stage, need to determine whether the final control interface to which the current interface image belongs is the interface or main screen interface required after executing the previous serial port simulation operation command. If so, it means that the correct interface has been entered after executing the previous serial port simulation operation command; otherwise, it means that the wrong interface has been entered after executing the previous serial port simulation operation command. Therefore, it is necessary to slide and adjust the interface displayed on the mobile terminal screen to enter the correct interface.

[0075] S4 generates the current serial port simulation operation command based on the final control interface to which the current interface image belongs, the final interface state, the position of the target control, and the final next action plan. Based on the current serial port simulation operation command, the mapping model, and the OCR recognition result, it executes the corresponding interactive action on the final control interface to which the current interface image belongs to switch to the next interface and obtain the next interface image.

[0076] Specifically, based on the results of step S3, such as the final control interface to which the current interface image belongs, the final interface state, the position of the target control, and the final next action plan, serial port simulation operation commands (such as mouse movement, click, slide, and input) are automatically generated to realize operations such as mouse movement, click, and password input, simulating human interaction behavior. The position of the target control includes the target control and its corresponding operation coordinates. The location of the target control can be determined through OCR recognition results or swipe search (such as "Settings", "About phone", "Version number", etc.). Combined with the final page state and the final next action plan (such as needing to swipe down to search for a control), a serial port simulation operation command is formed. The operation coordinates contained in the serial port simulation operation command are obtained by transforming the coordinates in the mapping model and the OCR recognition results. The action instructions contained in the serial port simulation operation command are determined based on the final interface state and the final next action plan. Specifically, the best interactive action can be determined by combining the final interface state and the final next action plan, and the corresponding action instructions can be generated. If the final interface state is whether to click "Version number" continuously, and the final next action plan is to click, then the determined interactive action is to click "Version number" N times continuously. The serial port simulation operation command is sent to the target mobile phone via the OTG serial port interface, driving the target mobile phone to perform relevant interactive actions, such as enabling USB debugging. The main process is as follows: Determine whether the current screen is on the home screen based on the "page module" and simulate clicking the "Settings" icon; swipe to enter the "About phone" interface; simulate clicking 7 times consecutively in the "Build number" area to trigger "Developer mode" to be enabled; return to the main menu and swipe to the "Settings" or "Developer options" interface; simulate clicking and turning on the "USB debugging" switch to complete the permission configuration. This process consists of a complete flow of interface image recognition, interface judgment, action reasoning, and serial port control: interface judgment generates the next action plan → execute the interactive action → the OCR recognition result of the next interface image is input again into step S3 → update the next action plan → continue to execute the interactive action, ensuring that the operation process can be reliably completed under different brands and system versions.

[0077] S5, take the next interface image as the current interface image and repeat steps S2-S4 until the target process is completed by executing interactive actions in sequence. Based on the target process, complete the evidence collection-related settings and collect data to obtain the evidence collection results.

[0078] In a specific embodiment, the target process includes enabling USB debugging operation, WiFi cloning forensics, or direct connection forensics. The operation coordinates contained in the serial port simulation operation instructions are obtained by coordinate transformation based on the mapping model and the OCR recognition results. The action instructions contained in the serial port simulation operation instructions are determined based on the final interface state and the final next action plan.

[0079] Specifically, the closed-loop operation process formed by steps S2-S4 in the embodiments of this application is to complete one of the interactive actions in the target process. However, completing the entire target process requires completing multiple interactive actions sequentially; therefore, steps S2-S4 need to be repeated multiple times to ultimately complete the entire target process. The target process corresponding to the forensics process in the embodiments of this application includes enabling USB debugging, WiFi cloning forensics, or direct connection forensics. After completing the target process, forensics-related settings can be completed and data can be collected to obtain the forensics results.

[0080] refer to Figure 3 The following explanation uses the target process of enabling USB debugging as an example, combined with the above steps S2-S4.

[0081] First, determine whether the interface image captured after entering the main screen is the main screen interface. If so, generate a serial port simulation operation command to simulate clicking the settings icon and complete the interactive action of simulating clicking the settings icon.

[0082] Therefore, the acquired interface image is the interface image obtained after completing the interactive action of simulating clicking the settings icon; the OCR recognition result corresponding to the interface image is obtained using step S2; the OCR recognition result corresponding to the interface image is input into step S3 to identify the final control interface, final interface state, target control position and final next action plan of the interface image, and further generate the corresponding serial port simulation operation command using step S4.

[0083] Determine whether the final control interface to which the interface image belongs is the settings interface. If so, the corresponding serial port simulation operation instruction generated in step S4 is the serial port simulation operation instruction to enter the About Device interface, and the About Device interface is used as the next interface. Otherwise, the corresponding serial port simulation operation instruction generated is to slide and adjust the interface displayed on the screen of the mobile terminal and repeat the above judgment process.

[0084] Using the next interface as the current interface, repeat steps S2-S4 to execute the serial port simulation operation command to enter the "About Device" interface and complete the interactive action of entering the "About Device" interface. Determine whether the final control interface to which the interface image acquired after completing the interactive action of entering the "About Device" interface belongs is the "About Phone" interface. If so, the corresponding serial port simulation operation command generated is the serial port simulation operation command to complete the action corresponding to triggering the opening of developer mode. First, when executing the serial port simulation operation command corresponding to triggering the opening of developer mode, simulate clicking the version number several times, then return to the settings interface, then enter the developer selection, click to enable the USB debugging switch, confirm the pop-up window and complete the configuration. After the configuration is completed, the opening of developer mode will be triggered, and then you can proceed to the next process.

[0085] refer to Figure 4 The target process of the embodiments of this application also includes a WiFi forensics process, and it can be deeply integrated with existing forensics software. During the WiFi forensics process, the mobile phone can automatically simulate entering "phone cloning", select and connect to the forensics hotspot; simulate clicking the "phone cloning", "start migration" and other buttons on the mobile phone; automatically confirm the cloning request, keep the mobile phone on the forensics page, and wait for the data to be pulled.

[0086] Its core operating mechanism is a continuous loop: real-time interface image → OCR recognition in step S2 → interface recognition in step S3 → interface judgment → action planning → execution of simulated serial port operation commands → acquisition of the next interface image. Through this closed loop, the "phone cloning" process is automated.

[0087] The automatic initiation of the "phone cloning" process is implemented as follows:

[0088] The camera continuously captures images of the phone screen, uses OCR technology to obtain corresponding OCR recognition results, and uses a Large Language Model (LLM) to identify whether the interface image is the cloning entry interface. The input of the LLM includes: the text of the OCR recognition result, a preset interface keyword library (such as "phone cloning", "swap cloning", etc.), and contextual prompts. The output of the LLM includes: the control interface to which the current interface image belongs (such as "desktop main interface" or "settings interface"), the interface status (whether the cloning entry has been found), and the next action plan (such as "click the phone cloning icon" or "need to swipe up to search"). Then, combined with the OCR recognition results, serial port simulated operation commands are generated and corresponding interactive actions are executed. For example, by recognizing the entry, one can enter the settings, then the system and updates, select the swap cloning button, and complete the phone cloning function.

[0089] The automatic selection and connection to evidence-gathering hotspots is implemented as follows:

[0090] When LLM determines that the control interface to which the current interface image belongs is "WiFi List" or "Hotspot Selection Page", it uses OCR recognition technology to detect the hotspot name, determines which hotspot is for evidence collection, for example: "Forensic_WiFi_xxx", and executes the next action "click the hotspot name". When the password box pops up: use the OCR recognition result to locate the input box → send serial keyboard input → click "connect".

[0091] The following method is used to automatically click buttons such as "Phone Clone" and "Start Migration":

[0092] Using OCR recognition technology, buttons in the current interface image are identified, such as "Phone Cloning" and "Start Migration". The LLM determines which stage of the cloning process is currently in based on the text in the OCR recognition result and a preset interface keyword library, such as "Waiting to Select Mode", "Waiting to Start Migration", or "Preparing to Connect". The LLM output includes: the current interface state and the next action plan (e.g., "Click the Start Migration button"). The coordinates of the click position are calculated based on the coordinates of the text in the OCR recognition result and the following interactive actions are executed: mouse movement, click, and confirmation of successful action execution. The process continues to loop until the interface enters the migration preparation state and waits for evidence collection to be completed.

[0093] The entire process achieves "unmanned operation" from manual WiFi connection to the clone, ensuring safe and efficient data fixation of mobile terminals even without human intervention.

[0094] refer to Figure 5 The mobile terminal forensics method based on OCR recognition and serial port simulation operation proposed in the embodiments of this application has been verified in the intelligent forensics keyboard system software and can be applied to: automatic USB debugging on mobile phones, WiFi intelligent forensics, and direct connection intelligent forensics. It has good promotional value and market prospects in automatic forensics and law enforcement operations.

[0095] Further reference Figure 6 As an implementation of the methods shown in the above figures, this application provides an embodiment of a mobile terminal forensics device based on OCR recognition and serial port simulation operation. This device embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0096] This application provides a mobile terminal forensics device based on OCR recognition and serial port simulation operation, including:

[0097] Model building module 1 is configured to build a mapping model between a preset moving distance based on serial port simulation operation and the actual moving distance of the corresponding control point in the interface displayed on the screen of the mobile terminal.

[0098] OCR recognition module 2 is configured to acquire the current interface image and perform OCR recognition to obtain OCR recognition results, which include text and its corresponding coordinates.

[0099] The reasoning module 3 is configured to combine text and prompt words and input them into the large language model, and output reasoning results. The reasoning results include the control interface to which the current interface image belongs and its next action plan. Based on the reasoning results and OCR recognition results, the final control interface to which the current interface image belongs, the final interface state, the position of the target control and its final next action plan are determined.

[0100] Execution module 4 is configured to generate the current serial port simulation operation command based on the final control interface to which the current interface image belongs, the final interface state, the position of the target control, and the final next action plan. Based on the current serial port simulation operation command, the mapping model, and the OCR recognition result, it executes the corresponding interactive action on the final control interface to which the current interface image belongs to switch to the next interface and obtain the next interface image.

[0101] The repeat module 5 is configured to use the next interface image as the current interface image and repeat the OCR recognition module 2 to the execution module 4 until the target process is completed by sequentially executing interactive actions. Based on the target process, the relevant settings for evidence collection are completed and data is collected to obtain the evidence collection results.

[0102] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 7 As shown, the electronic device of this embodiment includes a processor 701 and a memory 702; wherein the memory 702 is used to store computer execution instructions; and the processor 701 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0103] Alternatively, the memory 702 can be either standalone or integrated with the processor 701.

[0104] When the memory 702 is set up independently, the electronic device also includes a bus 703 for connecting the memory 702 and the processor 701.

[0105] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by processor 701, implement the above method.

[0106] This invention also provides a computer program product, including a computer program that, when executed by a processor 701, implements the above-described method.

[0107] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0108] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0109] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0110] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 701 to execute some steps of the methods of the various embodiments of this application.

[0111] It should be understood that the processor 701 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 701 can be any conventional processor 701. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 701 executing the steps, or as a combination of hardware and software modules within the processor 701 executing the steps.

[0112] The memory 702 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0113] Bus 703 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 703 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 703 in the accompanying drawings of this application is not limited to only one bus 703 or one type of bus 703.

[0114] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0115] An exemplary storage medium is coupled to a processor 701, enabling the processor 701 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor 701. The processor 701 and the storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor 701 and the storage medium can exist as discrete components in an electronic device or a host device.

[0116] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mobile terminal forensics method based on OCR recognition and serial port simulation operation, characterized in that, Includes the following steps: S1, Construct a mapping model between the preset moving distance based on serial port simulation operation and the actual moving distance of the corresponding control point in the interface displayed on the screen of the mobile terminal; S2, acquire the current interface image and perform OCR recognition to obtain the OCR recognition result, which includes the text and its corresponding coordinates; S3, combine the text with the prompt words and input them into the large language model, output the reasoning result, the reasoning result includes the control interface to which the current interface image belongs, the interface state and its next action plan, and determine the final control interface, the final interface state, the target control position and its final next action plan to which the current interface image belongs based on the reasoning result and the OCR recognition result. S4. Based on the final control interface to which the current interface image belongs, the final interface state, the position of the target control, and the final next action plan, generate the current serial port simulation operation command. Based on the current serial port simulation operation command, the mapping model, and the OCR recognition result, execute the corresponding interactive action on the final control interface to which the current interface image belongs to switch to the next interface and obtain the next interface image. S5, take the next interface image as the current interface image and repeat steps S2-S4 until the target process is completed by executing the interactive actions in sequence. Based on the target process, complete the evidence collection-related settings and collect data to obtain the evidence collection results.

2. The mobile terminal forensics method based on OCR recognition and serial port emulation operation according to claim 1, characterized in that, Step S1 specifically includes: The system controls the corresponding control point on the screen of the mobile terminal to move a preset distance along a specified direction by means of preset serial port simulated operation commands. The interface images before and after the execution of the process of moving a corresponding control point along a specified direction and a preset distance in the interface displayed on the screen of the mobile terminal are obtained. Optical flow analysis or image difference analysis is performed in combination with OCR recognition technology to detect the coordinate changes of the control point in the interface images before and after the process is executed. The coordinate difference is identified based on the coordinate changes of the control point in the interface image before and after the process is executed, so as to calculate the actual movement distance corresponding to the preset movement distance in the interface displayed on the screen of the mobile terminal. Repeat the above steps under multiple specified directions and preset movement distances to establish a mapping model between the preset movement distance based on serial port simulation operation and the actual movement distance of the corresponding control point in the interface displayed on the mobile terminal screen. 3.The mobile terminal forensics method based on OCR recognition and serial port emulation operation of claim 1, wherein, The prompts include the currently executing forensic task, a preset interface keyword library, interface differences between different mobile terminals, and preset state flows; the control interface includes the main menu, settings, and interfaces corresponding to the about device or developer options; the next action plan includes clicking a button, swiping up, swiping down, or entering a password.

4. The mobile terminal forensics method based on OCR recognition and serial port emulation operation according to claim 3, characterized in that, Based on the reasoning results and the OCR recognition results, the final control interface, final interface state, target control position, and final next action plan of the current interface image are determined, specifically including: The text in the OCR recognition result is compared with the preset interface keyword library to determine the control interface to which the current interface image of the OCR recognition belongs. It is then determined whether the control interface to which the current interface image of the OCR recognition belongs is consistent with the control interface to which the current interface image in the inference result belongs. If so, the control interface to which the current interface image belongs, the interface state, and the next action plan in the inference result are taken as the final control interface, the final interface state, and the final next action plan of the current interface image. The position of the target control is obtained based on the coordinates in the OCR recognition result. Otherwise, an abnormal recovery process is triggered. The abnormal recovery process includes repeating step S2 to re-perform OCR recognition on the current interface image, adjusting the sliding direction of the current step action plan, or returning to the previous interface and repeating steps S2-S3.

5. The mobile terminal forensics method based on OCR recognition and serial port emulation operation according to claim 1, characterized in that, The target process includes enabling USB debugging operation process, WiFi cloning forensics process or direct connection forensics process. The operation coordinates contained in the serial port simulation operation instructions are obtained by coordinate transformation based on the mapping model and the OCR recognition result. The action instructions contained in the serial port simulation operation instructions are determined based on the final interface state and the final next action plan.

6. The mobile terminal forensics method based on OCR recognition and serial port simulation operation according to claim 1, characterized in that, Step S4 further includes: Determine whether the final control interface to which the current interface image belongs is the target interface. The target interface includes the interface or main screen interface to be entered after executing the previous serial port simulation operation command. If so, the generated current serial port simulation operation command is a serial port simulation operation command that controls the execution of the corresponding interactive action on the final control interface to which the current interface image belongs. Otherwise, the generated current serial port simulation operation command is to slide and adjust the interface displayed on the screen of the mobile terminal and repeat the above determination process until the final control interface to which the current interface image belongs is the target interface.

7. A mobile terminal forensics device based on OCR recognition and serial port simulation operation, characterized in that, include: The model building module is configured to build a mapping model between a preset moving distance based on serial port simulation operation and the actual moving distance of the corresponding control point in the interface displayed on the screen of the mobile terminal. The OCR recognition module is configured to acquire the current interface image and perform OCR recognition to obtain the OCR recognition result, which includes text and its corresponding coordinates. The reasoning module is configured to combine the text with the prompt words and input them into the large language model, and output the reasoning result. The reasoning result includes the control interface to which the current interface image belongs and its next action plan. Based on the reasoning result and the OCR recognition result, the final control interface to which the current interface image belongs, the final interface state, the position of the target control and its final next action plan are determined. The execution module is configured to generate a current serial port simulation operation instruction based on the final control interface to which the current interface image belongs, the final interface state, the position of the target control, and the final next action plan. Based on the current serial port simulation operation instruction, the mapping model, and the OCR recognition result, the module executes the corresponding interactive action on the final control interface to which the current interface image belongs to switch to the next interface and obtain the next interface image. The repeat module is configured to use the next interface image as the current interface image and repeat the OCR recognition module to the execution module until the target process is completed by sequentially executing the interactive actions. Based on the target process, the evidence collection-related settings are completed and data is collected to obtain the evidence collection result.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images