Data processing method and device, program product and electronic equipment

By screening screen image data and associated application running status data on the terminal device, and generating and storing running status records, the problem of low integrity of local storage application data is solved, the richness and security of data are improved, and the cloud is reduced Service dependency.

CN118964087APending Publication Date: 2024-11-15NETEASE (HANGZHOU) NETWORK CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410977992.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Due to permission restrictions, the running data of local storage applications has low data integrity and cannot meet the user's rich data usage needs.

Method used

By periodically seizing screen image data and binding it to the running state data of the associated application, generating running state records, and storing them in the terminal device, avoiding permission restrictions and improving data integrity and security.

Benefits of technology

It improves the richness and diversity of local storage applications' running data, improves user experience, reduces dependence on cloud services, reduces data transmission costs, and improves access efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964087A_ABST
    Figure CN118964087A_ABST
Patent Text Reader

Abstract

The invention provides a data processing method and device, a program product and electronic equipment, and relates to the technical field of computers. The data processing method is applied to the terminal equipment comprising a screen, and comprises the following steps: acquiring an interception frequency of screen image data; performing screen snapshot on the terminal equipment according to the interception frequency to obtain the screen image data; acquiring an associated application of the screen image data, and acquiring running state data of the associated application; and binding the screen image data with the running state data of the associated application, generating a running state record of the associated application, and storing the running state record in the terminal equipment. According to the method and the device, the problem of permission limitation in the process of locally acquiring the running state data of the application is avoided to a certain extent, and the integrity of the running data of the application acquired locally is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, a data processing device, a computer program product, and an electronic device. Background Art

[0002] In the related art, the running data of applications stored locally is often limited due to the permission restrictions of the applications, resulting in low integrity of the running data of applications stored locally. Therefore, a data processing method is urgently needed to improve the problem of low integrity of the running data of applications stored locally. Summary of the invention

[0003] The present disclosure provides a data processing method, a data processing device, a computer program product and an electronic device, thereby improving the problem of low integrity of running data of locally stored applications at least to a certain extent.

[0004] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0005] According to a first aspect of the present disclosure, there is provided a data processing method, which is applied to a terminal device including a screen, and the method comprises: obtaining a capture frequency of screen image data; taking a screenshot of the terminal device according to the capture frequency to obtain the screen image data; obtaining an associated application of the screen image data, and obtaining running status data of the associated application; binding the screen image data and the running status data of the associated application to generate a running status record of the associated application, and storing the running status record in the terminal device.

[0006] According to a second aspect of the present disclosure, there is provided a data processing apparatus, which is applied to a terminal device including a screen, and the apparatus includes: an interception frequency acquisition module, which is configured to acquire the interception frequency of screen image data; a screen image data acquisition module, which is configured to take a screen snapshot of the terminal device according to the interception frequency to acquire the screen image data; an operation status data acquisition module, which is configured to acquire an associated application of the screen image data, and acquire the operation status data of the associated application; an operation status record storage module, which is configured to bind the screen image data and the operation status data of the associated application, generate an operation status record of the associated application, and store the operation status record in the terminal device.

[0007] According to a third aspect of the present disclosure, a computer program is provided, which, when executed by a processor, implements the data processing method of the first aspect and possible implementation manners thereof.

[0008] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor, wherein the processor is configured to execute the data processing method of the first aspect and possible implementation thereof by executing the executable instructions.

[0009] The technical solution disclosed in this disclosure has the following beneficial effects:

[0010] On the one hand, a solution is provided for periodically capturing screen image data and binding the screen image data with the running status data of the associated application to obtain the running status record of the associated application, which avoids the problem of permission restriction in the process of obtaining the running status data of the application locally to a certain extent, improves the completeness of the running data of the application obtained locally, effectively improves the richness and diversity of the running data of the application stored locally, meets the user's demand for the use of the running data of the application, and improves the user experience. On the other hand, the running status record of the associated application is stored in the terminal device, which improves the problem of low security in the process of storing the running data to a certain extent compared with the method of storing the running data of the application in the cloud in the related technology, and at the same time, reduces the data transmission cost brought by storing the running data of the application in the cloud, facilitates the user to quickly access the running data of the application, and improves the efficiency of the user to access the running data of the application. On the other hand, storing the running status record of the application in the terminal device can reduce the user's dependence on the cloud service in the process of accessing the running data, and further improves the flexibility of the user to access the running data of the application.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0013] Figure 1 The system operation architecture of this exemplary embodiment is shown;

[0014] Figure 2 A flow chart showing a data processing method in this exemplary embodiment;

[0015] Figure 3 A flowchart showing a method for obtaining running status data of an associated application in this exemplary embodiment is shown;

[0016] Figure 4 A flowchart showing a method for acquiring and storing operation status records in this exemplary embodiment is shown;

[0017] Figure 5 A flowchart showing a method for data privacy protection processing in this exemplary embodiment;

[0018] Figure 6 A flowchart showing a method for restoring the running state of an associated application in this exemplary embodiment is shown;

[0019] Figure 7 A flowchart showing a method for obtaining a target operating status record in this exemplary embodiment is shown;

[0020] Figure 8 A flow chart showing a method for obtaining an operation state vector in this exemplary embodiment is shown;

[0021] Fig. 9 A flow chart showing a method for accelerating a process of generating a running state vector in this exemplary embodiment;

[0022] Fig.10 A flow chart showing a method for matching a target operating state vector in a vector database in this exemplary embodiment is shown;

[0023] Fig.11 A flowchart showing a method for restoring the running state of an application in this exemplary embodiment is shown;

[0024] Fig.12 A flowchart showing a method for backtracking an application according to a stored running state in this exemplary embodiment is shown;

[0025] Fig.13 A schematic diagram showing the structure of a data processing device in this exemplary embodiment is shown;

[0026] Fig.14 A schematic structural diagram of an electronic device in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0028] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0029] In the related art, the staff usually calls the Application Programming Interface (API) provided by the application to obtain the application's running data and save the application's running data locally. However, this method often limits the application's permission, resulting in a low integrity of the application's running data that can be stored locally, making it difficult to meet the user's demand for rich application running data.

[0030] In view of one or more of the above problems, the exemplary embodiment of the present disclosure first provides a data processing method. Figure 1 The system architecture of the operating environment of this exemplary embodiment will be described.

[0031] refer to Figure 1 As shown, the system architecture 100 may include a terminal device 110. The terminal device 110 may be an electronic device with a screen, such as a smart phone, a tablet computer, or a laptop computer, and the terminal device 110 may be used to acquire screen image data.

[0032] The data processing method in this exemplary embodiment may be executed by the terminal device 110. For example, in a scenario where a user browses a web page through a browser in the terminal device 110, the associated application may include the browser, and the terminal device 110 may execute the data processing method to take a screenshot of the terminal device according to the capture frequency to save screen image data including the browser, and may also obtain the browser's operating status data, so as to obtain the browser's operating status record through the screen image data and the browser's operating status data, and store the operating status record in a designated location of the terminal device 110.

[0033] It can be seen from the above that the data processing method in this exemplary embodiment can be executed by the above-mentioned terminal device 110.

[0034] Combine the following Figure 2 The data processing method is described. Figure 2 An exemplary process of the data processing method is shown, including the following steps S210 to S240:

[0035] Step S210, obtaining the capture frequency of the screen image data;

[0036] Step S220, taking a screen snapshot of the terminal device according to the capture frequency to obtain screen image data;

[0037] Step S230, obtaining an application associated with the screen image data, and obtaining running status data of the associated application;

[0038] Step S240 , binding the screen image data and the running status data of the associated application, generating a running status record of the associated application, and storing the running status record in the terminal device.

[0039] Based on the above method, on the one hand, a solution is provided for periodically capturing screen image data and binding the screen image data with the running status data of the associated application to obtain the running status record of the associated application, which avoids the problem of permission restriction in the process of obtaining the running status data of the application locally to a certain extent, improves the completeness of the running data of the application obtained locally, effectively improves the richness and diversity of the running data of the application stored locally, meets the user's demand for the use of the running data of the application, and improves the user experience. On the other hand, the running status record of the associated application is stored in the terminal device, which improves the problem of low security in the process of storing the running data to a certain extent compared with the method of storing the running data of the application in the cloud in the related technology, and at the same time, reduces the data transmission cost brought by storing the running data of the application in the cloud, facilitates the user to quickly access the running data of the application, and improves the efficiency of the user to access the running data of the application. On the other hand, storing the running status record of the application in the terminal device can reduce the user's dependence on the cloud service in the process of accessing the running data, and further improve the flexibility of the user to access the running data of the application.

[0040] Below Figure 2 Each step in the procedure is described in detail.

[0041] refer to Figure 2 , in step S210, the capture frequency of the screen image data is obtained.

[0042] The screen image data includes an image obtained by capturing all or part of the current display content on the screen. The capture frequency can be used to characterize the acquisition period of the screen image data. For example, the capture frequency can be the default capture frequency of the system. If the default value of the capture frequency is 5 seconds, it means that a screenshot operation is performed every 5 seconds to obtain the screen image data. The user can also adjust the capture frequency through the setting operation.

[0043] In one implementation, the above-mentioned acquisition of the capture frequency of the screen image data may include the following steps:

[0044] In response to a setting operation for the interception frequency, the interception frequency of the screen image data is determined.

[0045] For example, the system default interception frequency is 5 seconds. The user can adjust the interception frequency within a preset range based on his or her own needs by setting the interception frequency. For example, the preset range of the interception frequency can generally be 1 second to 60 seconds, that is, the user can set the interception frequency to any value between 1 second and 60 seconds.

[0046] This exemplary embodiment provides a solution for users to customize the interception frequency through setting operations, so that users can customize and adjust the interception frequency, thereby increasing the user's freedom in the process of acquiring screen image data and improving the user experience.

[0047] Continue to refer Figure 2 In step S220, a screen snapshot of the terminal device is taken according to the capture frequency to obtain screen image data.

[0048] For example, the capture frequency can be 5 seconds, then the operating system API, such as PrintWindow of the Windows operating system or CGWindowListCreateImage of the macOS operating system, can be used to take a screenshot of the terminal device every 5 seconds to obtain screen image data; after obtaining the screen image data, the screen image data can be named according to the timestamp of the screenshot data, such as snapshot_YYYYMMDD_HHMMSS.png (YYYYMMDD_HHMMSS is the timestamp when the screenshot data is captured); finally, the named screen image data can be stored in a file directory specified by the user to facilitate the user's subsequent processing of the screen image data.

[0049] In one implementation, in response to detecting a modification operation on the capture frequency, after the current screen image data is acquired, a screen snapshot of the terminal device is taken according to the modified capture frequency.

[0050] That is to say, after the user modifies the capture frequency, if the current moment is in the preparation period for obtaining screen image data, a screenshot will be taken according to the capture frequency modified by the user the next time the screen image data is obtained; if the screen image data is being obtained at the current moment, the screen image acquisition will be completed after this time, and a screenshot will be taken according to the capture frequency modified by the user the next time the screen image data is obtained.

[0051] For example, a user can input or select an interception frequency through an interception frequency setting interface, and the value range of the interception frequency can be 1 second to 60 seconds; after the user modifies the interception frequency, the modification of the interception frequency can be monitored by monitoring events in the interception frequency setting interface or periodically checking the change timestamp of the configuration file, and reset the timer parameters according to the user-modified interception frequency to reconfigure the scheduled task, thereby applying the modified interception frequency to obtain screen image data.

[0052] By monitoring the setting operation of the interception frequency, it can respond quickly when the user triggers the modification of the interception frequency, take a screenshot with the modified interception frequency, and the whole process can be achieved without restarting the screen image data acquisition process, effectively improving the efficiency of screen image data acquisition, thereby improving the user experience.

[0053] It should be noted that in the process of obtaining screen image data through screenshots, the display area of ​​the screen containing digital rights management (DRM) protected content will not be captured. For example, when taking a screenshot, the Windows operating system API, such as the DRMIsContentProtected function, can be used to detect whether the display content on the screen contains DRM protected content. If it is confirmed that the display content on the screen contains DRM protected content, the screenshot is paused, or in this screenshot, only the non-DRM protected content on the screen is captured; thereby ensuring that the DRM protected content will not be saved.

[0054] Continue to refer Figure 2 In step S230, the associated application of the screen image data is obtained, and the running status data of the associated application is obtained.

[0055] The above-mentioned associated application refers to the application running in the system at the moment of capturing the screen image. The above-mentioned associated application of obtaining screen image data may include: while obtaining the screen image data, determining the application in running state in the terminal device as the associated application. Exemplarily, while capturing the screen image data, the running status information of the application currently running in the system can be obtained through the APIs of the Windows operating system: EnumProcesses and EnumProcessModules, or the API of the macOS operating system: NSWorkspace.

[0056] The above-mentioned running status data may include user identification information of the application, such as the name of the application, serial number, etc., and user usage data for the application, such as the user's input content in the browser and the scrolling position of the browser, etc.

[0057] In one embodiment, the above-mentioned acquisition of the running status data of the associated application refers to Figure 3 As shown, steps S310 to S320 may be included:

[0058] Step S310, obtaining the running status information and identification information of the associated application.

[0059] The above-mentioned running status information may include the running status and usage information of the associated application. For example, the running status information may include the startup time of the application, the tabs opened by the browser, etc. For the associated application that supports the system API call, the running status information of the application may be obtained through the system API; for the associated application that does not support the system API call, its running status information may be obtained through the screen image data. In one embodiment, the above-mentioned obtaining of the running status information of the associated application may include:

[0060] Extracting usage data of associated applications from screen image data;

[0061] The usage data of the associated application is stored in a specified format to obtain the running status information.

[0062] The specified format may be a data format for storing usage data, for example, the specified format may include a json data format.

[0063] For example, the optical character recognition (OCR) technology can be used to extract text from screen image data, and the extracted text information, such as the content input by the user in the browser text box, the text written in the document, etc., can be filtered to obtain the usage data of the associated application; the usage data is then stored as a code segment in the json data format to obtain the running status information. For example, the running status information that can be obtained is as follows:

[0064]

[0065] The above OCR technology can convert characters and words in images into editable text by recognizing them. Commonly used OCR algorithms may include: Tesseract OCR, EasyOCR and Calamari OCR.

[0066] Among them, Tesseract is an open source OCR engine that supports text recognition in multiple languages. When using Tesseract, you need to first install Tesseract and pytesseract libraries, and then preprocess the read screen image data, such as grayscale and binarization, to improve the accuracy of character recognition, and then call the API of the Tesseract library, such as pytesseract.image_to_string(image) for character recognition, where image can be the incoming screen image data.

[0067] EasyOCR is another open source OCR engine that supports text recognition in multiple languages ​​and is mostly used to process complex characters and non-Latin characters. When using EasyOCR, you need to first install the EasyOCR library, then call the EasyOCR API to extract text from the read screen image data.

[0068] Calamari OCR is an open source OCR tool based on deep learning. It has high accuracy and flexibility and is mostly used for handwritten and printed text recognition. When using Calamari OCR, you need to first install the Calamari library, then call its API to extract text from the read screen image data.

[0069] The identification information of the associated application can be obtained when the associated application is obtained through the API of the operating system. After obtaining the identification information and running status information of the associated application, continue to refer to Figure 3 In step S320, the identification information of the associated application is bound to the running status information to obtain the running status data of the associated application.

[0070] Exemplarily, the identification information and the running status information may be written into the same json code block in a specified format to obtain the running status data of the associated application. The running status data of the associated application chrome.exe is as follows:

[0071]

[0072]

[0073] based on Figure 3 The method extracts text from screen image data to obtain running status information, and binds the identification information and running status information of the associated application after obtaining them to obtain the running status data of the associated application. This can effectively improve the completeness of the running data of the associated application and avoid the permission restriction problem existing in the process of locally obtaining the running status data of the application to a certain extent.

[0074] Continue to refer Figure 2 In step S240, the screen image data and the running status data of the associated application are bound to generate a running status record of the associated application, and the running status record is stored in the terminal device.

[0075] That is, after the screen image data and the running status data are obtained, the screen image data and the running status data can be saved together to generate a running status record of the associated application, and the running status record is stored in a user-specified storage location of the terminal device.

[0076] The above-mentioned running status record refers to a code block that contains both the storage information of the screen image data and the running status data of the associated application, and the storage information may include the storage path of the screen image data. It should be noted that the same screen image data may correspond to multiple associated applications, that is, the running status records of multiple associated applications may store the same screen image data.

[0077] In one implementation, the above-mentioned binding of the screen image data and the running status data of the associated application to generate the running status record of the associated application may include:

[0078] The storage information of the screen image data is written into the running status data of the associated application according to a specified format, so as to bind the screen image data with the running status data and obtain the running status record of the associated application.

[0079] For example, the running status record corresponding to the associated application named custom_app.exe is as follows:

[0080]

[0081] By writing the storage information of the screen image data into the running status data in a specified format to obtain the running status record of the associated application, it is helpful to quickly retrieve the corresponding screen image data through the identification information of the application.

[0082] In one embodiment, reference Figure 4 As shown, the running status of the application can be stored locally according to steps S401 to S409:

[0083] Step S401, obtaining the capture frequency of screen image data according to the setting operation of the user;

[0084] Step S402, based on the user's selection operation, determining a designated storage location for the running status record;

[0085] Step S403, taking a screenshot of the terminal device according to the capture frequency, and saving the acquired screen image data in a designated storage location; at the same time, using the operating system API to query the list of applications currently running, to obtain the associated applications of the screen image data;

[0086] Step S404, traverse all associated applications to confirm whether the associated applications support API calls of the operating system. If yes, jump to step S405; otherwise, jump to step S406;

[0087] Step S405, using the operating system API to obtain the identification information and running status information of the associated application;

[0088] Step S406, using the operating system API to obtain identification information of the associated application, and extracting text from the screen image data using OCR technology, filtering the extracted text content to obtain usage data, and generating running status information based on the usage data;

[0089] Step S407, storing the identification information and running status information of the associated application in JSON data format to obtain running status data;

[0090] Step S408, writing the storage path of the screen image data into the running status data of the associated application in json data format to generate a running status record of the associated application;

[0091] Step S409: storing the running status record of the associated application in the designated storage location in step S402 according to the same name as the screen image data.

[0092] based on Figure 4 The method can efficiently process and record screen image data, and record the running status of the application. For applications that cannot directly obtain the running status information through the API, obtaining the running status information of the application through the screen image data can effectively improve the integrity of the running data of the locally stored application and further improve the efficiency of obtaining the running data of the application.

[0093] Since the application's running status records often store the user's private data, after obtaining the application running data with high integrity through the screen image data, it is also necessary to protect the data privacy of the running status records. The process of protecting the privacy of the running status records is explained below.

[0094] First, the acquired running status record needs to be encrypted and stored. In one implementation, the above method may further include the following steps:

[0095] The operation status record in the terminal device is encrypted, and the encrypted operation status record is stored in the terminal device.

[0096] For example, the above-mentioned operation status record includes screen image data and operation status data. For the screen image data, after obtaining the screen image data, a random key can be generated for each screen image data to encrypt the screen image data, such as using the built-in random number generator of the operating system (Windows's CryptGenRandom or Linux's / dev / urandom) to generate a unique and random AES-256 key for the screen image data, and after generating the AES-256 key, use a public key encryption algorithm (such as RSA encryption algorithm) to encrypt it, and store the screen image data with the AES-256 key together with the encrypted AES-256 key to achieve encryption of the screen image data. If the user needs to access the screen image data, the encrypted screen image data needs to be decrypted before the screen image data can be viewed. In the process of decrypting the screen image data, the user needs to provide a private key corresponding to the above-mentioned public key to decrypt the AES key, and then use the decrypted AES-256 key and the initialization vector generated during the encryption process to decrypt the screen image data, so as to access the screen image data.

[0097] At the same time, the drive storing the screen image data can also be encrypted through the BitLocker function of the Windows operating system. BitLocker uses the Trusted Platform Module (TPM) for key management to ensure the security of the encryption key. Users can manage BitLocker encryption through the control panel or command line tools (such as manage-bde). For example, after obtaining the screen image data, the encrypted screen image data is stored in a user-specified drive, which is a drive that has enabled BitLocker encryption, thereby realizing encryption of the screen image data; when the user needs to access the screen image data in the subsequent process, BitLocker can automatically decrypt to provide the encrypted screen image data to the user, and the user can further decrypt the screen image data to view the screen image data.

[0098] In addition, the operating status data and screen image data can be encrypted through the encryption function of the operating system, such as the built-in device encryption function of Windows 11, so that users can view and manage the encryption status of the operating status records in many devices.

[0099] By encrypting the operation status records, users' concerns about data privacy can be taken into account, and the privacy and security of screen image data and operation status data can be improved, thereby improving the user experience.

[0100] Secondly, by setting data access permissions for users and applications, the privacy of the application's operating status records is protected. In one embodiment, in response to the access permission setting operation for the operating status records, access permission management is performed on users and applications that access the operating status records.

[0101] For example, when a user needs to access the running status records stored in the terminal device, the user's identity needs to be verified, such as through passwords, fingerprint recognition, facial recognition and other authentication methods. After the user's identity is identified, the user can manage the access rights of users and applications that access the running status records in the visual access right setting interface. For example, the access rights can be set to control which applications and users can access the running status records. Furthermore, the type of data that users or applications can access can be set, such as only allowing a certain application to access screen image data.

[0102] By setting access permissions for users and applications, users can have more freedom in managing operating status records. Different users can customize data access permissions according to their own needs, thereby meeting the data access management needs of different users and effectively improving the security of local storage of operating status records.

[0103] In addition to controlling whether users and applications have access to running status records, the scope of screen image data accessed by users can also be controlled. In one embodiment, the above method can also include: obtaining target screen image data accessed by the target user from the screen image data stored in the terminal device, and setting access permissions for the target screen image data so that the target screen image data can only be accessed by the target user.

[0104] Exemplarily, the target screen image data can be restricted to be accessed only by the target user through the Access Control List (ACL) of the Windows operating system.

[0105] By controlling the access scope of screen image data, in a scenario where multiple users access operating status data, users can only access screen image data within the scope of their permissions, ensuring the security and privacy of screen image data in multi-user scenarios.

[0106] Thirdly, in order to meet the data privacy requirements of different users, in one embodiment, in response to an editing operation on the application management list, a non-associated application list is determined based on the edited application management list;

[0107] Take screenshots of the terminal device at the capture frequency to obtain screen image data, including:

[0108] When it is detected that the associated application of the screenshot data belongs to the non-associated application list, the screenshot of the display area of ​​the associated application in the terminal device is stopped.

[0109] The application management list may be a list of applications that can be filtered, and the non-associated application list is a list of applications that need to be filtered and are selected by the user. The filtered applications refer to applications that do not need to obtain screen image data.

[0110] For example, the user can add or delete filtered applications or websites based on the editing operation of the application in the application management list. For example, the user can select the filtered application in the application management list, or add the website to be filtered in the application management list by entering the URL or keyword of the website to be filtered. After the user completes editing the application management list, the non-associated application list can be determined according to the editing results. For example, if the non-associated application list contains the URL of a website, if the user opens the website through the URL in the browser, then at this time, the screenshot of the display area of ​​the website is suspended or the screenshot of the entire screen is suspended.

[0111] By allowing users to customize the applications to be filtered, the data privacy needs of different users can be met, data privacy protection can be customized, and the user experience can be improved.

[0112] Finally, when the user decides to delete the running status record, the running status record needs to be securely deleted to prevent the traces left after the running status record is deleted from threatening the user's data privacy. In one embodiment, in response to the deletion operation on the running status record, the running status record is erased.

[0113] For example, users can delete part or all of the operation status records in the visual interface "Operation Status Record Deletion Interface". When the user chooses to delete the operation status records, security erasure operations such as "multiple overwrites" or "data shredding" will be performed to completely delete the operation status data that needs to be cleaned up to ensure that the deleted operation status data cannot be recovered.

[0114] By erasing the operating status records that need to be deleted, it is ensured that the operating status records are permanently deleted, thereby avoiding to a certain extent the threat to the user's data privacy posed by the traces left after the deletion of the operating status records.

[0115] In one embodiment, reference Figure 5 As shown, data privacy protection processing of operation status records can be implemented according to steps S501 to S507:

[0116] Step S501, generating AES-256 keys for screen image data and operation status data respectively, so as to encrypt the operation status record;

[0117] Step S502, encrypting the AES-256 key using a public key encryption algorithm;

[0118] Step S503, storing the running status record with the AES-256 key together with the encrypted AES-256 key in the terminal device;

[0119] Step S504, setting identity authentication for the running status record to identify the user who accesses the running status record;

[0120] Step S505, in response to the user's operation of setting access rights for the running status record, performing access rights management on the user and application accessing the running status record;

[0121] Step S506, in response to the user's editing operation on the application management list, determining a list of non-associated applications based on the edited application management list, so that when taking a screenshot, the display area of ​​the non-associated application in the terminal device is not subjected to a screenshot process;

[0122] Step S507: in response to the deletion operation on the running status record, data erasure processing is performed on the running status record to ensure that the cleared running status record is completely deleted.

[0123] based on Figure 5 A comprehensive privacy protection method is used in the four stages of obtaining, storing, accessing, and deleting running status records. At the same time, users are given a high degree of freedom in the privacy protection process to meet their diverse needs for data privacy protection.

[0124] The application process of the running status record is described below. For example, the user can directly view the running status record stored in the terminal device after decryption, and can also restore the associated application to the historical running state through the running status record.

[0125] After acquiring and safely storing the running status record of the application, the user can view the screen image data or the running status data by accessing the running status record on the terminal device. In one embodiment, the running status record is desensitized before being viewed.

[0126] Among them, data desensitization refers to the anonymization of user privacy information in the operation status records. For example, data desensitization technologies such as shielding sensitive information and data disguise can be used to protect user privacy information. For example, the middle part of the ID number can be replaced with asterisks, such as 123456******7890.

[0127] Before the running status record is viewed, the running status record is desensitized to prevent the user's privacy information from being leaked. At the same time, since the acquisition, storage and access of the running status record are all performed on the terminal device, compared with the related technology of storing data in the cloud, when the user needs to access it, a communication link is established to obtain the application's running data from the cloud, which effectively avoids the risk of privacy leakage during data transmission and improves the efficiency and real-time performance of data access.

[0128] In one implementation, in response to a data access operation on the running status record, a data access log is generated to record access information of the running status record.

[0129] The access information may include the visitor (user or application) of the running status record, the access time, and the operation performed on the running status record. After the running status record is accessed, a data access log is generated to record the access process of the running status record, which provides convenience for users to review and review the access operation of the running status record.

[0130] Since in the related art, the running data of the application stored locally is often limited due to the permission restrictions of the application, and it is difficult to restore the running status of the application based on the locally stored application running data, and this exemplary embodiment provides a method for obtaining the running data of the application through the screen image data, which can effectively improve the integrity of the locally stored application running data, making it possible to fully restore the running status of the application based only on the locally stored application running data. Therefore, in one embodiment, the above method may also include: restoring the running status of the associated application through the running status record stored in the terminal device.

[0131] The above restores the running status of the associated application through the running status record stored in the terminal device, refer to Figure 6 As shown, steps S610 to S640 may be included:

[0132] Step S610, in response to receiving the natural language query operation, obtaining application status query data according to the natural language query content corresponding to the natural language query operation;

[0133] The application status query data may be a word segment corresponding to the natural language query content input by the user; obtaining the application status query data according to the natural language query content corresponding to the natural language query operation may include the following steps:

[0134] A large language model is used to correct text errors in natural language query content, and the corrected natural language query content is segmented to obtain application status query data.

[0135] For example, a pre-trained natural language processing (NLP) model (such as BERT or GPT) can be used to perform spelling correction and word segmentation on the natural language query content entered by the user. Among them, BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer) are able to capture semantic and contextual information in natural language through large-scale text data training. For example, if the natural language query content entered by the user is "show me the files I edited last Wednesday", the natural language query content corrected by the large language model is "show me the documents I edited last Wednesday". Segmenting this content can make the word segmentation results more accurate.

[0136] Users can query the application's operating status records through natural language, which eliminates the need for users to remember precise details during the query process, thereby improving the intelligence of the data query process.

[0137] Continue to refer Figure 6 In step S620, the target running status record matching the application status query data is retrieved from the running status records stored in the terminal device.

[0138] In one implementation, the acquired application query state data may be matched with the operation state records stored in the terminal device for similarity, so as to obtain a target operation state record that meets the application query state data requirements.

[0139] However, considering that in actual application scenarios, a large number of running status records are often stored in the terminal device, in order to reduce the time cost of the matching process, vector matching can be used to improve the matching speed. In one embodiment, the target running status record matching the application status query data is retrieved from the running status records stored in the terminal device. Figure 7 As shown, steps S710 to S730 may be included:

[0140] Step S710: acquiring an operation state vector corresponding to an operation state record stored in a terminal device, and acquiring an application state query vector corresponding to application state query data.

[0141] In one implementation, the above-mentioned operation status record includes screen image data; obtaining the operation status vector corresponding to the operation status record stored in the terminal device may include the following steps:

[0142] Performing image segmentation on the screen image data to extract picture data from the screen image data, and extracting text data associated with the picture data;

[0143] Generate image vectors based on image data, and generate text vectors based on text data;

[0144] The image vector and the text vector are fused with features through the attention mechanism to obtain the running state vector.

[0145] The above-mentioned picture data may include pictures included in the screen image data, and the text data associated with the picture data may include the title of the picture, the text in the picture, and other data related to the picture. Taking a webpage screenshot (screen image data) containing multiple pictures of cats as an example, the webpage screenshot may contain multiple pictures of cats, titles, text, and detailed descriptions, etc., then the above-mentioned picture data may include the pictures of cats in the webpage screenshot, and the text data may include the titles, text, and detailed descriptions corresponding to the pictures of cats.

[0146] For example, first, the image vector of the image data can be obtained through the Vision Transformer (ViT). ViT is an image processing model based on the Transformer architecture, which can divide the image into patches of fixed size, flatten these patches and input them into the Transformer encoder for processing. The principle is to divide the image into patches of fixed size (such as 16x16 pixels), then flatten each patch into a one-dimensional vector, and embed it into a high-dimensional vector space through a linear transformation, and add a position code to each patch to retain its position information in the original image; finally, the patches and their position information are input into the Transformer encoder, and multi-layer self-attention calculations are performed to extract the global features of the image, and the features output by the Transformer are aggregated through the fully connected layer to obtain the high-dimensional vector representation corresponding to the image:

[0147] Specifically, the screen image data can be read through the storage path of the screen image data in the running status record, or the storage directory of the screen image data. The file format of the screen image data can be PNG or JPEG. The process of reading the screen image data includes: for any screen image data, each pixel in the screen image data is represented as a three-dimensional vector (RGB), and the RGB vectors of all pixels in the screen image data are stored in a matrix to obtain an image matrix; then the read image matrix is ​​divided into patches of fixed size, usually, each patch size is 16x16 pixels, and each patch after division is normalized, for example, the pixel value is divided by 255 so that the value of each pixel is between 0 and 1.

[0148] Since the size of each patch is 16x16 pixels, and each pixel is a three-dimensional vector, the dimension of any patch is 16x16x3. Based on the dimensional matrix corresponding to the patch, the corresponding one-dimensional array is obtained, and a one-dimensional vector of size 768 (i.e., 16x16x3) can be obtained. The one-dimensional vector is multiplied by the preset weight matrix to convert the low-dimensional one-dimensional vector into a high-dimensional vector; the dimension of the weight matrix can be [768, d_model], where d_model is the dimension of the target high-dimensional vector.

[0149] After obtaining the high-dimensional vector of each patch in the screen image data, the position information of each patch can be obtained: According to formulas (1) and (2), the position encoding can be performed for each patch to obtain the position information PE of each patch:

[0150]

[0151] Among them, pos is the position index of the patch, which is used to indicate the position of the patch in the image, i is the dimension index, and the value range of i is [0, d_model / 2].

[0152] After obtaining the high-dimensional vector and position information of each patch, the position information can be embedded into the high-dimensional vector to obtain an embedded vector. The embedded vector of each patch can be input into the Transformer encoder. The Transformer encoder consists of multiple layers, each of which contains a multi-head self-attention mechanism and a feedforward neural network. The Transformer encoder can extract features from the embedded vector according to formula (3) combined with the multi-head self-attention mechanism:

[0153]

[0154] Among them, Q, K, and V represent the query vector (Q), key vector (K), and value vector (V) obtained by linear transformation of the embedding vector, respectively. T represents the matrix transposition of the vector K. is the scaling factor, d k is the dimension of K, and Softmax is the normalization function.

[0155] After obtaining the attention result of the embedded vector, the attention can be input into the feedforward neural network of the Transformer model, which usually includes two fully connected layers and a nonlinear activation function. The Transformer model also includes layer normalization and residual connection to ensure gradient stability to improve the model training effect.

[0156] After obtaining the output results of the Transformer model for each patch in the screen image data, it is necessary to perform feature aggregation on the output results of the Transformer model. For example, feature aggregation can be completed by taking the average value of the output results of the Transformer model corresponding to the patches of the screen image data, or the output result of the Transformer model corresponding to the first patch of the screen image data as the image vector I_vec.

[0157] Secondly, the text vector associated with the image vector in the screen image data can be obtained. For example, the screen image data can be processed by OCR to extract the title and description text under each picture in the screen image data and convert them into a text vector T_vec.

[0158] Finally, the multi-head self-attention mechanism can be used to perform multi-modal data fusion on the image vector I_vec and the text vector T_vec to obtain the running state vector. The multi-head self-attention mechanism uses multiple independent attention heads to capture features in different subspaces, thereby enhancing the expressiveness of the model; the image vector I_vec and the text vector T_vec can be used as the input vectors of the multi-head self-attention mechanism, and the multi-head self-attention mechanism can perform linear transformation on the image vector I_vec and the text vector T_vec respectively to obtain the query vector (Q), key vector (K) and value vector (V); the dot product of the query vector (Q) and the key vector (K) is calculated by the softmax function to obtain the attention weight, and the value vector (V) is weighted summed using the attention weight to output the fusion vector, and the fusion vectors output by multiple heads are spliced ​​and linearly transformed to obtain the running state vector corresponding to the screen image data.

[0159] By acquiring the image vector and text vector of the screen image data and performing feature fusion on the image vector and the text vector, an operation status vector is obtained, thereby achieving seamless fusion of the image data and the text data, so that the operation status vector can represent the operation status data stored in the screen image data, thereby providing the possibility of obtaining the corresponding operation status record through efficient vector retrieval.

[0160] In one embodiment, reference Figure 8 As shown, the running state vector corresponding to the screen image data can be obtained according to steps S801 to S807:

[0161] Step S801, extracting text data from screen image data using OCR technology;

[0162] Step S802, performing word segmentation processing on the text data, and generating a corresponding text vector based on key information in the text data after word segmentation;

[0163] Step S803, performing image segmentation on the screen image data to obtain patches corresponding to the screen image data, and obtaining a high-dimensional vector of each patch;

[0164] Step S804, obtaining a position code of each patch based on the position of each patch in the screen image data;

[0165] Step S805, embedding the position code of each patch into its corresponding high-dimensional vector to obtain an embedding vector corresponding to each patch;

[0166] Step S806, input the embedding vector corresponding to each patch into the Transformer model, and perform feature aggregation on the output result of each embedding vector in the Transformer model to obtain an image vector;

[0167] Step S807, performing multimodal data fusion on the image vector and the text vector through a multi-head self-attention mechanism to obtain a running state vector.

[0168] based on Figure 8 The method can achieve seamless integration of image data and text data, so that the running state vector can represent the running state data stored in the screen image data.

[0169] In order to further improve the efficiency of obtaining the running state vector, in one embodiment, the above-mentioned generation of the image vector based on the picture data and the generation of the text vector based on the text data may include:

[0170] In the case where the screen image data includes multiple picture data and multiple text data, the neural processing unit is used to process the multiple picture data and the multiple text data in parallel to generate multiple image vectors and multiple text vectors.

[0171] That is to say, the neural processing unit can be used to process image data and text data in parallel to generate multiple image vectors and multiple text vectors at the same time. The neural processing unit (NPU) can perform a large number of matrix operations and vector calculations through hardware accelerators and parallel computing capabilities to achieve efficient processing of image data and text data, thereby improving the efficiency of obtaining the running state vector. Common NPU hardware units can include Qualcomm Snapdragon NPU or Google Edge TPU.

[0172] For example, the screen image data first needs to be preprocessed, such as grayscale and binarization, and then the preprocessed screen image data is segmented to obtain patches of the screen image data, and the patches are processed in parallel by the NPU to simultaneously obtain multiple image vectors corresponding to the patches; multiple text data in the screen image data are extracted through OCR technology, and then the multiple text data are processed in parallel by the NPU to simultaneously generate multiple text vectors; finally, the multiple image vectors and multiple text vectors are feature fused through the multi-head self-attention mechanism to generate the running status vector corresponding to the screen image data.

[0173] In one embodiment, reference Fig. 9 As shown, the generation process of the running state vector can be accelerated according to steps S901 to S906:

[0174] Step S901, obtaining screen image data, and preprocessing the screen image data (grayscale, binarization, etc.);

[0175] Step S902, extracting text data from the pre-processed screen image data by using OCR technology;

[0176] Step S903, performing preprocessing such as word segmentation and stop word removal on the text data, and performing parallel feature extraction and feature aggregation on the preprocessed text data through the NPU to simultaneously generate multiple text vectors;

[0177] Step S904, performing image segmentation on the pre-processed screen image data to obtain patches corresponding to the screen image data;

[0178] Step S905, using the NPU to obtain the embedding vector of each patch in parallel, and inputting multiple embedding vectors into the Transformer model in parallel, and performing feature aggregation on the output results of the Transformer model of each embedding vector in parallel to obtain multiple image vectors at the same time;

[0179] Step S906, performing multimodal data fusion on multiple image vectors and multiple text vectors through a multi-head self-attention mechanism to obtain an operating state vector corresponding to the screen image data.

[0180] based on Fig. 9 The method uses NPU to obtain image vectors and text vectors in parallel, which improves the acquisition speed of image vectors and text vectors and further improves the acquisition efficiency of running state vectors.

[0181] In one implementation, after the running state vector is generated, the generated running state vector may be encrypted.

[0182] For example, before storing the running state vector, each running state vector can be individually encrypted using an AES-256 key, and the encrypted running state vector is stored together with the AES-256 key (encrypted using the public key). When a user accesses the running state vector, the running state vector is decrypted using the authentication and decryption key to access the running state vector.

[0183] By encrypting the running state vector, the security and privacy of the vector data can be ensured, providing comprehensive privacy protection for application running data.

[0184] In one implementation, the obtaining of the application state query vector corresponding to the application state query data may include:

[0185] The deep model is used to extract key query information from the application status query data and transform the key query information into an application status query vector.

[0186] Among them, the key query information may include keywords of the application status query data. For example, the keywords in the application status query data can be extracted through an NLP model (such as BERT or GPT), and the semantic features in the extracted keywords can be converted into corresponding high-dimensional vectors to obtain an application status query vector.

[0187] After obtaining the running status vector and application status query vector, continue to refer to Figure 7 In step S720, similarity matching is performed on the running state vector and the application state query vector, and a target running state vector is obtained according to the result of the similarity matching.

[0188] In one embodiment, obtaining the target running state vector according to the similarity matching result includes:

[0189] From the similarity matching results, select a preset number of candidate running state vectors that are most similar to the application state query vector;

[0190] Generate a search result list based on the candidate running state vector;

[0191] In response to a selection operation on the search result list, a target operation state vector is determined among the candidate operation state vectors.

[0192] The above-mentioned preset number is the number of candidate running state vectors. The present disclosure does not specifically limit the specific value of the preset number. For example, the preset number may be 5. Then, after similarity matching between the running state vector and the application state query vector, the 5 running state vectors with the highest similarity to the application state query vector may be selected as candidate running state vectors. The search result list is used to display relevant information of the candidate running state vectors to the user, so that the user can select the target running state vector according to the relevant information of the candidate running state vectors.

[0193] For example, the cosine similarity between the application state query vector and the running state vector can be calculated by formula (4) to perform similarity matching between the running state vector and the application state query vector:

[0194]

[0195] Wherein, A and B are the application state query vector and the running state vector respectively; cosine_similarity can be cosine similarity, and the larger its value, the more similar the application state query vector and the running state vector are. After obtaining the cosine similarity between the application state query vector and all the running state vectors, the running state vectors can be sorted from high to low only according to the cosine similarity to obtain the result of similarity matching; the cosine similarity, timestamp and user's selection preference corresponding to each running state vector can also be weighted to obtain the score of each running state vector, and the running state vectors can be sorted from high to low according to the score to obtain the result of similarity matching; wherein, the user's selection preference can be obtained according to the user's historical query record, that is, if the user enters content related to a certain running state vector in the previous query content, the flag bit of the user's selection preference corresponding to the running state vector can be "1", and if a certain running state vector has not been queried before, its flag bit can be "0"; according to the product of the flag bit and the weight corresponding to the user's selection preference, the score corresponding to the running state vector in the "user's selection preference" can be obtained. The score of the running status vector in terms of "timestamp" can be obtained based on the product of the timestamp and the corresponding weight. The closer the timestamp of the running status vector is to the user query time or the target query time (the query time information contained in the application status query data), the greater the weight assigned to the timestamp; that is, the running status vector with the latest or most relevant timestamp is displayed first.

[0196] The target running state vector is determined by the user's selection operation in the candidate running state vectors. This ensures that the user determines the target running state vector from the candidate running state vectors that are more closely matched with the query content, providing the possibility for subsequent running state recovery of the application through the target running state vector.

[0197] In order to further speed up the vector matching speed, in one implementation, after obtaining the running state vector corresponding to the running state record stored in the terminal device, the above method may further include the following steps:

[0198] Store the running state vector in the specified vector database;

[0199] Perform similarity matching on the running state vector and the application state query vector, and obtain the target running state vector according to the similarity matching result, including:

[0200] In the vector database, similarity matching is performed on the application state query vector to obtain a target operation state vector according to the result of the similarity matching.

[0201] The specified vector database may include a database capable of performing efficient vector retrieval. Exemplarily, the vector database may include FAISS (Facebook AI Similarity Search). FAISS can process large-scale vector data and provide a fast nearest neighbor search function by optimizing the index structure and search algorithm.

[0202] Since the data volume of the running state vector is often huge, storing the above running state vector in a specified vector database can speed up the retrieval process and effectively improve the efficiency of obtaining the target running state vector.

[0203] In one embodiment, reference Fig.10 As shown, according to steps S1001 to S1008, a target running state vector matching the application query state vector can be obtained in the vector database:

[0204] Step S1001, in response to receiving a natural language query operation, according to the natural language query content corresponding to the natural language query operation, using a large language model to correct text errors in the natural language query content, and performing word segmentation processing on the corrected natural language query content to obtain application status query data;

[0205] Step S1002, extracting key query information from the application status query data using a deep model, and converting the key query information into an application status query vector;

[0206] Step S1003, generating a corresponding running state vector through the screen image data stored in the running state record;

[0207] Step S1004, storing the running state vector in the specified vector database FAISS;

[0208] Step S1005, in FAISS, similarity matching is performed on the application state query vector, and from the similarity matching results, a preset number of candidate running state vectors that are most similar to the application state query vector are selected;

[0209] Step S1006, weighting the cosine similarity, timestamp and user selection preference corresponding to the candidate running state vector to obtain a score corresponding to the candidate running state vector;

[0210] Step S1007, sorting the candidate running state vectors from high to low according to the scores corresponding to the candidate running state vectors to generate a search result list;

[0211] Step S1008 : In response to a user's selection operation on the search result list, a target running state vector is determined from among the candidate running state vectors.

[0212] based on Fig.10 The method generates an application state query vector and stores the running state vector in a vector database, which speeds up the vector retrieval process, is beneficial to improving the acquisition efficiency of the target running state vector, and further improves the running state recovery speed of the application.

[0213] After obtaining the target running state vector, continue to refer to Figure 7 In step S730, a target running state record associated with the target running state vector is obtained in the running state record.

[0214] Since the running state vector is generated according to the screen image data, and the running state record stores the screen image data, the target running state record associated with the target running state vector can be obtained from the running state record stored in the terminal device through the screen image data.

[0215] based on Figure 7 Compared with the related art, which requires data transmission with the cloud to obtain the target running status record, the method reduces the time cost of data transmission, realizes efficient retrieval of the target running status record locally, and is conducive to accelerating the application recovery process, thereby improving the user experience.

[0216] Step S630, obtaining the application to be restored based on the running status data in the target running status record;

[0217] In one implementation, the running status data includes identification information, and the acquiring the application to be restored based on the running status data in the target running status record may include: determining the application to be restored based on the identification information.

[0218] Since the running status data is generated according to the identification information and the running status information of the application, the application to be restored whose running status is to be restored can be determined according to the identification information.

[0219] Continue to refer Figure 6 , step S640, restoring the running state of the application to be restored according to the screen image data and / or running state data in the target running state record.

[0220] For example, since the screen image data and running status data in the target running status record record the historical running status of the application to be restored, such as the application's process ID, program name, open document path, tab URL, and scroll position, etc.; therefore, based on this information, the user can choose to restore the application's running status to a historical moment, and after the application's running status is restored to a historical moment, continue to publish or create the application based on the historical moment.

[0221] The following takes a browser as an example to illustrate the process of restoring the running state of the application to be restored.

[0222] In one embodiment, the application to be restored includes a browser, and restoring the running state of the application to be restored according to the screen image data and / or running state data in the target running state record may include the following steps:

[0223] Launch your browser;

[0224] Obtaining the network resource address stored in the running status data and the relative scrolling position of the target tab page corresponding to the network resource address displayed in the screen image data;

[0225] A target tab is opened in the browser based on the network resource address, and a scroll position of the target tab is set according to a relative scroll position to complete the recovery of the browser.

[0226] Exemplarily, the browser can be opened through the API provided by the operating system, such as ShellExecute of the Windows operating system, and then an automated script, such as Selenium, can be used to automatically open the target tab in the browser and automatically scroll the target tab to the relative scroll position displayed in the screen image data to restore the browser operation.

[0227] In one implementation, the target tab page includes a text box, the running status data includes text box visible information of the text box to be restored, and the method may further include the following steps:

[0228] Obtaining the display position of the text box to be restored in the screen image data;

[0229] Determine the text box to be restored based on the display position in the text box in the target tab page;

[0230] Write the visible information of the text box into the text box to be restored to restore the browser.

[0231] That is to say, the method of restoring the browser's operating status based on the target operating status record in this exemplary embodiment can not only restore the tabs opened by the browser at historical moments, but also restore the content entered by the user in the form or text box in the tab, effectively improving the comprehensiveness of the browser's historical operating status recovery, thereby improving the user experience.

[0232] In one embodiment, reference Fig.11 As shown, the running state of the application can be restored according to steps S1101 to S1106:

[0233] Step S1101, determining an associated target operating state record based on a target operating state vector;

[0234] Step S1102, obtaining identification information of the running status data in the target running status record;

[0235] Step S1103, determining the application to be restored based on the identification information;

[0236] Step S1104, starting the application to be restored;

[0237] Step S1105, reading the operation status data and screen image data in the target operation status record;

[0238] Step S1106: restore the running state of the application to be restored according to the screen image data and / or the running state data.

[0239] based on Fig.11 The method can realize backtracking of the running state of the document or browser waiting for the application to be restored through the target running state record, so that the user can continue to develop or create the application after the restoration.

[0240] In one embodiment, reference Fig.12 As shown, according to steps S1201 to S1220, the running status of the application can be stored locally in the terminal device, and the application can be backtracked according to the stored running status:

[0241] Step S1201, in response to the setting operation for the capture frequency, determining the capture frequency of the screen image data; and selecting a storage path for recording the running status on the terminal device;

[0242] Step S1202, taking a screen snapshot of the terminal device according to the capture frequency to obtain screen image data;

[0243] Step S1203, naming the screen image data according to the capture timestamp of the screen image data;

[0244] Step S1204, while acquiring the screen image data, determining the application in the terminal device in the running state as the associated application; and acquiring the running state data of the associated application;

[0245] Step S1205, binding the named screen image data and the running status data of the associated application to generate a running status record of the associated application;

[0246] Step S1206, encrypting the running status record using the AES-256 key, and encrypting the AES-256 key using a public key encryption algorithm;

[0247] Step S1207, storing the running status record with the AES-256 key and the encrypted AES-256 key in the storage path of step 1201;

[0248] Step S1208, in response to the user's operation of setting access rights for the running status record, performing access rights management on the user and application accessing the running status record;

[0249] Step S1209: in response to the deletion operation on the running status record, data erasure processing is performed on the running status record to ensure that the cleared running status record is completely deleted.

[0250] Step S1210, reading the running status record stored in the screen image data, and performing image preprocessing on the screen image data in the running status record;

[0251] Step S1211, using OCR technology to extract text from the screen image data after image preprocessing to obtain text data;

[0252] Step S1212, performing preprocessing such as word segmentation and stop word removal on the text data, and performing parallel feature extraction and feature aggregation on the preprocessed text data through the NPU to simultaneously generate multiple text vectors;

[0253] Step S1213, using the ViT model to perform image segmentation on the screen image data after image preprocessing to extract picture data from the screen image data;

[0254] Step S1214, using the NPU to obtain the embedding vector of each patch in parallel, and inputting multiple embedding vectors into the Transformer model in parallel, and performing feature aggregation on the output results of the Transformer model of each embedding vector in parallel to obtain multiple image vectors at the same time;

[0255] Step S1215, performing feature fusion processing of multiple image vectors and multiple text vectors in parallel through the NPU to obtain an operation state vector corresponding to the screen image data, and adding the operation state vector to the vector database;

[0256] Step S1216, in response to receiving the natural language query operation, obtaining application status query data according to the natural language query content corresponding to the natural language query operation;

[0257] Step S1217, extracting key query information from the application status query data using the deep model, and converting the key query information into an application status query vector;

[0258] Step S1218, performing similarity matching on the application state query vector in the vector database, so as to obtain a target operation state vector according to the result of the similarity matching;

[0259] Step S1219, obtaining a target running state record associated with the target running state vector in the running state record;

[0260] Step S1220: Restore the running state of the application to be restored according to the screen image data and / or running state data in the target running state record.

[0261] based on Fig.12 The method can efficiently retrieve screen image data and save and restore the application running status at any time, realize the local application running status recovery, and reduce the dependence of the application recovery process on the cloud.

[0262] The exemplary embodiment of the present disclosure also provides a data processing device, which is applied to a terminal device including a screen. Fig.13 As shown, the data processing device 1300 may include:

[0263] The interception frequency acquisition module 1310 is configured to acquire the interception frequency of the screen image data;

[0264] The screen image data acquisition module 1320 is configured to take a screen snapshot of the terminal device according to the capture frequency to acquire screen image data;

[0265] The running status data acquisition module 1330 is configured to acquire the associated application of the screen image data and acquire the running status data of the associated application;

[0266] The running status record storage module 1340 is configured to bind the screen image data and the running status data of the associated application, generate the running status record of the associated application, and store the running status record in the terminal device.

[0267] In one implementation, the above-mentioned associated application for acquiring screen image data may include:

[0268] While acquiring the screen image data, the applications in the terminal device that are in a running state are determined as associated applications.

[0269] In one implementation, the obtaining of the running status data of the associated application may include:

[0270] Obtaining the running status information and identification information of the associated application;

[0271] The identification information of the associated application is bound to the running status information to obtain the running status data of the associated application.

[0272] In one implementation, the obtaining of the running status information of the associated application may include:

[0273] Extracting usage data of associated applications from screen image data;

[0274] The usage data of the associated application is stored in a specified format to obtain the running status information.

[0275] In one implementation, the above-mentioned binding of the screen image data and the running status data of the associated application to generate the running status record of the associated application may include:

[0276] The storage information of the screen image data is written into the running status data of the associated application according to a specified format, so as to bind the screen image data with the running status data and obtain the running status record of the associated application.

[0277] In one embodiment, the above device may further include:

[0278] The running status of the associated application is restored through the running status record stored in the terminal device.

[0279] In one implementation, the above-mentioned restoring the running status of the associated application through the running status record stored in the terminal device may include:

[0280] In response to receiving the natural language query operation, obtaining application status query data according to the natural language query content corresponding to the natural language query operation;

[0281] Retrieving a target running status record matching the application status query data from the running status records stored in the terminal device;

[0282] Acquire the application to be restored based on the running status data in the target running status record;

[0283] The running state of the application to be restored is restored according to the screen image data and / or the running state data in the target running state record.

[0284] In one implementation, obtaining application status query data according to the natural language query content corresponding to the natural language query operation may include:

[0285] A large language model is used to correct text errors in natural language query content, and the corrected natural language query content is segmented to obtain application status query data.

[0286] In one implementation, retrieving the target running status record matching the application status query data from the running status records stored in the terminal device may include:

[0287] Obtaining an operation state vector corresponding to an operation state record stored in a terminal device, and obtaining an application state query vector corresponding to application state query data;

[0288] Perform similarity matching on the running state vector and the application state query vector, and obtain the target running state vector according to the result of similarity matching;

[0289] A target running state record associated with the target running state vector is obtained in the running state record.

[0290] In one implementation, the above-mentioned operation status record includes screen image data; and obtaining the operation status vector corresponding to the operation status record stored in the terminal device may include:

[0291] Performing image segmentation on the screen image data to extract picture data from the screen image data, and extracting text data associated with the picture data;

[0292] Generate image vectors based on image data, and generate text vectors based on text data;

[0293] The image vector and the text vector are fused with features through the attention mechanism to obtain the running state vector.

[0294] In one implementation, the obtaining of the application state query vector corresponding to the application state query data may include:

[0295] The deep model is used to extract key query information from the application status query data and transform the key query information into an application status query vector.

[0296] In one implementation, obtaining the target running state vector according to the similarity matching result may include:

[0297] From the similarity matching results, select a preset number of candidate running state vectors that are most similar to the application state query vector;

[0298] Generate a search result list based on the candidate running state vector;

[0299] In response to a selection operation on the search result list, a target operation state vector is determined among the candidate operation state vectors.

[0300] In one implementation, after acquiring the running state vector corresponding to the running state record stored in the terminal device, the apparatus may further include:

[0301] Store the running state vector in the specified vector database;

[0302] Perform similarity matching on the running state vector and the application state query vector, and obtain the target running state vector according to the similarity matching result, including:

[0303] In the vector database, similarity matching is performed on the application state query vector to obtain a target operation state vector according to the result of the similarity matching.

[0304] In one implementation, the above-mentioned running status data includes identification information, and obtaining the application to be restored based on the running status data in the target running status record may include:

[0305] An application to be restored is determined based on the identification information.

[0306] In one embodiment, the application to be restored includes a browser, and restoring the running state of the application to be restored according to the screen image data and / or running state data in the target running state record may include:

[0307] Launch your browser;

[0308] Obtaining the network resource address stored in the running status data and the relative scrolling position of the target tab page corresponding to the network resource address displayed in the screen image data;

[0309] A target tab is opened in the browser based on the network resource address, and a scroll position of the target tab is set according to a relative scroll position to complete the recovery of the browser.

[0310] In one implementation, the target tab page includes a text box, the running status data includes text box visible information of the text box to be restored, and the device may further include:

[0311] Obtaining the display position of the text box to be restored in the screen image data;

[0312] Determine the text box to be restored based on the display position in the text box in the target tab page;

[0313] Write the visible information of the text box into the text box to be restored to restore the browser.

[0314] In one implementation, the above-mentioned capture frequency of obtaining screen image data may include:

[0315] In response to a setting operation for the interception frequency, the interception frequency of the screen image data is determined.

[0316] In one embodiment, the above device may further include:

[0317] The operation status record in the terminal device is encrypted, and the encrypted operation status record is stored in the terminal device.

[0318] The specific details of each part of the above device have been described in detail in the implementation method part, so they will not be repeated here.

[0319] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0320] The exemplary embodiments of the present disclosure further provide a computer program product, the computer program product comprising a computer program, and the computer program implements the above data processing method when executed by a processor.

[0321] Computer programs for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer program may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0322] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0323] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0324] Computer programs can be carried or transmitted through electrical, magnetic, optical, electromagnetic, infrared and other signals. Electronic devices can convert signals carrying computer programs into digital signals to run computer programs. When a computer program runs on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device can execute) the method steps of various exemplary embodiments of the present disclosure, such as the above-mentioned data processing method, which includes the following steps: obtaining the capture frequency of screen image data; taking a screenshot of the terminal device according to the capture frequency to obtain screen image data; obtaining the associated application of the screen image data, and obtaining the running status data of the associated application; binding the screen image data and the running status data of the associated application to generate a running status record of the associated application, and storing the running status record in the terminal device.

[0325] By executing the above method steps through a computer program, on the one hand, a solution for periodically capturing screen image data and binding the screen image data with the running status data of the associated application to obtain the running status record of the associated application is provided, which avoids the problem of permission restriction in the process of obtaining the running status data of the application locally to a certain extent, improves the completeness of the running data of the application obtained locally, effectively improves the richness and diversity of the running data of the application stored locally, satisfies the user's demand for the use of the running data of the application, and improves the user experience. On the other hand, the running status record of the associated application is stored in the terminal device, which improves the problem of low security in the process of storing the running data to a certain extent compared with the method of storing the running data of the application in the cloud in the related technology, and at the same time, reduces the data transmission cost brought by storing the running data of the application in the cloud, facilitates the user to quickly access the running data of the application, and improves the efficiency of the user to access the running data of the application. On the other hand, storing the running status record of the application in the terminal device can reduce the user's dependence on the cloud service in the process of accessing the running data, and further improves the flexibility of the user to access the running data of the application.

[0326] The exemplary embodiments of the present disclosure also provide an electronic device, such as the terminal device 110 described above. The electronic device may include a processor and a memory. The memory stores executable instructions of the processor, such as a computer program. The processor executes the method steps of various exemplary embodiments of the present disclosure by executing the executable instructions. In addition, the electronic device may also include a display for acquiring screen image data.

[0327] Reference below Fig.14 , the electronic device is exemplarily described in the form of a general-purpose computing device. It should be understood that Fig.14 The electronic device 1400 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0328] like Fig.14 As shown, the electronic device 1400 may include: a processor 1410 , a memory 1420 , a bus 1430 , an I / O (input / output) interface 1440 , a network adapter 1450 , and a display 1460 .

[0329] Processor 1410 may include one or more processing units. For example, processor 1410 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0330] Processor 1410 can be used to execute executable instructions stored in memory 1420, such as the above-mentioned data processing method, which includes the following steps: obtaining the capture frequency of screen image data; taking a screenshot of the terminal device according to the capture frequency to obtain screen image data; obtaining the associated application of the screen image data, and obtaining the running status data of the associated application; binding the screen image data and the running status data of the associated application to generate a running status record of the associated application, and storing the running status record in the terminal device.

[0331] By executing the above method steps through a computer program, on the one hand, a solution for periodically capturing screen image data and binding the screen image data with the running status data of the associated application to obtain the running status record of the associated application is provided, which avoids the problem of permission restriction in the process of obtaining the running status data of the application locally to a certain extent, improves the completeness of the running data of the application obtained locally, effectively improves the richness and diversity of the running data of the application stored locally, satisfies the user's demand for the use of the running data of the application, and improves the user experience. On the other hand, the running status record of the associated application is stored in the terminal device, which improves the problem of low security in the process of storing the running data to a certain extent compared with the method of storing the running data of the application in the cloud in the related technology, and at the same time, reduces the data transmission cost brought by storing the running data of the application in the cloud, facilitates the user to quickly access the running data of the application, and improves the efficiency of the user to access the running data of the application. On the other hand, storing the running status record of the application in the terminal device can reduce the user's dependence on the cloud service in the process of accessing the running data, and further improves the flexibility of the user to access the running data of the application.

[0332] The memory 1420 may include a volatile memory, such as a RAM 1421, a cache unit 1422, and may also include a non-volatile memory, such as a ROM 1423. The memory 1420 may also include one or more program modules 1424, such program modules 1424 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof may include the implementation of a network environment. For example, the program module 1424 may include each module in the above-mentioned apparatus 1300.

[0333] The bus 1430 is used to realize the connection between different components of the electronic device 1400, and may include a data bus, an address bus, and a control bus.

[0334] The electronic device 1400 can communicate with one or more external devices 1500 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 1440 .

[0335] The electronic device 1400 can communicate with one or more networks through the network adapter 1450. For example, the network adapter 1450 can provide mobile communication solutions such as 3G / 4G / 5G, or provide wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 1450 can communicate with other modules of the electronic device 1400 through the bus 1430.

[0336] The electronic device 1400 may capture screen image data through the display 1460 .

[0337] although Fig.14 Not shown, other hardware and / or software modules may also be provided in the electronic device 1400, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0338] As can be seen from the above, the technical solution of the present disclosure can be implemented as a method, an apparatus, a system, a computer program product, an electronic device, etc. Those skilled in the art can understand that various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, such as being respectively referred to as a "circuit", "module" or "system".

[0339] It should be understood that the present disclosure is not limited to the specific method steps or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. Those skilled in the art will easily think of other embodiments based on the specific embodiments provided by the present disclosure. Therefore, the specific embodiments provided by the present disclosure are only exemplary, and the scope and spirit of the present disclosure are indicated by the claims, and any variations, uses or adaptive changes of the present disclosure should be covered, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the technical field that are not disclosed in the present disclosure.

Claims

1. A data processing method, characterized in that: Applied to a terminal device including a screen, the method comprises: Get the capture frequency of screen image data; Taking a screen snapshot of the terminal device according to the capture frequency to obtain the screen image data; Acquire an application associated with the screen image data, and acquire running status data of the associated application; The screen image data and the running status data of the associated application are bound together to generate a running status record of the associated application, and the running status record is stored in the terminal device.

2. The method according to claim 1, characterized in that The associated application of obtaining the screen image data includes: While acquiring the screen image data, an application in a running state in the terminal device is determined as the associated application.

3. The method according to claim 1, characterized in that The obtaining the running status data of the associated application includes: Obtaining the running status information and identification information of the associated application; The identification information of the associated application is bound to the running status information to obtain the running status data of the associated application.

4. The method according to claim 3, characterized in that The obtaining the running status information of the associated application includes: extracting usage data of the associated application from the screen image data; The usage data of the associated application is stored in a specified format to obtain the running status information.

5. The method according to claim 1, characterized in that The step of binding the screen image data and the running status data of the associated application to generate the running status record of the associated application includes: The storage information of the screen image data is written into the running status data of the associated application in a specified format, so as to bind the screen image data with the running status data and obtain the running status record of the associated application.

6. The method according to claim 1, characterized in that The method further comprises: The running status of the associated application is restored through the running status record stored in the terminal device.

7. The method according to claim 6, characterized in that The restoring the running state of the associated application by using the running state record stored in the terminal device includes: In response to receiving the natural language query operation, obtaining the application status query data according to the natural language query content corresponding to the natural language query operation; Retrieving a target running status record matching the application status query data from the running status records stored in the terminal device; Acquire the application to be restored based on the running status data in the target running status record; The running state of the application to be restored is restored according to the screen image data and / or running state data in the target running state record.

8. The method according to claim 7, characterized in that The obtaining the application status query data according to the natural language query content corresponding to the natural language query operation includes: The text errors in the natural language query content are corrected using a large language model, and the corrected natural language query content is subjected to word segmentation processing to obtain the application status query data.

9. The method according to claim 7, characterized in that: The step of retrieving a target running status record matching the application status query data from the running status records stored in the terminal device includes: Acquire the running state vector corresponding to the running state record stored in the terminal device, and acquire the application state query vector corresponding to the application state query data; Performing similarity matching on the running state vector and the application state query vector, and obtaining a target running state vector according to a result of the similarity matching; A target running state record associated with the target running state vector is obtained from the running state record.

10. The method according to claim 9, characterized in that The running status record includes screen image data; and obtaining the running status vector corresponding to the running status record stored in the terminal device includes: Performing image segmentation on the screen image data to extract picture data from the screen image data, and extracting text data associated with the picture data; generating an image vector based on the image data, and generating a text vector based on the text data; The image vector and the text vector are feature fused through an attention mechanism to obtain the running state vector.

11. The method according to claim 9, characterized in that The obtaining the application state query vector corresponding to the application state query data includes: The deep model is used to extract key query information from the application status query data, and the key query information is converted into the application status query vector.

12. The method according to claim 9, characterized in that The obtaining of a target operating state vector according to the result of the similarity matching includes: Selecting a preset number of candidate running state vectors that are most similar to the application state query vector from the similarity matching results; Generate a search result list based on the candidate running state vector; In response to a selection operation on the search result list, the target operating state vector is determined from among the candidate operating state vectors.

13. The method according to claim 9, characterized in that After acquiring the running state vector corresponding to the running state record stored in the terminal device, the method further includes: storing the running state vector in a specified vector database; The performing similarity matching on the running state vector and the application state query vector, and obtaining a target running state vector according to a result of the similarity matching, includes: In the vector database, similarity matching is performed on the application state query vector to obtain a target running state vector according to a result of the similarity matching.

14. The method according to claim 7, characterized in that The running status data includes identification information, and acquiring the application to be restored based on the running status data in the target running status record includes: The application to be restored is determined based on the identification information.

15. The method according to claim 7, characterized in that The application to be restored includes a browser, and restoring the running state of the application to be restored according to the screen image data and / or the running state data in the target running state record includes: Launching the browser; Acquire the network resource address stored in the running status data, and the relative scroll position of the target tab page corresponding to the network resource address displayed in the screen image data; The target tab is opened in the browser based on the network resource address, and the scroll position of the target tab is set according to the relative scroll position to complete the recovery of the browser.

16. The method according to claim 15, characterized in that The target tab page includes a text box, the running status data includes text box visible information of the text box to be restored, and the method further includes: Acquire the display position of the to-be-restored text box in the screen image data; Determine, based on the display position in the text box in the target tab page, a text box to be restored; The visible information of the text box is written into the text box to be restored, so as to restore the browser.

17. The method according to claim 1, characterized in that The acquisition of the capture frequency of the screen image data includes: In response to a setting operation for the interception frequency, an interception frequency of the screen image data is determined.

18. The method according to claim 1, characterized in that The method further comprises: The operation status record in the terminal device is encrypted, and the encrypted operation status record is stored in the terminal device.

19. A data processing device, characterized in that: Applied to a terminal device including a screen, the device comprises: An interception frequency acquisition module is configured to acquire an interception frequency of screen image data; a screen image data acquisition module, configured to take a screen snapshot of the terminal device according to the capture frequency to acquire screen image data; A running status data acquisition module, configured to acquire an associated application of the screen image data, and acquire running status data of the associated application; The running status record storage module is configured to bind the screen image data and the running status data of the associated application, generate the running status record of the associated application, and store the running status record in the terminal device.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 18 is implemented.

21. An electronic device, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 18 by executing the executable instructions.

Citation Information

Cited By

  • Data processing method and device, electronic equipment, storage medium and program product

    CN121980595A