Implementation method and implementation system for backtracking historical memory under Linux system

Abstract

The invention provides an implementation method and an implementation system for backtracking historical memory under a Linux system, and belongs to the technical field of internet application, and the implementation system comprises an operation history recording module, a time axis backtracking module, a content search module, a privacy protection module, a database manager module and an AI capability manager module. According to the implementation method and the implementation system, through content search and operation history backtracking functions, a user can quickly position and utilize the browsed information, so that the working efficiency is improved; according to the privacy protection measure, it is ensured that a user obtains safe experience in the using process, and the power consumption of the system is ensured; the invention also provides a brand-new operation history record and backtracking solution for the Linux operation system, and fills the blank in the prior art.

Description

Technical Field

[0001] The present invention relates to the recall of Internet historical memories, and particularly to a method for implementing the recall of historical memories in a Linux system, belonging to the technical field of Internet applications. Background Art

[0002] With the rapid development of information technology, the application scenarios of computers in daily work and life are increasing day by day. It has become normal for users to browse and process a large amount of data on computers. However, although users have browsed this content before, it has become extremely difficult to accurately recall the specific content of the document or its storage location afterwards. This not only reduces the efficiency of information utilization but also brings a lot of inconvenience to users.

[0003] In the existing Linux operating system, although there are some search applications, these applications mainly search based on file names or metadata and cannot achieve in-depth search for document content. Therefore, when users need to recall historical browsing content, they often face great challenges.

[0004] In the prior art, the Linux operating system lacks a solution that can record user operations and browsing history and assist users in quickly recalling history. Specific problems include: 1. Lack of content search function: Existing search applications cannot search for document content, web pages, pictures, etc., resulting in users having difficulty quickly locating the information they need.

[0005] 2. Insufficient operation history records: The operating system fails to effectively record the operation history of users, making it impossible for users to obtain complete history records when they need to recall.

[0006] 3. System performance and privacy protection: In the prior art, when recording the operation history of users, the system power consumption and user privacy protection are not fully considered, which may lead to a decrease in system battery life and privacy leakage.

[0007] Therefore, developing a solution that can record user operations and browsing history in a Linux system and assist users in quickly recalling history has become an urgent problem to be solved. This solution not only needs to solve the defects in the prior art but also needs to consider system power consumption and protect user privacy to ensure a safe and smooth user experience for users. Summary of the Invention

[0008] In view of the above problems, the present invention provides a method and system for implementing the recall of historical memories in a Linux system to achieve real-time recording of user operation history and improve the search efficiency of historical memories.

[0009] To achieve the above object, the technical solution of the present invention is: a system for realizing historical memory backtracking in a Linux system, including an operation history record module, a timeline backtracking module, a content search module, a privacy protection module, a database manager module, and an AI capability manager module; The operation history record module is connected to the window manager, the database manager module, and the timeline backtracking module. By listening to the window manager signal and taking screenshots of the screen, it realizes real-time recording of the user's operation behavior, stores the screenshots as videos, transfers the activated application information at the time of screenshot to the timeline backtracking module, and stores the screenshot information of the screenshots in the database manager module; The timeline backtracking module is connected to the operation history record module and the database manager module, and is used to receive the activated application information when the operation history record module takes screenshots of the screen, and to obtain past screenshot information from the database manager module. By displaying the icons of the activated applications on the timeline segmented by application, it enables the user to preview the screenshots of the applications; the display of the activated application icons on the timeline means that this activated application has been used during this period, and the longer the timeline of the activated application, the longer the usage time. When the mouse hovers over the timeline, it will also display the specific screenshot, the date and time of the screenshot, and the name of the activated application.

[0010] The content search module is connected to the database manager module, and searches the historical memory through text matching search and image matching search, matches and locates the screenshot information required by the user with the screenshot information in the database manager module, and then obtains the specific screenshot from the video through the screenshot information; the search results of the content search module will also display the screenshot, the screenshot date, time, and application name; The privacy protection module is connected to the window manager and is used to provide the window manager with the application blacklist configured by the user; The database manager module is connected to the operation history record module and is used to store the screenshot information identified by the operation history record module, including a traditional database and a vector database; The AI capability manager module is connected to the operation history record module and the content search module, and provides the following AI capabilities for the operation history record module and the content search module: OCR capability, text vectorization capability, and image vectorization capability.

[0011] Furthermore, the working process of the operation history record module is as follows: (1) Set the threshold of the screen change rate of the window manager, denoted as threshold. When the ratio of the area of the screen content change to the total screen area is greater than threshold, the window manager sends a signal to the operation history record module; (2) Listen for the screen change signal sent by the window manager. If the screen change signal sent by the window manager is received, obtain a screen capture from the window manager; otherwise, continue listening. (3) Calculate the similarity between the latest obtained screen capture and the previously obtained screen capture. If the calculated similarity value is greater than the pre-set similarity threshold similarity, obtain the currently active application; otherwise, continue listening for the signal from the window manager. (4) Determine whether the currently active application is the same as the previously active application. If not, send a signal to notify the timeline backtracking module that the active application has changed. (5) Add the screen capture to the video file. (6) Continue listening for the signal sent by the window manager.

[0012] Further, the process of adding the screen capture to the video file is as follows: (1) Determine whether the capture time of the screen capture is a new day; if so, merge all the screen captures in the memory to the end of the last video of the previous day. (2) Temporarily store the incoming screen capture in the memory. (3) Send a signal notifying the timeline backtracking module of the change in the total number of screen captures for the current day. The signal stores the total number of screen captures for the current day. (4) Determine whether the number of temporarily stored screenshots in the memory is equal to the pre-set maximum number maxFrameNum. (5) If so, merge all the screen captures in the memory into a new video, and name the video with the time of the first screen capture; if not, directly end.

[0013] Further, the timeline backtracking module and the content search module support copying the text content in the screenshot when a specific screenshot is selected.

[0014] Further, the text matching search process is as follows: (1) Set the time range for searching the traditional database during text matching search. (2) Set the number of results to be obtained for the text matching search request, denoted as number. (3) Perform an exact match on the OCR text in the traditional database within the time range set in step (1) from the back to the front to obtain the number of results set in (2). (4) Traverse all the exact match results, and obtain and store the information of the screen captures that are exactly matched. (5) Combine the obtained screen capture information and the video file to obtain the screen capture.

[0015] Furthermore, the image matching search process is as follows: (1) Set the time range for searching the vector database during image matching search; (2) Set the number of results obtained by the image matching search request, denoted as number; (3) Perform text vectorization on the search text to obtain a text vector; (4) Perform similarity matching on the image vectors in the vector database within the time range set in step (1) to obtain the number of results set in (2); (5) Traverse all similarity matching results, obtain and store the screenshot information that is accurately matched; (6) Combine the obtained screenshot information and video file to obtain a screenshot.

[0016] Furthermore, the privacy protection module provides the configured application blacklist to the window manager, and when the window manager provides a screenshot to the memory recall application, it removes the drawing window of the application in the blacklist from the screenshot.

[0017] Further, the conventional database stores the OCR-recognized text information of the screenshot and the screenshot information, wherein the screenshot information includes the name of the video storing the screenshot, the frame number of the screenshot in the corresponding video, the specific time of the screenshot, the name of the activated application in the screenshot, and the frame number of the screenshot on the day; The vector database stores the image vectorized data of the screenshot, the name of the video storing the screenshot, the frame number of the screenshot in the corresponding video, and the specific time of the screenshot.

[0018] Furthermore, the operation history module uses the AI ​​capability manager module to perform the following operations after adding the screenshot to the video file: (1) Perform OCR on the screenshot and store the text recognized by the OCR on the screenshot, the name of the video where the screenshot is stored, the frame number of the video to which the screenshot belongs, the specific time of the screenshot, the application name, and the frame number of the day in a traditional database; (2) Perform image vectorization processing on the screenshot and store the vectorized data, the name of the video where the screenshot is stored, the frame number of the screenshot in the corresponding video, and the specific time of the screenshot into a vector database.

[0019] The implementation method of tracing back historical memory in Linux system using the above implementation system includes the following process: S1. After the program is started, the database manager module loads the data in the database; S2. The operation history record module listens for the screen change signal of the window manager, stores the screen screenshots into a video, and stores the OCR results and image vectorization results of the screen screenshots into a traditional database and a vector database respectively; S3. The timeline backtracking module listens for the signals of new screen screenshots and changed active windows in real time, and updates the timeline and preview screenshot information in real time; S4. The content search module first obtains all the screenshot information from the database manager module, and then obtains the screen screenshots from the video through the screenshot information.

[0020] The beneficial effects of an implementation method and an implementation system for recalling historical memories in a Linux system according to the present invention are as follows: (1) The operation history record module of the present invention records the user's operation behavior by taking screenshots of the screen, and can dynamically adjust the screenshot time interval, balancing the granularity of recording the user's operation behavior and the system power consumption; at the same time, the screen screenshots are synthesized into multiple videos for storage, greatly reducing the memory occupancy and improving the stability of user data.

[0021] (2) A timeline is implemented above the operation interface of the present invention. The timeline is roughly divided by day, and each day is further divided by application. The icons of the active applications are displayed on the timeline divided by application. When the mouse hovers over the timeline, a preview of the screen screenshot will be displayed beside the mouse; when any position on the timeline is clicked with the mouse, the screen screenshot at that position will be selected and displayed at the center position of the application, and then the OCR technology can be used to recognize the text in any area of the screenshot and copy it to the clipboard.

[0022] (3) The content search module of the present invention consists of two parts, one is text matching search, and the other is image matching search. The text matching search finds the screen screenshots in which the search text appears in the historical memories by means of exact matching, and the image matching search finds the screen screenshots with a relatively high similarity to the content described by the search text in the historical memories by means of similarity matching.

[0023] (4) In order to protect the privacy of users, the present invention sets up an application blacklist that can be configured by users. When obtaining screenshots from the window manager, the window manager can remove the drawing windows of the applications in the blacklist from the screen screenshots; and users can independently choose to store and delete historical records, and can temporarily pause the memory function. Brief Description of the Drawings

[0024] The present invention will be further described in detail below with reference to the drawings and specific embodiments.

[0025] Figure 1 It is the work flow chart of the operation history record module of the present invention; Figure 2 It is a flowchart of the operation history record module of the present invention adding a screen capture to a video file; Figure 3 It is a schematic diagram of the timeline backtracking module of the present invention previewing and displaying a screen capture on the timeline and centering and displaying a specific screen capture; Figure 4 It is a flowchart of text matching search of the present invention; Figure 5 It is a flowchart of image matching search of the present invention. Specific embodiments

[0026] Next, in combination with the accompanying drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Many specific details are set forth in the following description in order to fully understand the present invention, but the present invention may also be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without departing from the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0027] Related terms 1. Traditional database: A traditional database refers to a common general database, such as sqlite, mysql, etc.

[0028] 2. Vector database: A vector database is a database system specifically used for storing and retrieving high-dimensional vectors. A vector is a point in a multi-dimensional space and can be used to represent various data, such as text, images, audio, etc. In this database, data is transformed into vector form for efficient similarity search and retrieval.

[0029] 3. Text vectorization: Text vectorization is the process of transforming text data into mathematical vectors. The purpose of text vectorization is to represent text in a form that can be processed by machine learning models, and these vectors capture the semantic information and features in the text. The result of vectorization is a multi-dimensional digital vector, and each dimension represents a certain feature or semantic information of the text.

[0030] 4. Image vectorization: Image vectorization is the process of transforming image data into mathematical vectors. This transformation changes the image from a pixel-level representation to a high-dimensional vector form, enabling the computer to process the image in a numerical space. The purpose of image vectorization is to extract the features of the image so that it can be effectively processed and analyzed by machine learning models.

[0031] 5. Similarity search: Similarity search refers to finding the vector most similar to a given query vector in a set of vectors by calculating the similarity between vectors. This process usually involves comparing vectors in a high-dimensional space and is achieved through some distance metrics or similarity metrics.

[0032] 6. Text matching search: Text matching search refers to the exact matching of the search text. By searching for the text in the OCR text of the screenshot, it is determined whether the screenshot is the one being sought.

[0033] 7. Image matching search: Image matching search refers to the similarity matching of the search text, and then a similarity search is performed between the vectorized result of the search text and the vectorized set of the screenshot to find the screenshot with the highest similarity.

[0034] 8. OCR: OCR is the abbreviation of "Optical Character Recognition", which is usually translated as "Optical Character Recognition" in Chinese. OCR technology is a technology that converts the text in an image into editable text.

[0035] 9. Window Manager: The Window Manager is responsible for managing and controlling operations such as the creation, display, layout, focus switching, size adjustment, and closing of windows. It provides a visual framework in the graphical user interface for effectively managing and operating multiple windows. Example 1

[0036] An implementation system for recalling historical memories in a Linux system, including an operation history record module, a timeline recall module, a content search module, a privacy protection module, as well as a database manager module and an AI capability manager module. Among them: The operation history record module is connected to the window manager, the database manager module, and the timeline recall module. It realizes real-time recording of the user's operation behavior by listening to the window manager signal and taking screenshots of the screen, stores the screenshots as a video, passes the activated application information at the time of screenshot to the timeline recall module, and stores the screenshot information of the screenshots in the database manager module; The timeline recall module is connected to the operation history record module and the database manager module, and is used to receive the activated application information when the operation history record module takes screenshots of the screen, and is used to obtain past screenshot information from the database manager module, and by displaying the icons of the activated applications on the timeline segmented by application, enables the user to quickly preview the screenshots of the applications they are interested in through the timeline; The content search module is connected to the database manager module, and through text matching search and image matching search, deeply searches the content such as documents and web pages browsed by the user, matches with the screenshot information in the database manager module, helps the user quickly locate the required screenshot information, and then obtains the specific screenshot from the video through the screenshot information; The privacy protection module is connected to the window manager to provide the window manager with a user-configurable application blacklist to ensure that the applications in the blacklist will not be recorded by the operation history module, thereby ensuring the security of the user's privacy data; The database manager module is connected to the operation history module and is used to store the screenshot information identified by the operation history module.

[0037] The database manager module includes a traditional database and a vector database. The traditional database is used to store the text information recognized by OCR of the screenshot and some other information of the screenshot. The vector database is used to store the image vectorized data of the screenshot, the name of the video storing the screenshot, the frame of the screenshot in the video, and the specific time of the screenshot. The AI ​​capability manager module is connected to the operation history module and the content search module, and is responsible for providing the operation history module and the content search module with necessary AI capabilities, such as OCR capability, text vectorization capability, and image vectorization capability. Example 2

[0038] Based on Example 1, Figure 1 The specific functions and workflow of the operation history module are detailed as follows: The operation history module can record the user's operation behavior in real time. All the user's operation behaviors on the computer can be recorded, such as: operating applications, opening files, browsing web pages, editing documents, etc. The operation history module does not record all the minor changes on the screen, but only records when the screen change rate is large and exceeds a certain time interval. For example: when the screen is locked and the screen is static, no screenshots will be taken, thereby reducing system power consumption; the time interval can be dynamically adjusted according to the computer's hardware performance. In order to have a better user experience, it can be set to 1-5 seconds. This design strategy effectively balances the relationship between the level of detail of the operation history and system performance.

[0039] The operation history module records user operation behaviors by taking screenshots of the screen. The operation history module listens to the screen content change signal sent by the window manager. When the screen content change rate is greater than the set threshold, it will send a signal to the application, and then the application will obtain the screenshot from the window manager, ensuring that the operation history module does not need to frequently operate to actively obtain screenshots, but only takes screenshots when they are needed. The specific workflow is as follows: (1) Set the threshold of the window manager screen change rate, denoted as threshold. When the ratio of the area of ​​the screen content change to the total screen area is greater than the threshold, the window manager sends a signal to the operation history module. (2) Listen for the screen change signal sent by the window manager. If the screen change signal sent by the window manager is received, obtain a screen capture from the window manager; otherwise, continue listening. (3) Calculate the similarity between the latest obtained screen capture and the previously obtained screen capture. If the calculated similarity value is greater than the pre-set similarity threshold similarity, obtain the currently active application; otherwise, continue listening for the signal from the window manager. (4) Determine whether the currently active application is the same as the previously active application. If not, send a signal to notify the timeline backtracking module that the active application has changed. (5) Add the screen capture to the video file and store the screen capture information in the database manager module. (6) Continue listening for the signal sent by the window manager.

[0040] The operation history record module stores the screen captures as a video, which can reduce the memory occupancy of the operation history record module. To ensure that the screen captures are quickly synthesized into a video file and that the corruption of a part of the video file does not affect other video files, the operation history record module synthesizes the screen captures of one day into multiple video files. Whenever the number of temporary screen captures in the memory reaches 100, all the screen captures will be synthesized into a video, and the video name is named after the time of the first screen capture, with the naming format of XXXX-XX-XX_XX-XX-XX.mp4, for example: 2024-11-30_16-41-05.mp4. In addition, when the application is closed, even if the number of screenshots is less than 100, the temporary screen captures stored in the memory will also be synthesized into a video file. Combined Figure 2 As shown, the specific process of adding the screen capture to the video file is as follows: (1) Determine whether the capture time of the screen capture is a new day; if so, merge all the screen captures in the memory into the end of the last video of the previous day. (2) Temporarily store the incoming screen capture in the memory. (3) Send a signal notifying the timeline backtracking module of the change in the total number of screen captures for the current day. The signal stores the total number of screen captures for the current day. (4) Determine whether the number of screenshots temporarily stored in the memory is equal to the pre-set maximum number maxFrameNum. (5) If so, synthesize all the screen captures in the memory into a new video, and the video name is named after the time of the first screen capture, with the naming format of XXXX-XX-XX_XX-XX-XX.mp4, for example: 2024-11-30_16-41-05.mp4, and then end. If not, directly end. Embodiment 3

[0041] Based on Example 2, Figure 3 Further refine the specific functions and workflow of the timeline backtracking module: The timeline backtracking module implements a timeline that is roughly divided by day, and the time of each day is further divided by the granularity of the application. The icons of the activated applications are displayed on the timeline divided by application, such as Figure 3 A, B, C in the timeline, so that users can quickly preview the screenshots of the application they are interested in. Figure 3 As shown in the figure, when the mouse hovers over the timeline, the screenshot will be previewed next to the mouse; click anywhere on the timeline with the mouse to select the screenshot at that location and display it in the center of the application.

[0042] The timeline backtracking module supports copying text content such as files and web pages. When the screenshot is displayed in the center of the application, you can use OCR technology to recognize the text in any area of ​​the screenshot and copy it to the clipboard. Example 4

[0043] Based on Example 3, Figure 4 and Figure 5 Further refine the specific functions and workflow of the content search module: The content search module consists of two parts: one is text matching search and the other is image matching search. Text matching search finds screenshots in historical memory that have appeared in the search text by exact matching. Image matching search finds screenshots in historical memory that have a high degree of similarity with the search text description by similarity matching.

[0044] Among them, Figure 4 As shown, the text matching search process is as follows: (1) Set the time range for searching traditional databases when performing text matching searches; (2) Set the number of results obtained by the text matching search request, denoted as number; (3) Perform an accurate match from back to front on the OCR text in the traditional database within the time range set in step (1) to obtain the number of results set in (2); the number of results actually obtained may be less than the number of results requested; (4) Traverse all exact matching results, obtain and store the exact matching screenshot information: store the video name of the screenshot, the frame number of the screenshot in the video, the specific time of the screenshot, the name of the activated application in the screenshot, and the frame number of the screenshot on that day; (5) Obtain a screenshot by combining the obtained screenshot information and the video file; the screenshot information includes the name of the video to which the screenshot belongs and the frame number of the screenshot in the video to which it belongs. Based on this information, the specific screenshot can be obtained.

[0045] As Figure 5 shown, the image matching search process is as follows: (1) Set the time range of the search vector database during image matching search; (2) Set the number of results obtained by the image matching search request, denoted as number; (3) Perform text vectorization processing on the search text to obtain a text vector; (4) Perform similarity matching on the image vectors in the vector database within the time range set in step (1) to obtain the number of results set in (2); the actual number of results obtained may be less than the number of results desired to be obtained; (5) Traverse all similarity matching results, obtain and store the screenshot information that is exactly matched: the name of the video storing the screenshot, the frame number of the screenshot in the video to which it belongs, and the specific time of the screenshot; (6) Obtain a screenshot by combining the obtained screenshot information and the video file; the screenshot information includes the name of the video to which the screenshot belongs and the frame number of the screenshot in the video to which it belongs. Based on this information, the specific screenshot can be obtained. Embodiment 5

[0046] On the basis of Embodiment 4, further refine the specific functions and implementation methods of the privacy protection module: To protect the privacy of users, the present invention sets up an application blacklist that can be configured by users, and the applications in the blacklist will not be recorded by the operation history record module. The privacy protection module provides the application blacklist configured by the user to the window manager, and when the window manager provides a screenshot to the timeline backtracking module (the screenshot is provided to the operation history record module, and the operation history record module then passes the screenshot to the timeline backtracking module. The timeline backtracking module does not only provide time information on the video of the screenshot. It is similar to the timeline when watching a video and can quickly preview historical memories), the drawing windows of the applications in the blacklist can be removed from the screenshot, which can ensure that privacy is not recorded. For example, communication applications, enterprise internal applications, etc. can be added to the blacklist, ensuring that privacy data is avoided from being obtained at the data source. If the user configures an application blacklist, the privacy protection module will first send the blacklist to the window manager at the start of its operation, and then the operation history record module in Embodiment 2 sets the threshold of the screen change rate of the window manager and performs subsequent operation steps.

[0047] Users can also choose to store and delete history records on their own. If users enter some plaintext passwords on the web page, the memory function can be temporarily suspended. If there is browsing privacy data in the past period of time, users can also delete the past hour, past day, past week, past month and all data through the application's settings interface. Example 6

[0048] Based on Example 5, the specific functions and implementation methods of other auxiliary modules are refined: Database Manager Module: It consists of two parts: one is the traditional database, which is used to store the OCR-recognized text information of the screenshot and some other information of the screenshot, such as: the name of the video of the screenshot, the frame of the screenshot in the video, the specific time of the screenshot, the application activated in the screenshot, and the frame of the screenshot on the day. This information can be used to quickly find the application information and screenshot data we need when recalling the memory. The other part is the vector database, which is used to store the image vectorization data of the screenshot, the name of the video of the screenshot, the frame of the screenshot in the video, and the specific time of the screenshot; Traditional databases store more information items. You can use the information in traditional databases and video files to obtain any screenshot and detailed data of the screenshot. Traditional databases provide data for both timeline preview and text search; vector databases only store necessary information and can only provide data for image search.

[0049] AI Capability Manager Module: The AI ​​capability manager module is responsible for providing the necessary AI capabilities for applications, such as OCR capabilities, text vectorization capabilities, and image vectorization capabilities.

[0050] The specific application of the AI ​​Capability Manager Module Assisted Operation History Module refers to performing the following steps after adding the screenshot to the video file: (1) Perform OCR on the screenshot and store the text recognized by the OCR on the screenshot, the name of the video where the screenshot is stored, the frame number of the video to which the screenshot belongs, the specific time of the screenshot, the activated application in the screenshot, and the frame number of the day in the screenshot into a traditional database; (2) Perform image vectorization processing on the screenshot and store the vectorized data, the name of the video where the screenshot is stored, the frame number of the screenshot in the corresponding video, and the specific time of the screenshot into a vector database.

[0051] The specific application of the AI ​​capability manager assisted content search module refers to the records of the above-mentioned embodiment 4. Example 7

[0052] Based on the system implemented in the above embodiments, the specific process of implementing historical memory retrieval in the Linux system is as follows: S1. After the program starts, the database manager module loads the data in the database; S2. The operation history record module listens for the screen change signal of the window manager, stores the screen capture in the video when necessary, and stores the OCR result and the image vectorization result of the screen capture in the traditional database and the vector database respectively; S3. The timeline retrieval module listens for the signals of new screen captures and active window changes in real time, and updates the timeline and preview screen capture information in real time; S4. The content search module first obtains all the screen capture information from the database manager module, and then obtains the screen captures from the video through the screen capture information.

[0053] Among them, the process of S4 is only executed when the user searches.

[0054] In summary, the technical solution of the present invention has achieved the following technical improvements: 1. Improved information utilization efficiency: Through the content search and operation history retrieval functions, users can quickly locate and utilize the information they have browsed before, improving work efficiency.

[0055] 2. Enhanced user experience: Privacy protection measures ensure that users obtain a secure experience during use and ensure the power consumption of the system.

[0056] 3. Filled a technical gap: Provided a new operation history record and retrieval solution for the Linux operating system, filling the gap in the existing technology.

[0057] Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

Claims

1. An implementation system for recalling historical memories in the Linux system, characterized in that, It includes an operation history module, a timeline backtracking module, a content search module, a privacy protection module, a database manager module, and an AI capability manager module; The operation history module is connected to the window manager, the database manager module, and the timeline backtracking module. It realizes real-time recording of the user's operation behavior by listening to the window manager signal and taking screenshots of the screen, stores the screenshots as a video, passes the active application information at the time of screenshot to the timeline backtracking module, and stores the screenshot information of the screenshots in the database manager module; The timeline backtracking module is connected to the operation history module and the database manager module. It is used to receive the active application information when the operation history module takes screenshots of the screen, and to obtain the past screenshot information from the database manager module, and enables the user to preview the screenshots of the application by displaying the icons of the active application on the timeline segmented by application; The content search module is connected to the database manager module. It searches the historical memory through text matching search and image matching search, matches and locates the screenshot information required by the user with the screenshot information in the database manager module, and then obtains the specific screenshot from the video through the screenshot information; The privacy protection module is connected to the window manager and is used to provide the application blacklist configured by the user to the window manager; The database manager module is connected to the operation history module and is used to store the screenshot information identified by the operation history module, including a traditional database and a vector database; The AI capability manager module is connected to the operation history module and the content search module and provides the following AI capabilities for the operation history module and the content search module: OCR capability, text vectorization capability, and image vectorization capability.

2. The implementation system for recalling historical memories in the Linux system according to claim 1, characterized in that, The working process of the operation history module is as follows: (1) Set the threshold of the screen change rate of the window manager, denoted as threshold. When the ratio of the area of the screen content change to the total screen area is greater than threshold, the window manager sends a signal to the operation history module; (2) Listen and wait for the screen change signal sent by the window manager. If the screen change signal sent by the window manager is received, obtain the screenshot from the window manager; otherwise, continue to listen; (3) Calculate the similarity between the latest obtained screenshot and the previous obtained screenshot. If the calculated similarity value is greater than the pre-set similarity threshold similarity, obtain the current active application; otherwise, continue to listen to the signal of the window manager; (4) Judge whether the current active application is the same as the previous active application. If not, send a signal to notify the timeline backtracking module that the active application has changed; (5) Add the screenshot to the video file; (6) Continue to listen to the signal sent by the window manager.

3. The implementation system for recalling historical memories in the Linux system according to claim 2, characterized in that, The process of adding the screenshot to the video file is as follows: (1) Determine whether the screenshot time is a new day; if so, merge all screenshots in the memory into the end of the last video of the previous day; (2) Temporarily store the incoming screenshot in memory; (3) Send a signal indicating the change in the total number of screenshot frames of the day to notify the timeline backtracking module. The signal stores the total number of screenshots of the day. (4) Determine whether the number of screenshots temporarily stored in the memory is equal to the preset maximum number maxFrameNum; (5) If yes, combine all the screenshots in the memory into a new video, and name the video after the time of the first screenshot; if no, then end directly.

4. The implementation system for recalling historical memories in the Linux system according to claim 1, characterized in that, The timeline backtracking module and the content search module support copying the text content in the screenshot when a specific screenshot is selected.

5. The implementation system for recalling historical memories in the Linux system according to claim 1, characterized in that, The text matching search process is as follows: (1) Set the time range for searching traditional databases when performing text matching searches; (2) Set the number of results obtained by the text matching search request, denoted as number; (3) Perform an accurate match from back to front on the OCR text in the traditional database within the time range set in step (1) to obtain the number of results set in (2); (4) Traverse all exact matching results, obtain and store the exact matching screenshot information; (5) Combine the obtained screenshot information and video file to obtain a screenshot.

6. The implementation system for recalling historical memories in the Linux system according to claim 1, characterized in that, The image matching search process is as follows: (1) Set the time range for searching the vector database during image matching search; (2) Set the number of results obtained by the image matching search request, denoted as number; (3) Perform text vectorization on the search text to obtain a text vector; (4) Perform similarity matching on the image vectors in the vector database within the time range set in step (1) to obtain the number of results set in (2); (5) Traverse all similarity matching results, obtain and store the screenshot information that is accurately matched; (6) Combine the obtained screenshot information and video file to obtain a screenshot.

7. The implementation system for retrieving historical memories in a Linux system according to claim 1, wherein, The privacy protection module provides the configured application blacklist to the window manager. When the window manager provides a screenshot to the memory recall application, it removes the drawing window of the application in the blacklist from the screenshot.

8. The implementation system for retrieving historical memories in a Linux system according to claim 1, wherein, The traditional database stores OCR-recognized text information of the screenshot and screenshot information, wherein the screenshot information includes the name of the video storing the screenshot, the frame number of the screenshot in the video, the specific time of the screenshot, the name of the activated application in the screenshot, and the frame number of the screenshot on the day; The vector database stores the image vectorized data of the screenshot, the name of the video storing the screenshot, the frame number of the screenshot in the corresponding video, and the specific time of the screenshot.

9. The implementation system for retrieving historical memories in a Linux system according to claim 2, wherein, The operation history module uses the AI ​​capability manager module to perform the following operations after adding the screenshot to the video file: (1) Perform OCR recognition on the screen capture, and store the text recognized by the OCR of the screen capture, the name of the video where the capture is stored, the frame number of the capture in the video to which it belongs, the specific time of the screen capture, the application name, and the frame number of the day in the traditional database; (2) Perform image vectorization processing on the screen capture, and store the vectorized data, the name of the video where the capture is stored, the frame number of the capture in the video to which it belongs, and the specific time of the screen capture in the vector database.

10. The implementation method for retrieving historical memories in a Linux system by using the implementation system according to any one of claims 1-9, wherein, The process includes the following: S1. After the program starts, the database manager module loads the data in the database; S2. The operation history record module listens to the screen change signal of the window manager, stores the screen capture in the video, and stores the OCR result and image vectorization result of the screen capture in the traditional database and the vector database respectively; S3. The timeline backtracking module listens to the signals of new screen captures and changes in the active window in real time, and updates the timeline and preview capture information in real time; S4. The content search module first obtains all the capture information from the database manager module, and then obtains the screen capture from the video through the capture information.

Images