Artificial intelligence-based screen recording system and electronic device

Abstract

The present invention provides a screen recording system and electronic device based on artificial intelligence, and relates to the field of screen recording technology. The screen recording system includes: a data processing module, which collects screen information and audio information from an electronic device, and processes the screen information and audio information to obtain screen data and audio data; a data storage module, which locally stores screen data and audio data and establishes an index; an artificial intelligence processing module, which identifies and processes screen data and audio data based on a preset artificial intelligence model, and responds to user query instructions according to the index; a plug-in processing module, which responds to user instructions to set a custom function plug-in to generate a target media pipeline; a media pipeline processing module, which responds to user instructions and processes and converts the data stream of screen data and audio data based on the target media pipeline. The present invention has the characteristics of high privacy security, strong scalability, high artificial intelligence integration, and good adaptability to electronic device terminals.

Description

Technical Field

[0001] The present invention relates to the technical field of screen recording, and in particular to an artificial intelligence-based screen recording system and electronic device. Background Art

[0002] With the development of artificial intelligence technology, it is widely used in various application fields. For example, the intelligent screen recording system is an artificial intelligence-based recording system installed on electronic devices that can automatically perform multiple recording tasks.

[0003] The intelligent screen recording system can automatically start and stop recording information based on user activity, intelligently capture only relevant screen actions, and automatically edit recording content. These features allow users to efficiently complete screen recording tasks on electronic devices without manual intervention.

[0004] However, the inventors found that the current intelligent screen recording system has problems such as poor privacy security, insufficient functional scalability, poor artificial intelligence integration, and high requirements for the configuration performance of electronic devices. Summary of the Invention

[0005] The present invention provides an artificial intelligence-based screen recording system and electronic device, which are used to solve the problems of current intelligent screen recording systems, such as poor privacy security, insufficient functional scalability, poor artificial intelligence integration, and high requirements on the configuration performance of the electronic device.

[0006] One aspect of the present invention provides an artificial intelligence-based screen recording system. The screen recording system includes a data processing module, a data storage module, an artificial intelligence processing module, a plug-in processing module, and a media pipeline processing module. The data processing module collects screen information and audio information from the electronic device and processes the screen information and audio information to obtain screen data and audio data; the data storage module locally stores the screen data and audio data and creates an index; the artificial intelligence processing module identifies and processes the screen data and audio data based on a preset artificial intelligence model and responds to user query instructions based on the index; the plug-in processing module responds to user instructions to set a custom function plug-in to generate a target media pipeline; and the media pipeline processing module responds to user instructions and processes and converts the data streams of the screen data and audio data based on the target media pipeline.

[0007] According to some embodiments of the present invention, the media pipeline processing module responds to the user's pipeline download instruction and downloads the pipeline file of the target media pipeline; the media pipeline processing module saves the pipeline file to the local data directory and creates and saves the corresponding pipeline information to the database.

[0008] According to some embodiments of the present invention, the media pipeline processing module responds to the user's pipeline activation instruction and reads the pipeline information from the database; the media pipeline processing module sets the status of the target media pipeline, creates the object file of the target media pipeline based on the pipeline information, and adds the object file to the object file collection.

[0009] According to some embodiments of the present invention, the media pipeline processing module responds to a user's pipeline update instruction and reads pipeline information from a database; the media pipeline processing module updates the configuration of the target media pipeline based on the pipeline information and saves the updated pipeline information to the database.

[0010] According to some embodiments of the present invention, when a target media pipeline is loaded, the media pipeline processing module traverses all enabled target media pipelines, creates object files of the target media pipeline, and adds the object files to the object file collection.

[0011] According to some embodiments of the present invention, when running a target media pipeline, the media pipeline processing module determines the execution result of the automated task-type pipeline after the object file of the automated task-type pipeline in the object file collection is completed; the media pipeline processing module sends the execution result of the automated task-type pipeline to the backend service.

[0012] According to some embodiments of the present invention, when running the target media pipeline, the media pipeline processing module determines the execution result of the request-response pipeline after the object file of the request-response pipeline in the object file collection is completed; the media pipeline processing module sends the execution result of the request-response pipeline to the front-end service.

[0013] According to some embodiments of the present invention, after the data processing module determines the preset audio acquisition device, it creates a speech recognition channel to collect audio information; the artificial intelligence processing module also performs voice activity detection processing, as well as encoding, speech transcription and decoding processing of the audio information to obtain audio data.

[0014] According to some embodiments of the present invention, the data processing module reads a preset configuration file and, after initializing the data processing engine, continuously obtains screen information of the electronic device; the artificial intelligence processing module recognizes text information in the screen information to obtain screen data.

[0015] Another aspect of the present invention provides an electronic device, which includes the screen recording system described above.

[0016] The screen recording system provided by the present invention includes a data processing module, a data storage module, an artificial intelligence processing module, a plug-in processing module, and a media pipeline processing module. The system can continuously collect screen and audio information from an electronic device and analyze and process the collected recording information based on the artificial intelligence processing module. Furthermore, the system can respond to user instructions to set custom function plug-ins through the plug-in processing module and process and convert data streams based on a target media pipeline through the media pipeline processing module in response to user instructions.

[0017] By adopting a localized data storage and processing method, the present invention can avoid the risk of uploading user data on the electronic device to the cloud (or a third-party server), thereby protecting the user's privacy and data security; the present invention can also meet different application scenarios and meet the user's customization requirements, thereby increasing the scalability of the screen recording system and reducing the user's development cost and operation difficulty; the present invention can also realize intelligent content indexing, query and automated operation based on artificial intelligence; the present invention can also automatically select the most suitable intelligent processing engine according to different operating system platforms, etc., which can ensure the processing performance of intelligent processing and has high adaptability to electronic devices with different operating systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic structural diagram of a screen recording system according to an embodiment of the present invention is shown;

[0020] Figure 2 A schematic diagram illustrating the architecture of a screen recording system according to an embodiment of the present invention is shown;

[0021] Figure 3 A flowchart illustrating audio data collection according to an embodiment of the present invention is shown;

[0022] Figure 4 A flowchart showing the collection of screen data according to an embodiment of the present invention is shown.

[0023] Description of reference numerals:

[0024] Screen recording system 1; data processing module 10; data storage module 20; artificial intelligence processing module 30; plug-in processing module 40; media pipeline processing module 50. DETAILED DESCRIPTION

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Like reference numerals in the drawings represent like or similar parts, and thus repeated description thereof will be omitted.

[0026] The described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of these specific details, or other methods, components, materials, devices, etc. may be employed. In these cases, well-known structures, methods, devices, implementations, materials or operations will not be shown or described in detail.

[0027] Furthermore, the terms "include," "comprise," and "have," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0028] The terms "first", "second" and the like in the specification, claims and drawings of the present invention are used to distinguish different objects rather than to describe a specific order.

[0029] The following provides a clear and complete description of the technical solutions of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by those skilled in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0030] The full names of the English abbreviations involved in this invention and their Chinese meanings are:

[0031] PII: Personally Identifiable Information, personally identifiable information;

[0032] OCR: Optical Character Recognition, optical character recognition;

[0033] VAD: Voice Activity Detection, voice activity detection;

[0034] TypeScript: open source programming language;

[0035] Whisper: a general-purpose speech recognition model.

[0036] SQLite: lightweight database;

[0037] Tauri - open source cross-platform GUI framework.

[0038] React components: the core building blocks of React applications;

[0039] IPC: Inter Process Communication, a technology used to exchange data between different processes;

[0040] Rust - a systems-level programming language.

[0041] Python: an open source programming language;

[0042] TypeScript: an open source programming language;

[0043] Uniform Resource Locator: Uniform Resource Locator.

[0044] The inventors of the present invention have discovered that the current intelligent screen recording system still has at least the following problems:

[0045] 1. The data processing platform stores data in the cloud, which increases the risk of data leakage and also causes higher latency in remote interfaces when data is transmitted remotely.

[0046] 2. It is difficult to expand and customize functions according to user needs, resulting in insufficient functional scalability.

[0047] 3. It is difficult to implement artificial intelligence-based data processing and analysis, and it is impossible to directly search and locate the time of an event based on screen or sound content.

[0048] 4. Graphics card configuration is required during deployment, and cluster deployment is required, which places high requirements on the configuration performance of the electronic equipment side.

[0049] Based on this, according to one aspect of the present invention, the present invention provides a screen recording system based on artificial intelligence. Figure 1 A schematic diagram of the structure of the screen recording system according to an embodiment of the present invention is shown. Figure 1 As shown, the screen recording system 1 includes a data processing module 10, a data storage module 20, an artificial intelligence processing module 30, a plug-in processing module 40 and a media pipeline processing module 50.

[0050] According to an exemplary embodiment, the screen recording system 1 can be provided on an electronic device to record screen information of the electronic device. For example, the electronic device includes but is not limited to a computer, a notebook, or a mobile phone, etc., which are electronic devices for user operation.

[0051] The data processing module 10 collects screen information and audio information from the electronic device, and processes the screen information and audio information to obtain screen data and audio data.

[0052] For example, the data processing module 10 can continuously collect multimedia information generated by the electronic device, such as screen information and audio information, and perform preliminary data processing on the screen information and audio information in real time, thereby obtaining processed screen data and audio data.

[0053] Exemplarily, data processing includes but is not limited to data processing operations such as compression and format conversion.

[0054] According to an exemplary embodiment, the data storage module 20 locally stores screen data and audio data, and creates indexes.

[0055] For example, the data storage module 20 stores the processed screen data and audio data in the local device.

[0056] By adopting a localized data storage and processing method, the present invention can avoid the risk of uploading user data on the electronic device to the cloud (or a third-party server), thereby protecting the user's privacy and data security.

[0057] Optionally, the data processing module 10 can also automatically remove the personally identifiable information (PII) of the user of the electronic device. With this configuration, the present invention can further enhance the privacy and security of user data.

[0058] For example, the data processing module 10 can customize filtering rules based on a preset rule engine to filter user PII information (such as ID card information, email address information, and address information). For example, if data containing sensitive information appears, the data processing module 10 overwrites the data (e.g., by adding an asterisk to cover the sensitive information) and then stores the data in the database.

[0059] As an example, the data processing module 10 loads parameter configurations (such as frame rate settings, audio and video segment durations, and enabled engines), then initiates a main loop to continuously collect screen and audio information. For each video frame, the data processing module 10 captures the screen content and extracts OCR text data from the video frame using optical character recognition (OCR). The data storage module 20 inserts the video frame and OCR text data into a database and then stores the video frame and OCR text information in a database (e.g., SQLite).

[0060] The data processing module 10 collects audio information through a preset audio device, identifies speech segments based on VAD, encodes, transcribes, and decodes the audio information, and then stores the processed audio data in a database (such as SQLite).

[0061] According to an example embodiment, the data storage module 20 may also create an index.

[0062] For example, the data storage module 20 responds to user instructions and establishes a multi-level index architecture, including a time series data index layer, a content type index layer, and a full-text search index layer. The time series data index layer can establish a primary index based on timestamps, allowing for rapid location of data within a specific time range. The time series data index layer has a tree structure that allows hierarchical management of time ranges for fast range queries and supports incremental updates, allowing for rapid addition of new data to the index.

[0063] The content type indexing layer can distinguish different types of data, such as OCR text data, audio transcription data, and video frames, etc. It can create independent sub-indexes for each type of data and optimize query functions for specific types of data.

[0064] The full-text search index layer can build an inverted index based on data (such as OCR text data, audio transcription data, and video frames), and can perform fuzzy matching and keyword searches.

[0065] According to an exemplary embodiment, the artificial intelligence processing module 30 performs recognition processing on the screen data and the audio data based on a preset artificial intelligence model, and responds to the user query instruction according to the index.

[0066] For example, the artificial intelligence processing module 30 can call the corresponding AI model preset in the screen recording system 1 according to the preset configuration and user needs. For example, it can intelligently import and use an AI model adapted to the corresponding hardware requirements based on the memory size of the electronic device. In this way, the present invention can optimize the utilization of system resources and ensure processing efficiency and accuracy under various hardware configurations.

[0067] Exemplarily, AI models include but are not limited to speech recognition processing (used for speech recognition, speech-to-text, etc.), computer vision (used for OCR optical recognition, target detection, etc.), natural language processing (used for question-answering processing), etc., and the present invention does not impose any restrictions on this.

[0068] The artificial intelligence processing module 30 can perform intelligent processing on the screen data and audio data stored in the data storage module 20 according to the called AI model. For example, the intelligent processing includes but is not limited to OCR processing, speech-to-text processing, etc., which are not limited in the present invention.

[0069] The artificial intelligence processing module 30 can automatically select the most suitable OCR processing engine and VAD voice processing engine according to different operating system platforms (such as Windows, MacOS, Linux and other operating system platforms), so as to ensure the processing performance of the intelligent processing of the artificial intelligence processing module 30.

[0070] According to an exemplary embodiment, the artificial intelligence processing module 30 searches for matching content in a database in response to a user query. For example, the artificial intelligence processing module 30 server searches a SQLite database for content (e.g., video frame data, OCR text data, audio transcription data, etc.) that matches the user query and returns the relevant content to the user as query results.

[0071] According to an example embodiment, the plug-in processing module 40 sets a custom function plug-in in response to a user instruction to generate a target media pipeline.

[0072] For example, a media pipeline is a chain of multiple media elements that can be used to process and transmit media data. The target media pipeline can be customized according to user needs.

[0073] For example, the plug-in processing module 40 can generate a target media pipeline based on a user-defined functional plug-in according to user programming instructions based on TypeScript, so that the plug-in processing module 40 can execute the plug-in logic customized by the user.

[0074] Exemplarily, the target media pipeline generated by the custom function plug-in may include:

[0075] Pipeline Meeting Minutes Plugin: Used to manage meeting minutes and summaries.

[0076] Pipeline Mail Log Plugin: Manages emails and sends them to a daily activity log.

[0077] Pipeline Screen to Customer Management System Plugin: used to integrate screen data with customer management systems.

[0078] Pipeline OCR text stream plugin: used to stream OCR text data from screen data.

[0079] Pipeline Focus Notification Assistant Plugin: used for focus reminders, which can be used for online learning or online exams, etc.

[0080] Pipeline safety inspection plug-in: used to issue an alarm when suspicious activities are detected.

[0081] Pipeline Time Assistant plug-in: used to save or search user historical data.

[0082] With such a configuration, the present invention can meet different application scenarios and meet the customized requirements of users, thereby increasing the scalability of the screen recording system and reducing the development cost and operation difficulty of users.

[0083] According to an exemplary embodiment, the media pipeline processing module 50 processes and converts data streams of screen data and audio data based on a target media pipeline in response to a user instruction.

[0084] For example, the media pipeline processing module 50 flexibly processes the data streams of the screen data and the audio data based on the user's instruction and the custom type of the target media pipeline.

[0085] The screen recording system provided by the present invention includes a data processing module, a data storage module, an artificial intelligence processing module, a plug-in processing module, and a media pipeline processing module. The system can continuously collect screen and audio information from an electronic device and analyze and process the collected recording information based on the artificial intelligence processing module. Furthermore, the system can respond to user instructions to set custom function plug-ins through the plug-in processing module and process and convert data streams based on a target media pipeline through the media pipeline processing module in response to user instructions.

[0086] By adopting a localized data storage and processing method, the present invention can avoid the risk of uploading user data on the electronic device to the cloud (or a third-party server), thereby protecting the user's privacy and data security; the present invention can also meet different application scenarios and meet the user's customization requirements, thereby increasing the scalability of the screen recording system and reducing the user's development cost and operation difficulty; the present invention can also realize intelligent content indexing, query and automated operation based on artificial intelligence; the present invention can also automatically select the most suitable intelligent processing engine according to different operating system platforms, etc., which can ensure the processing performance of intelligent processing and has high adaptability to electronic devices with different operating systems.

[0087] Figure 2 A schematic diagram illustrating the architecture of a screen recording system according to an embodiment of the present invention is shown.

[0088] As an example, Figure 2 As shown, the data processing module 10 performs screen capture and audio collection according to preset functional plug-ins; the data storage module 20 can store data such as SQL database embedding, screen text with timestamps, user operation history, and audio transcription with timestamps; the artificial intelligence processing module 30 can identify and process the stored data based on index vector search, keyword search, semantic search, classification, labeling, metadata acquisition, etc.; the media pipeline processing module 50 can perform flexible processing of data streams such as using LLM search history, creating / viewing summaries / indicators, triggering time-based operations, etc.

[0089] As an embodiment, the startup process of the screen recording system may include:

[0090] 1. Initialization phase of the user interface built by the Tauri framework: Start the Tauri main process, load the front-end web interface, initialize the front-end React components, and then establish an IPC communication bridge between the front-end and back-end.

[0091] 2. The core initialization phase of the screen recording system: Initialize the Rust backend main process, load the necessary system configuration and environment variables, then initialize the core service components and establish a communication channel with the Tauri frontend.

[0092] 3. Parallel service startup phase: The screen recording system starts three important service threads simultaneously, including:

[0093] Screen recording service, responsible for continuously capturing screen images to obtain screen information and storing screen data in a local cache in real time;

[0094] Audio recording service, responsible for collecting input data from system audio and audio devices (such as microphones), and processing and locally caching the audio data in real time;

[0095] The pipeline processing service is responsible for initializing the plugin system, loading preconfigured processing pipelines, and preparing data processing flows.

[0096] 4. Data processing thread startup phase: Initialize the OCR service, prepare the text recognition engine and configure recognition parameters, initialize the speech-to-text service, load the speech recognition model and establish the audio processing queue, etc.

[0097] 5. Data storage system initialization phase: establish a connection with the local database, initialize the data table structure, and determine the data caching mechanism.

[0098] 6. AI service connection phase: Establish a secure connection with the AI ​​service, verify service authorization, and initialize the AI ​​processing pipeline.

[0099] 7. UI status synchronization phase: Push the startup status of each service to the Tauri front end, activate the user interface interaction function and display the system's readiness status.

[0100] Through the above-described embodiments, the screen recording system provided by the present invention utilizes the Tauri framework to implement front-end interface rendering, and a multi-threaded parallel processing mechanism enables efficient collaboration among back-end components. Furthermore, the screen recording system utilizes an event-driven architecture to achieve real-time communication and state synchronization between the front-end and back-end, providing a smooth operating experience and a stable operating environment for continuous recording and real-time processing of screen and audio data.

[0101] Optionally, after the data processing module 10 determines the preset audio acquisition device, it creates a speech recognition channel to collect audio information. The artificial intelligence processing module 30 also performs voice activity detection processing, as well as encoding, voice transcription, and decoding processing of the audio information to obtain audio data.

[0102] Figure 3 FIG. 1 shows a flow chart of audio data acquisition according to an embodiment of the present invention. Figure 3 As shown, the audio data collection flow chart includes steps S11-S18.

[0103] In step S11 , the data processing module 10 determines available audio acquisition devices.

[0104] In step S12 , the data processing module 10 obtains a default input / output device.

[0105] In step S13 , the data processing module 10 creates a speech recognition channel (eg, a Whisper speech recognition channel).

[0106] In step S14 , the data processing module 10 collects audio information.

[0107] In step S15 , the artificial intelligence processing module 30 performs voice activity detection processing (eg, VAD).

[0108] In step S16 , the artificial intelligence processing module 30 encodes the audio information.

[0109] In step S17 , the artificial intelligence processing module 30 performs speech transcription (eg, using a Whisper model to perform speech transcription).

[0110] In step S18 , the artificial intelligence processing module 30 decodes the audio information, obtains audio data, and stores the audio data (for example, in the data storage module 20 ).

[0111] Figure 4FIG. 1 shows a flow chart of collecting screen data according to an embodiment of the present invention. Figure 4 As shown, the screen data collection flow chart includes steps S21-S25.

[0112] In step S21 , the data processing module 10 reads a preset configuration file.

[0113] In step S22 , the data processing module 10 initializes a data processing engine (eg, an OCR processing engine).

[0114] In step S23 , the data processing module 10 continuously obtains screen information (such as screenshots) of the electronic device.

[0115] In step S24 , the artificial intelligence processing module 30 performs filtering processing on the acquired screen information.

[0116] In step S25 , the artificial intelligence processing module 30 recognizes text information in the screen information based on OCR and processes the text recognition result to obtain screen data.

[0117] As an embodiment, the data collection, storage and search process of the screen recording system can be as follows:

[0118] 1. Data collection stage: The plug-in processing module 40 can be used to collect multi-source data, including real-time capture of screen images (i.e., screen information), collection of system audio information, and recording of interactive events. During the data collection stage, the backend of the screen recording system continuously runs the collection service to ensure continuous data collection (e.g., 24 hours a day).

[0119] 2. Data processing stage: Data processing can include optical character recognition (OCR) of screen images, voice-to-text processing of audio information, data standardization, unified data format, extraction of key information, selection of whether to filter sensitive information based on configuration, and generation of metadata tags.

[0120] 3. Data storage stage: Data storage may include using a local database for storage, storing content classification, original audio and video clips, processed text data, metadata information, and establishing indexes to optimize query efficiency.

[0121] 4. Data search (query) stage: Data search can include providing multi-dimensional query interfaces, time dimension queries, content keyword queries, event type queries, query optimization, caching mechanisms, incremental updates, and fuzzy matching.

[0122] Optionally, the media pipeline processing module 50 responds to the user's pipeline download instruction and downloads the pipeline file of the target media pipeline. The media pipeline processing module 50 saves the pipeline file to the local data directory and creates and saves the corresponding pipeline information to the database.

[0123] For example, the pipeline file can be a Python or TypeScript script that implements a preset function (such as an alarm function or an automatic email function). The media pipeline processing module 50 can respond to the user's pipeline download instruction (i.e., the user requests to download the media pipeline) and download the pipeline file of the target media pipeline from the URL.

[0124] Optionally, the media pipeline processing module 50 responds to the user's pipeline activation instruction and reads pipeline information from the database. The media pipeline processing module 50 sets the state of the target media pipeline, creates an object file of the target media pipeline based on the pipeline information, and adds the object file to the object file collection.

[0125] For example, in response to a user's pipeline activation instruction (i.e., a user's request to activate a media pipeline), the media pipeline processing module 50 may set the target media pipeline's status to enabled. Furthermore, the media pipeline processing module 50 may create an object file (a Future file) for execution of the target media pipeline and then add the Future file to the object file collection (Pipe-Futures).

[0126] Optionally, the media pipeline processing module 50 responds to the user's pipeline update instruction, reads pipeline information from the database, updates the configuration information of the target media pipeline, and saves the updated pipeline information to the database.

[0127] For example, the configuration information of the target media pipeline may be configuration information of some custom variables in the media pipeline, such as alarm threshold information, website information, email address information, monitoring specific object name information, or video capture frame rate information.

[0128] Optionally, when loading a target media pipeline, the media pipeline processing module 50 traverses all enabled target media pipelines, creates an object file of the target media pipeline, and adds the object file to the object file collection.

[0129] For example, after traversing all media pipelines in the enabled state, the media pipeline processing module 50 creates a Future file for pipeline execution, and then adds the Future file to Pipe-Futures.

[0130] Optionally, when running the target media pipeline, the media pipeline processing module 50 determines the execution result of the automated task-based pipeline after the object file of the automated task-based pipeline in the object file set is completed. The media pipeline processing module 50 sends the execution result of the automated task-based pipeline to the backend service.

[0131] Optionally, when the target media pipeline is running, the media pipeline processing module 50 determines the execution result of the request-response pipeline after the object file of the request-response pipeline in the object file set is completed. The media pipeline processing module 50 sends the execution result of the request-response pipeline to the front-end service.

[0132] For example, target media pipelines can include automated task-based pipelines and request-response pipelines. Automated task-based pipelines can be tasks that need to run long-term in the background to collect audio and video information, such as scheduled tasks or monitoring tasks. Request-response pipelines can be tasks that respond to user requests, such as summarizing existing content, processing logs, sending and receiving emails, and searching for problems.

[0133] According to another aspect of the present invention, an electronic device is provided, which includes the artificial intelligence-based screen recording system described above.

[0134] Exemplarily, the electronic device includes but is not limited to a computer, a notebook, a mobile phone or other electronic device for user operation, and the present invention does not impose any limitation on this.

[0135] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions of the aforementioned embodiments or substitute equivalents for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A screen recording system based on artificial intelligence, characterized in that: The screen recording system is arranged at the electronic device end, and includes: A data processing module collects screen information and audio information of the electronic device, and performs data processing on the screen information and the audio information to obtain screen data and audio data; A data storage module, locally storing the screen data and the audio data, and establishing an index, wherein the index establishment includes: responding to user instructions, establishing a multi-level index architecture, wherein the multi-level index architecture includes a time series data index layer, a content type index layer, and a full-text search index layer; the time series data index layer establishes a primary index based on a timestamp to quickly locate data within a specific time range, the content type index layer distinguishes different types of data, and the full-text search index layer establishes an inverted index based on the data to perform fuzzy matching and keyword search; An artificial intelligence processing module, which recognizes and processes the screen data and the audio data based on a preset artificial intelligence model, and responds to user query instructions according to the index, including: responding to user query instructions, searching for matching content in the database, including recognizing and processing stored data based on index vector search, keyword search, semantic search, classification, labeling, and obtaining metadata; A plug-in processing module, responding to a user instruction to set a custom function plug-in to generate a target media pipeline; a media pipeline processing module, in response to a user instruction, processing and converting the data streams of the screen data and the audio data based on the target media pipeline; wherein the user instruction includes at least one of a pipeline download instruction, a pipeline enable instruction and a pipeline update instruction; Wherein, in the case of running the target media pipeline, the media pipeline processing module determines the execution result of the automated task-type pipeline after the object file of the automated task-type pipeline in the object file set is completed; the media pipeline processing module sends the execution result of the automated task-type pipeline to the backend service; When running the target media pipeline, the media pipeline processing module determines the execution result of the request-response type pipeline after the object file of the request-response type pipeline in the object file set is completed; the media pipeline processing module sends the execution result of the request-response type pipeline to the front-end service.

2. The screen recording system according to claim 1, characterized in that: The media pipeline processing module responds to the pipeline download instruction of the user to download the pipeline file of the target media pipeline; The media pipeline processing module saves the pipeline file to a local data directory, and creates and saves corresponding pipeline information to a database.

3. The screen recording system according to claim 2, characterized in that: The media pipeline processing module responds to a pipeline activation instruction of a user and reads the pipeline information from the database; The media pipeline processing module sets the state of the target media pipeline, creates an object file of the target media pipeline based on the pipeline information, and adds the object file to the object file collection.

4. The screen recording system according to claim 2, characterized in that: The media pipeline processing module responds to a pipeline update instruction of a user and reads the pipeline information from the database; The media pipeline processing module updates the configuration of the target media pipeline based on the pipeline information, and saves the updated pipeline information to the database.

5. The screen recording system according to claim 2, characterized in that: When the target media pipeline is loaded, the media pipeline processing module traverses all enabled target media pipelines, creates an object file of the target media pipeline, and adds the object file to the object file collection.

6. The screen recording system according to claim 1, characterized in that: After the data processing module determines the preset audio collection device, it creates a speech recognition channel to collect the audio information; The artificial intelligence processing module also performs voice activity detection processing, as well as encoding, voice transcription and decoding processing on the audio information to obtain the audio data.

7. The screen recording system according to claim 1, characterized in that: The data processing module reads a preset configuration file, and after initializing a data processing engine, continuously obtains the screen information of the electronic device; The artificial intelligence processing module recognizes text information in the screen information to obtain the screen data.

8. An electronic device, characterized in that: Comprising a screen recording system as described in any one of claims 1-7.

Images