Screen recording system based on artificial intelligence and electronic equipment

Abstract

The invention provides a screen recording system based on artificial intelligence and electronic equipment, and relates to the technical field of screen recording. The screen recording system comprises a data processing module used for collecting screen information and audio information of an electronic equipment end and carrying out data processing on the screen information and the audio information to obtain screen data and audio data; the data storage module is used for locally storing screen data and audio data and establishing an index; the artificial intelligence processing module is used for performing identification processing on the screen data and the audio data based on a preset artificial intelligence model, and responding to a user query instruction according to the index; the plug-in processing module is used for responding to a user instruction and setting a user-defined function plug-in so as to generate a target media pipeline; and the media pipeline processing module is used for processing and converting data streams of the screen data and the audio data based on a target media pipeline in response to a user instruction. The method has the advantages of being high in privacy security, high in expandability, high in artificial intelligence integration and good in adaptability to the electronic equipment end.

Description

Technical Field

[0001] The present invention relates to the technical field of screen recording, and in particular to an artificial intelligence-based screen recording system and electronic device. Background Art

[0002] With the development of artificial intelligence technology, artificial intelligence technology is widely used in various application fields. For example, the intelligent screen recording system is a recording system based on artificial intelligence technology, which is set on the electronic device end and can automatically perform multiple recording tasks.

[0003] The intelligent screen recording system can automatically start and stop recording information based on user activities, intelligently capture only relevant screen actions, and automatically edit recorded content. These features allow users to efficiently complete screen recording tasks on electronic devices without manual intervention.

[0004] However, the inventors found that the current intelligent screen recording system has problems such as poor privacy security, insufficient functional scalability, poor artificial intelligence integration and high requirements on the configuration performance of the electronic device. Summary of the invention

[0005] The present invention provides an artificial intelligence-based screen recording system and electronic device, which are used to solve the problems of poor privacy security, insufficient functional scalability, poor artificial intelligence integration and high requirements on the configuration performance of the electronic device end in the current intelligent screen recording system.

[0006] One aspect of the present invention provides a screen recording system based on artificial intelligence. The screen recording system includes a data processing module, a data storage module, an artificial intelligence processing module, a plug-in processing module and a media pipeline processing module. The data processing module collects screen information and audio information from the electronic device, and performs data processing on the screen information and audio information to obtain screen data and audio data; the data storage module locally stores the screen data and audio data, and establishes an index; the artificial intelligence processing module identifies and processes the screen data and audio data based on a preset artificial intelligence model, and responds to user query instructions according to the index; the plug-in processing module responds to user instructions to set a custom function plug-in to generate a target media pipeline; the media pipeline processing module responds to user instructions and processes and converts the data stream of screen data and audio data based on the target media pipeline.

[0007] According to some embodiments of the present invention, the media pipeline processing module responds to the user's pipeline download instruction and downloads the pipeline file of the target media pipeline; the media pipeline processing module saves the pipeline file to the local data directory and creates and saves the corresponding pipeline information to the database.

[0008] According to some embodiments of the present invention, the media pipeline processing module responds to the user's pipeline activation instruction and reads the pipeline information from the database; the media pipeline processing module sets the state of the target media pipeline, creates an object file of the target media pipeline based on the pipeline information, and adds the object file to the object file collection.

[0009] According to some embodiments of the present invention, the media pipeline processing module responds to the user's pipeline update instruction and reads the pipeline information from the database; the media pipeline processing module updates the configuration of the target media pipeline based on the pipeline information and saves the updated pipeline information to the database.

[0010] According to some embodiments of the present invention, when a target media pipeline is loaded, the media pipeline processing module traverses all enabled target media pipelines, creates object files of the target media pipeline, and adds the object files to the object file collection.

[0011] According to some embodiments of the present invention, when running the target media pipeline, the media pipeline processing module determines the execution result of the automated task-type pipeline after the object file of the automated task-type pipeline in the object file collection is completed; the media pipeline processing module sends the execution result of the automated task-type pipeline to the backend service.

[0012] According to some embodiments of the present invention, when running the target media pipeline, the media pipeline processing module determines the execution result of the request-response type pipeline after the object file of the request-response type pipeline in the object file collection is completed; the media pipeline processing module sends the execution result of the request-response type pipeline to the front-end service.

[0013] According to some embodiments of the present invention, after the data processing module determines the preset audio acquisition device, it creates a speech recognition channel to collect audio information; the artificial intelligence processing module also performs voice activity detection processing, as well as encoding, speech transcription and decoding processing of the audio information to obtain audio data.

[0014] According to some embodiments of the present invention, the data processing module reads a preset configuration file, and after initializing the data processing engine, continuously obtains screen information of the electronic device; the artificial intelligence processing module recognizes text information in the screen information to obtain screen data.

[0015] Another aspect of the present invention provides an electronic device, which includes the screen recording system as described above.

[0016] The screen recording system provided by the present invention includes a data processing module, a data storage module, an artificial intelligence processing module, a plug-in processing module and a media pipeline processing module. The present invention can continuously collect screen information and audio information of the electronic device end, and can analyze and process the collected recording information based on the artificial intelligence processing module; and the present invention can respond to user instructions through the plug-in processing module to set a custom function plug-in, and respond to user instructions through the media pipeline processing module to process and convert data streams based on the target media pipeline.

[0017] By adopting a localized data storage and processing method, the present invention can avoid the risk of uploading user data on the electronic device to the cloud (or a third-party server), thereby protecting the user's privacy and data security; the present invention can also meet different application scenarios and meet the user's customized requirements, thereby increasing the scalability of the screen recording system and reducing the user's development cost and operating difficulty; the present invention can also realize intelligent content indexing, query and automated operation based on artificial intelligence; the present invention can also automatically select the most suitable intelligent processing engine according to different operating system platforms, etc., to ensure the processing performance of intelligent processing, and has high adaptability to electronic devices with different operating systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram showing the structure of a screen recording system according to an embodiment of the present invention; Figure 2 A schematic diagram showing an architecture of a screen recording system according to an embodiment of the present invention; Figure 3 A flowchart of collecting audio data according to an embodiment of the present invention is shown; Figure 4 A flowchart of collecting screen data according to an embodiment of the present invention is shown.

[0020] Description of reference numerals: Screen recording system 1; data processing module 10; data storage module 20; artificial intelligence processing module 30; plug-in processing module 40; media pipeline processing module 50. DETAILED DESCRIPTION

[0021] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present invention will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0022] The described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of these specific details, or other modes, components, materials, devices, etc. may be adopted. In these cases, known structures, methods, devices, implementations, materials or operations will not be shown or described in detail.

[0023] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0024] The terms "first", "second" and the like in the specification, claims and the above drawings of the present invention are used to distinguish different objects rather than to describe a specific order.

[0025] The following is a clear and complete description of the technical solution of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0026] The full names of the English abbreviations involved in the present invention and their Chinese meanings are: PII: Personally Identifiable Information, personally identifiable information; OCR: Optical Character Recognition, optical character recognition; VAD: Voice Activity Detection, voice activity detection; TypeScript - open source programming language. Whisper - general-purpose speech recognition model. SQLite: lightweight database; Tauri - open source cross-platform GUI framework. React components: the core building blocks of React applications; IPC: Inter Process Communication, a technology used to exchange data between different processes; Rust - a systems-level programming language. Python: an open source programming language; TypeScript: an open source programming language; Uniform Resource Locator: Uniform Resource Locator.

[0027] The inventor of the present invention has found that the current intelligent screen recording system still has at least the following problems: 1. The data processing platform will store data in the cloud, which will increase the risk of data leakage and also cause higher latency in the remote interface when data is transmitted remotely.

[0028] 2. It is difficult to expand and customize functions according to user needs, resulting in insufficient functional scalability.

[0029] 3. It is difficult to implement artificial intelligence-based data processing and analysis, and it is impossible to directly search and locate the time of an event based on screen or sound content.

[0030] 4. Graphics card configuration is required during deployment, and cluster deployment is required, which places high requirements on the configuration performance of the electronic equipment side.

[0031] Based on this, according to one aspect of the present invention, the present invention provides a screen recording system based on artificial intelligence. Figure 1 A schematic diagram of the structure of the screen recording system according to an embodiment of the present invention is shown. Figure 1 As shown, the screen recording system 1 includes a data processing module 10, a data storage module 20, an artificial intelligence processing module 30, a plug-in processing module 40 and a media pipeline processing module 50.

[0032] According to an exemplary embodiment, the screen recording system 1 can be set on an electronic device to record screen information on the electronic device. Exemplarily, the electronic device includes but is not limited to a computer, a notebook or a mobile phone and other electronic devices for user operation.

[0033] The data processing module 10 collects screen information and audio information of the electronic device, and processes the screen information and audio information to obtain screen data and audio data.

[0034] For example, the data processing module 10 can continuously collect multimedia information generated by the electronic device, such as screen information and audio information of the electronic device, and perform preliminary data processing on the screen information and audio information in real time, thereby obtaining processed screen data and audio data.

[0035] Exemplarily, data processing includes but is not limited to data processing operations such as compression and format conversion.

[0036] According to an exemplary embodiment, the data storage module 20 locally stores screen data and audio data, and creates indexes.

[0037] For example, the data storage module 20 stores the processed screen data and audio data in the local device.

[0038] The present invention can avoid the risk of uploading user data on the electronic device to the cloud (or a third-party server) by adopting a localized data storage and processing method, thereby protecting the user's privacy and data security.

[0039] Optionally, the data processing module 10 can also automatically remove the personally identifiable information (PII) of the user of the electronic device. With such a configuration, the present invention can further enhance the privacy and security of user data.

[0040] For example, the data processing module 10 can customize filtering rules based on a preset rule engine, and can filter the user's PII information (such as ID card information, email information, and address information, etc.). Exemplarily, in the case of data with sensitive information, the data processing module 10 overwrites the data (such as adding * to overwrite sensitive information), and then stores the data in the database.

[0041] As an embodiment, the data processing module 10 loads parameter configuration (such as frame rate settings, audio and video segment duration, and enabled engine parameters), then starts the main loop and starts to continuously collect screen information and audio information. For each screen video frame, the data processing module 10 captures the screen content information and extracts OCR text data in the video frame based on OCR. The data storage module 20 inserts the video frame and OCR text data into the database, and then stores the video frame and OCR text information in the database (such as SQLite).

[0042] And, the data processing module 10 collects audio information through a preset audio device. The data processing module 10 recognizes the voice segment based on VAD, encodes, transcribes and decodes the audio information, and stores the processed audio data in a database (such as SQLite).

[0043] According to an example embodiment, the data storage module 20 may also create an index.

[0044] For example, the data storage module 20 responds to user instructions and establishes a multi-level index architecture, including a time series data index layer, a content type index layer, and a full-text search index layer. The time series data index layer can establish a primary index based on a timestamp, and can quickly locate data in a specific time range. The time series data index layer is a tree structure, and the time range can be hierarchically managed to facilitate fast range queries, and can also support incremental updates, and new data can be quickly added to the index.

[0045] The content type index layer can distinguish different types of data, such as OCR text data, audio transcription data, and video frames, etc. It can create independent sub-indexes for each type of data and optimize the query function for specific types of data.

[0046] The full-text search index layer can build an inverted index based on data (such as OCR text data, audio transcription data, and video frames), and can perform fuzzy matching and keyword searches.

[0047] According to an exemplary embodiment, the artificial intelligence processing module 30 performs recognition processing on the screen data and the audio data based on a preset artificial intelligence model, and responds to the user query instruction according to the index.

[0048] For example, the artificial intelligence processing module 30 can call the corresponding AI model preset in the screen recording system 1 according to the preset configuration and user needs. For example, the AI ​​model adapted to the corresponding hardware requirements can be intelligently imported and used according to the memory size of the electronic device. With such a setting, the present invention can optimize the utilization of system resources and ensure processing efficiency and accuracy under various hardware configurations.

[0049] Exemplarily, AI models include but are not limited to speech recognition processing (used for speech recognition, speech-to-text, etc.), computer vision (used for OCR optical recognition, target detection, etc.), natural language processing (used for question-and-answer processing), etc., and the present invention does not impose any restrictions on this.

[0050] The artificial intelligence processing module 30 can perform intelligent processing on the screen data and audio data stored in the data storage module 20 according to the called AI model. For example, the intelligent processing includes but is not limited to OCR processing, speech-to-text processing, etc., which is not limited in the present invention.

[0051] The artificial intelligence processing module 30 can automatically select the most suitable OCR processing engine and VAD voice processing engine according to different operating system platforms (such as Windows, MacOS, Linux and other operating system platforms), so as to ensure the processing performance of the intelligent processing of the artificial intelligence processing module 30.

[0052] According to an example embodiment, the artificial intelligence processing module 30 responds to the user query instruction and searches for matching content in the database. For example, the artificial intelligence processing module 30 server searches for content matching the user query (such as video frame data, OCR text data, audio transcription data, etc.) in the SQLite database, and returns the relevant content to the user as a query result.

[0053] According to an example embodiment, the plug-in processing module 40 sets a custom function plug-in in response to a user instruction to generate a target media pipeline.

[0054] For example, a media pipeline is a chain of multiple media elements that can be used to process and transmit media data. The target media pipeline can be customized according to user needs.

[0055] For example, the plug-in processing module 40 can generate a target media pipeline based on TypeScript according to user programming instructions and user-defined functional plug-ins according to user needs, so that the plug-in processing module 40 can execute the plug-in logic customized by the user.

[0056] Exemplarily, the target media pipeline generated by the custom function plug-in may include: Pipeline Meeting Minutes Plugin: For managing meeting minutes and summaries.

[0057] Pipeline Mail Log Plugin: Manages emails and sends them to a daily activity log.

[0058] Pipeline Screen to Customer Management System Plugin: Used to integrate screen data with customer management systems.

[0059] Pipeline OCR Text Stream Plugin: Used to stream OCR text data from screen data.

[0060] Pipeline Focus Notification Assistant Plugin: used for focus reminders, which can realize online learning or online examinations, etc.

[0061] Pipeline safety inspection plug-in: used to issue an alarm when suspicious activities are detected.

[0062] Pipeline Time Assistant Plugin: used to save or search user historical data.

[0063] With such configuration, the present invention can meet different application scenarios and meet the customized requirements of users, thereby increasing the scalability of the screen recording system and reducing the development cost and operation difficulty of users.

[0064] According to an exemplary embodiment, the media pipeline processing module 50 processes and converts the data streams of the screen data and the audio data based on the target media pipeline in response to a user instruction.

[0065] For example, the media pipeline processing module 50 performs flexible data stream processing on the screen data and the audio data based on the user's instruction and the custom type of the target media pipeline.

[0066] The screen recording system provided by the present invention includes a data processing module, a data storage module, an artificial intelligence processing module, a plug-in processing module and a media pipeline processing module. The present invention can continuously collect screen information and audio information of the electronic device end, and can analyze and process the collected recording information based on the artificial intelligence processing module; and the present invention can respond to user instructions through the plug-in processing module to set a custom function plug-in, and respond to user instructions through the media pipeline processing module to process and convert data streams based on the target media pipeline.

[0067] By adopting a localized data storage and processing method, the present invention can avoid the risk of uploading user data on the electronic device to the cloud (or a third-party server), thereby protecting the user's privacy and data security; the present invention can also meet different application scenarios and meet the user's customized requirements, thereby increasing the scalability of the screen recording system and reducing the user's development cost and operating difficulty; the present invention can also realize intelligent content indexing, query and automated operation based on artificial intelligence; the present invention can also automatically select the most suitable intelligent processing engine according to different operating system platforms, etc., to ensure the processing performance of intelligent processing, and has high adaptability to electronic devices with different operating systems.

[0068] Figure 2 A schematic diagram showing the architecture of a screen recording system according to an embodiment of the present invention is shown.

[0069] As an example, Figure 2 As shown, the data processing module 10 performs screen capture and audio collection according to preset function plug-ins; the data storage module 20 can store data such as SQL database embedding, screen text with timestamps, user operation history, and audio transcription with timestamps; the artificial intelligence processing module 30 can identify and process the stored data based on index vector search, keyword search, semantic search, classification, labeling, metadata acquisition, etc.; the media pipeline processing module 50 can perform flexible processing of data streams such as using LLM search history, creating / viewing summaries / indicators, triggering time-based operations, etc.

[0070] As an embodiment, the startup process of the screen recording system may include: 1. Initialization phase of the user interface built by the Tauri framework: start the Tauri main process, load the front-end web interface, initialize the front-end React components, and then establish an IPC communication bridge between the front-end and the back-end.

[0071] 2. The core initialization phase of the screen recording system: Initialize the Rust backend main process, load the necessary system configuration and environment variables, then initialize the core service components and establish a communication channel with the Tauri frontend.

[0072] 3. Parallel service startup phase: The screen recording system starts three important service threads at the same time, including: Screen recording service, responsible for continuously capturing screen images to obtain screen information, and storing screen data in real time in the local cache; Audio recording service, responsible for collecting input data from system audio and audio devices (such as microphones), and processing and caching the audio data in real time; The pipeline processing service is responsible for initializing the plugin system, loading preconfigured processing pipelines, and preparing data processing flows.

[0073] 4. Data processing thread startup phase: Initialize the OCR service, prepare the text recognition engine and configure the recognition parameters, initialize the speech-to-text service, load the speech recognition model and establish the audio processing queue, etc.

[0074] 5. Data storage system initialization phase: establish a connection with the local database, initialize the data table structure, and determine the data caching mechanism.

[0075] 6. AI service connection phase: Establish a secure connection with the AI ​​service, verify service authorization, and initialize the AI ​​processing pipeline.

[0076] 7. UI status synchronization phase: push the startup status of each service to the Tauri front end, activate the user interface interaction function, and display the system's ready status.

[0077] Through the above embodiments, the startup process of the screen recording system provided by the present invention adopts the Tauri framework to realize the front-end interface rendering, and through the multi-thread parallel processing mechanism, the various back-end components can work together efficiently. And the screen recording system adopts an event-driven architecture, which can realize real-time communication and state synchronization between the front and back ends, thereby providing a smooth operating experience and providing a stable operating environment for continuous recording and real-time processing of screen data and audio data.

[0078] Optionally, after the data processing module 10 determines the preset audio collection device, it creates a voice recognition channel to collect audio information. The artificial intelligence processing module 30 also performs voice activity detection processing, as well as encoding, voice transcription and decoding processing of the audio information to obtain audio data.

[0079] Figure 3 FIG. 2 is a flowchart of collecting audio data according to an embodiment of the present invention. Figure 3 As shown, the audio data collection flow chart includes steps S11-S18.

[0080] In step S11 , the data processing module 10 determines available audio acquisition devices.

[0081] In step S12, the data processing module 10 obtains a default input / output device.

[0082] In step S13 , the data processing module 10 creates a speech recognition channel (eg, a Whisper speech recognition channel).

[0083] In step S14, the data processing module 10 collects audio information.

[0084] In step S15 , the artificial intelligence processing module 30 performs voice activity detection processing (eg, VAD).

[0085] In step S16 , the artificial intelligence processing module 30 encodes the audio information.

[0086] In step S17 , the artificial intelligence processing module 30 performs speech transcription (eg, using a Whisper model to perform speech transcription).

[0087] In step S18 , the artificial intelligence processing module 30 decodes the audio information, obtains the audio data and stores it (for example, in the data storage module 20 ).

[0088] Figure 4 FIG. 2 is a flowchart showing the collection of screen data according to an embodiment of the present invention. Figure 4 As shown, the screen data collection flow chart includes steps S21-S25.

[0089] In step S21 , the data processing module 10 reads a preset configuration file.

[0090] In step S22 , the data processing module 10 initializes a data processing engine (eg, an OCR processing engine).

[0091] In step S23 , the data processing module 10 continuously obtains screen information (such as screenshots) of the electronic device.

[0092] In step S24 , the artificial intelligence processing module 30 performs filtering processing on the acquired screen information.

[0093] In step S25 , the artificial intelligence processing module 30 recognizes text information in the screen information based on OCR, and processes the text recognition result to obtain screen data.

[0094] As an embodiment, the data collection, storage and search process of the screen recording system can be as follows: 1. Data collection stage: The plug-in processing module 40 can be used to collect multi-source data, including real-time capture of screen images (i.e., screen information), collection of system audio information, and recording of interactive events. During the data collection stage, the backend of the screen recording system continuously runs the collection service to ensure continuous data collection (e.g., 24 hours a day).

[0095] 2. Data processing stage: Data processing can include OCR text recognition of screen images, speech-to-text processing of audio information, data standardization, unified data format, extraction of key information, selection of whether to filter sensitive information based on configuration, and generation of metadata tags.

[0096] 3. Data storage stage: Data storage may include using local databases for storage, storage of content classification, original audio and video clips, processed text data, metadata information, and indexing to optimize query efficiency.

[0097] 4. Data search (query) stage: Data search can include providing multi-dimensional query interface, time dimension query, content keyword query, event type query, query optimization, caching mechanism, incremental update and fuzzy matching, etc.

[0098] Optionally, the media pipeline processing module 50 responds to the pipeline download instruction of the user and downloads the pipeline file of the target media pipeline. The media pipeline processing module 50 saves the pipeline file to the local data directory and creates and saves the corresponding pipeline information to the database.

[0099] For example, the pipeline file can be a python or typescript script that can implement a preset function (such as an alarm function or an automatic email function). The media pipeline processing module 50 can respond to the user's pipeline download instruction (i.e., the user requests to download the media pipeline) and download the pipeline file of the target media pipeline in the URL.

[0100] Optionally, the media pipeline processing module 50 responds to the user's pipeline activation instruction and reads pipeline information from the database. The media pipeline processing module 50 sets the state of the target media pipeline, creates an object file of the target media pipeline based on the pipeline information, and adds the object file to the object file collection.

[0101] For example, the media pipeline processing module 50 may respond to the user's pipeline enabling instruction (i.e., the user requests to enable the media pipeline) and set the state of the target media pipeline to enabled. The media pipeline processing module 50 may create an object file (Future file) executed by the target media pipeline and then add the Future file to the object file collection (Pipe-Futures).

[0102] Optionally, the media pipeline processing module 50 responds to the pipeline update instruction of the user, reads the pipeline information from the database, updates the configuration information of the target media pipeline, and saves the updated pipeline information to the database.

[0103] For example, the configuration information of the target media pipeline may be configuration information of some custom variables in the media pipeline, such as alarm threshold information, website information, email address information, monitoring specific object name information, or video acquisition frame rate information.

[0104] Optionally, when loading a target media pipeline, the media pipeline processing module 50 traverses all enabled target media pipelines, creates an object file of the target media pipeline, and adds the object file to the object file collection.

[0105] For example, after traversing all media pipelines in the enabled state, the media pipeline processing module 50 creates a Future file for pipeline execution, and then adds the Future file to Pipe-Futures.

[0106] Optionally, when the target media pipeline is running, the media pipeline processing module 50 determines the execution result of the automated task-based pipeline after the object file of the automated task-based pipeline in the object file set is completed. The media pipeline processing module 50 sends the execution result of the automated task-based pipeline to the backend service.

[0107] Optionally, when the target media pipeline is running, the media pipeline processing module 50 determines the execution result of the request-response pipeline after the object file of the request-response pipeline in the object file set is completed. The media pipeline processing module 50 sends the execution result of the request-response pipeline to the front-end service.

[0108] For example, the target media pipeline can include automated task pipelines and request-response pipelines. Automated task pipelines can be tasks that need to run in the background for a long time to collect audio and video information, such as scheduled tasks or monitoring tasks; request-response pipelines can be tasks that respond to user requests, such as summarizing existing content, log processing, email sending and receiving processing, and search problem processing.

[0109] According to another aspect of the present invention, an electronic device is provided, which includes the artificial intelligence-based screen recording system as described above.

[0110] Exemplarily, the electronic device includes but is not limited to a computer, a notebook, a mobile phone or other electronic device for user operation, and the present invention is not limited to this.

[0111] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions of the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A screen recording system based on artificial intelligence, characterized in that: The screen recording system is arranged at the electronic device end, and includes: A data processing module collects screen information and audio information of the electronic device, and performs data processing on the screen information and the audio information to obtain screen data and audio data; A data storage module, storing the screen data and the audio data locally, and creating an index; An artificial intelligence processing module, which recognizes and processes the screen data and the audio data based on a preset artificial intelligence model, and responds to user query instructions according to the index; A plug-in processing module, responding to a user instruction to set a custom function plug-in to generate a target media pipeline; The media pipeline processing module responds to user instructions and processes and converts the data streams of the screen data and the audio data based on the target media pipeline.

2. The screen recording system according to claim 1, characterized in that: The media pipeline processing module responds to the pipeline download instruction of the user to download the pipeline file of the target media pipeline; The media pipeline processing module saves the pipeline file to a local data directory, and creates and saves corresponding pipeline information to a database.

3. The screen recording system according to claim 2, characterized in that: The media pipeline processing module responds to a pipeline activation instruction of a user and reads the pipeline information from the database; The media pipeline processing module sets the state of the target media pipeline, creates an object file of the target media pipeline based on the pipeline information, and adds the object file to the object file collection.

4. The screen recording system according to claim 2, characterized in that: The media pipeline processing module responds to a pipeline update instruction of a user and reads the pipeline information from the database; The media pipeline processing module updates the configuration of the target media pipeline based on the pipeline information, and saves the updated pipeline information to the database.

5. The screen recording system according to claim 2, characterized in that: When the target media pipeline is loaded, the media pipeline processing module traverses all enabled target media pipelines, creates an object file of the target media pipeline, and adds the object file to the object file collection.

6. The screen recording system according to claim 3, characterized in that: In the case of running the target media pipeline, the media pipeline processing module determines the execution result of the automated task-based pipeline after the object file of the automated task-based pipeline in the object file set is completed; The media pipeline processing module sends the execution result of the automated task-based pipeline to the backend service.

7. The screen recording system according to claim 3, characterized in that: In the case of running the target media pipeline, the media pipeline processing module determines the execution result of the request-response pipeline after the object file of the request-response pipeline in the object file set is completed; The media pipeline processing module sends the execution result of the request-response pipeline to the front-end service.

8. The screen recording system according to claim 1, characterized in that: After the data processing module determines the preset audio collection device, it creates a speech recognition channel to collect the audio information; The artificial intelligence processing module also performs voice activity detection processing, as well as encoding, voice transcription and decoding processing on the audio information to obtain the audio data.

9. The screen recording system according to claim 1, characterized in that: The data processing module reads a preset configuration file, and after initializing a data processing engine, continuously obtains the screen information of the electronic device; The artificial intelligence processing module recognizes text information in the screen information to obtain the screen data.

10. An electronic device, characterized in that: Comprising a screen recording system as described in any one of claims 1-9.

Images