Screen video stream processing method, electronic equipment, storage medium and program product

Abstract

The invention discloses a screen video stream processing method, electronic equipment, a storage medium and a program product. The method comprises the following steps: calling a machine learning model to identify first feature information of a first key video frame in response to the acquired first key video frame of a screen video stream; and performing semantic decision on the first feature information, and determining a first semantic event corresponding to the first key video frame from a plurality of semantic events corresponding to the screen video stream. The information extraction efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of screen video stream processing technology, and in particular to a screen video stream processing method, electronic device, storage medium, and program product. Background Technology

[0002] In various scenarios such as remote home monitoring, remote office work, and other IoT system monitoring, it is often necessary to analyze the screen content of client devices (such as personal computers, laptops, tablets, and mobile phones). Currently, the commonly used technical solution is to manually review the screen video streams recorded by the client devices, which is time-consuming and results in low information extraction efficiency. Summary of the Invention

[0003] This application provides a screen video stream processing method, electronic device, storage medium, and program product, aiming to improve information extraction efficiency. In a first aspect, embodiments of this application provide a screen video stream processing method applied to a server, comprising: in response to obtaining a first key video frame of the screen video stream, calling a machine learning model to identify first feature information of the first key video frame; performing semantic decision on the first feature information, and determining a first semantic event corresponding to the first key video frame from multiple semantic events corresponding to the screen video stream.

[0004] Secondly, embodiments of this application provide a screen video stream processing method applied to a receiving terminal, comprising: displaying the event log in response to obtaining an event log corresponding to a first key video frame of the screen video stream; wherein the event log is generated by a first semantic event and a timestamp carried by the first key video frame; the first semantic event is determined from multiple semantic events corresponding to the screen video stream by performing semantic decision on first feature information corresponding to the first key video frame; the first feature information is obtained by identifying the first key video frame through a machine learning model; and the screen video stream is obtained by recording the screen through the receiving terminal or the sending terminal.

[0005] Thirdly, embodiments of this application provide an electronic device, including: at least one processor; at least one memory for storing at least one program; and when at least one of the programs is executed by at least one of the processors, implementing the screen video stream processing method as described in the first or second aspect.

[0006] This application provides a computer-readable storage medium storing computer-executable instructions for performing the screen video stream processing method as described in the first or second aspect.

[0007] Fourthly, embodiments of this application provide a computer program product, including a computer program or computer instructions, the computer program or computer instructions being stored in a computer-readable storage medium, a processor of an electronic device reading the computer program or computer instructions from the computer-readable storage medium, and the processor executing the computer program or computer instructions to cause the electronic device to perform the screen video stream processing method as described in the first or second aspect.

[0008] In this embodiment, the server identifies the first key video frame of the screen video stream by calling a machine learning model to obtain first feature information; then, semantic decision-making is performed on the first feature information to determine the first semantic event corresponding to the first key video frame from multiple semantic events corresponding to the screen video stream. That is, in this embodiment, the extraction of semantic events from the screen video stream is automatic without manual intervention; this avoids users of receiving terminals spending a lot of time extracting semantic events from the screen video stream by simply viewing it. Furthermore, this embodiment does not determine the semantic events corresponding to all video frames of the screen video stream, but rather the semantic events corresponding to key video frames, thereby reducing the amount of data processing involved in determining semantic events. In summary, this embodiment improves information extraction efficiency. Attached Figure Description

[0009] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0010] Figure 1 This is a system architecture that can be applied to the embodiments of this application; Figure 2 This is one of the flowcharts illustrating the screen video stream processing method provided in the embodiments of this application; Figure 3 This is one of the display interfaces of the receiving terminal provided in the embodiments of this application; Figure 4 This is the second display interface of the receiving terminal provided in the embodiments of this application; Figure 5 This is a second schematic flowchart of the screen video stream processing method provided in the embodiments of this application; Figure 6 This is a schematic diagram of a server structure provided in an embodiment of this application; Figure 7 This is the third flowchart of the screen video stream processing method provided in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0012] It should be understood that in the description of the embodiments of this application, the use of terms such as "first" and "second" is only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of technical features indicated, or implicitly indicating the order of the technical features indicated. "At least one" refers to one or more, and "more" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any group of these items, including any group of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0013] To facilitate understanding of the solutions in the embodiments of this application, some contents involved in the embodiments of this application are described below: As mentioned in the background technology, manually viewing the screen video stream obtained by recording the screen of the client device takes a lot of time, resulting in low information extraction efficiency.

[0014] To address the technical challenge of improving information extraction efficiency, the applicant discovered that the purpose of manually viewing the screen video stream of a client device is to extract the following information from the screen video stream: actions, state changes, or interactive behaviors of the client device or its user that have a clear meaning and purpose; thereby enabling the monitoring of the client device's usage through this information.

[0015] In this application embodiment, "actions, state changes or interactive behaviors with clear meaning and purpose that occur on the client device or its user" are regarded as semantic events. Therefore, by automatically extracting semantic events from the screen video stream, the technical problem of how to improve the efficiency of information extraction can be solved.

[0016] Based on this, embodiments of this application provide a screen video stream processing method, electronic device, storage medium, and program product. The server identifies the first key video frame of the screen video stream by calling a machine learning model to obtain first feature information; then, semantic decision-making is performed on the first feature information to determine the first semantic event corresponding to the first key video frame from multiple semantic events corresponding to the screen video stream. That is, in the process of extracting semantic events from the screen video stream in this application embodiment, no manual intervention is required, achieving automatic extraction; this avoids users of receiving terminals used to receive semantic events spending a lot of time extracting semantic events from the screen video stream by simply viewing it. Furthermore, this application does not determine the semantic events corresponding to all video frames of the screen video stream, but rather the semantic events corresponding to key video frames of the screen video stream, thereby reducing the amount of data processing involved in determining semantic events. In summary, embodiments of this application improve information extraction efficiency.

[0017] A system applicable to embodiments of this application may include a server and a receiving terminal. In this embodiment, the client device is a receiving terminal. The receiving terminal communicates with the server via a network.

[0018] During the first user's use of the receiving terminal, the terminal automatically records the screen to obtain a screen video stream and sends this stream to the server. The server processes key video frames from the screen video stream into corresponding semantic events and sends the event log to the receiving terminal. The event log is generated based on the semantic events and the timestamps carried by the key video frames. The receiving terminal receives and displays the event log for the second user to view. This improves the efficiency of the second user in extracting semantic events from the screen video stream while monitoring the first user's use of the receiving terminal.

[0019] In some embodiments, during the process of the first user using the receiving terminal, the receiving terminal captures the screen image in real time and performs video encoding on the screen image to obtain a screen video stream.

[0020] It should be noted that after recording the screen to obtain the screen video stream, the receiving terminal can directly send the screen video stream to the server via the streaming media protocol, or the receiving terminal can first extract frames from the screen video stream to obtain key video frames, and then send the key video frames to the server via the streaming media protocol.

[0021] Both the receiving terminal and the server can extract key video frames from the screen video stream using the following frame extraction strategy.

[0022] Frame extraction strategy: At the screen video stream decoding unit level, frame extraction decisions are made by comparing the perceptual hash or structural similarity between consecutive frames, or by analyzing the motion vectors in the screen video stream.

[0023] Taking the frame-skipping decision made by comparing the perceptual hashes between consecutive frames as an example: Calculate the Hamming distance of the perceptual hash between consecutive frames. If the Hamming distance is less than or equal to a first preset distance threshold, it can be determined that the image changes of consecutive frames are minimal. If the image changes of multiple consecutive frames are minimal (e.g., the screen image remains on a static document or video interface), it is determined to be redundant data, meaning that these multiple consecutive frames are not critical video frames.

[0024] If the Hamming distance is greater than or equal to the second preset distance threshold, it can be determined that the image change in consecutive frames is extremely large. This indicates that a significant change has occurred in the screen image (such as application switching, pop-up windows, page scrolling, etc.). The video frame that caused the significant change is captured and marked as the key video frame.

[0025] In some embodiments, the first preset distance threshold is less than the second preset distance threshold, and both can be set based on experience; this embodiment does not impose specific limitations.

[0026] In some embodiments, a frame extraction strategy is also provided: the screen video stream is sampled at fixed time intervals (e.g., 1 second), and the sampled video frames are used as key video frames. This frame extraction strategy has simple calculation logic and high efficiency.

[0027] In some embodiments, the server receives a screen video stream, and then the server extracts frames from the screen video stream to obtain key video frames.

[0028] In this embodiment, the receiving terminal directly sends the screen video stream to the server via the streaming media protocol, without needing to extract frames from the screen video stream, thus saving the receiving terminal's computing resources.

[0029] In some embodiments, the server receives key video frames.

[0030] It's understandable that having the receiving terminal obtain the key video frames first, and then sending them to the server via a streaming media protocol, places lower demands on the receiving terminal's uplink bandwidth compared to having the receiving terminal directly send the screen video stream to the server via the streaming media protocol. Furthermore, storing the key video frames on the server saves server storage resources compared to storing the screen video stream.

[0031] In some embodiments, the receiving terminal may be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart TVs, smart in-vehicle devices, etc. Portable wearable devices may include head-mounted displays, etc. Head-mounted displays may include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0032] A server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0033] See Figure 1 , Figure 1 This is a system architecture that can be applied to the embodiments of this application, such as... Figure 1 As shown, it may include a sending terminal 102, a server 104, and a receiving terminal 106.

[0034] In this embodiment, the aforementioned client device is the sending terminal 102. Both the sending terminal 102 and the receiving terminal 106 communicate with the server 104 via a network.

[0035] During the user's use of the sending terminal 102, the sending terminal 102 automatically records the screen to obtain a screen video stream and sends the screen video stream to the server 104. The server 104 processes the key video frames of the screen video stream into corresponding semantic events and sends the event log to the receiving terminal 106. The event log is generated based on the semantic events and the timestamps carried by the key video frames. The receiving terminal 106 receives and displays the event log for the user of the receiving terminal 106 to view. Based on this system architecture, the receiving terminal 106 can perform cross-device monitoring of the usage of the sending terminal 102, and during the cross-device monitoring process, the information extraction efficiency of the receiving terminal 106 in extracting semantic events from the screen video stream can be improved.

[0036] In some embodiments, the transmitting terminal 102 captures the screen image in real time and performs video encoding on the screen image to obtain a screen video stream.

[0037] It should be noted that after recording the screen to obtain the screen video stream, the sending terminal 102 can directly send the screen video stream to the server 104 via the streaming media protocol, or the sending terminal 102 can first extract frames from the screen video stream to obtain key video frames, and then send the key video frames to the server 104 via the streaming media protocol.

[0038] Both the sending terminal 102 and the server 104 can extract key video frames from the screen video stream using a frame extraction strategy. Examples of this frame extraction strategy can be found in the relevant descriptions of the various embodiments of the system including the server and the receiving terminal described above; to avoid repetition, they will not be repeated here.

[0039] In some embodiments, the receiving terminal 106 may be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart TVs, smart in-vehicle systems, etc. Portable wearable devices may include smartwatches, head-mounted displays, etc. Head-mounted displays may include VR devices, AR devices, smart glasses, etc.

[0040] The transmitting terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart TVs, smart in-vehicle devices, etc. Portable wearable devices can include head-mounted displays, etc. Head-mounted displays can include VR devices, AR devices, smart glasses, etc.

[0041] Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0042] The embodiments of this application are applicable to various application scenarios, such as online education supervision, remote system auditing and workflow analysis, and automated information technology (IT) support and troubleshooting. The semantic events corresponding to different application scenarios are generally different.

[0043] The following uses the system of the embodiment of this application, which includes a sending terminal 102, a server 104, and a receiving terminal 106, as an example to briefly describe various application scenarios.

[0044] In some embodiments, in the context of online education supervision, semantic events can be, but are not limited to, "games," "instant messaging," "online learning," and "entertainment videos." Parents can understand their child's learning activities through the sending terminal 102 by receiving the event log generated by the semantic events displayed on the receiving terminal 106. For example, the percentage of time the child spends learning through the sending terminal 102 within a certain period can be determined by the semantic events corresponding to key video frames.

[0045] In some embodiments, in remote system auditing and workflow analysis scenarios, semantic events can be used to objectively assess system usage and optimize team workflows. Semantic events can be, but are not limited to, "approval dialog box appears," "reimbursement interface is active," and "sensitive data is exposed." Users of receiving terminal 106 (such as administrators) can use the event logs generated by semantic events displayed on receiving terminal 106 to conduct compliance audits on employees using the system on sending terminal 102. For example, if the system is a financial system, and the semantic event corresponding to a key video frame includes "sensitive data exposure," it can be determined that an employee has engaged in misconduct while using the financial system.

[0046] In some embodiments, semantic events can be used for rapid fault location in automated IT support and troubleshooting scenarios. Semantic events can be, but are not limited to, "driver error," "program crash," or "service stoppage." Operations personnel can quickly locate the fault and complete troubleshooting by receiving event logs generated from semantic events displayed on terminal 106.

[0047] The following uses the scenario of online education supervision as an example to illustrate various embodiments. Other application scenarios can refer to the scenario of online education supervision, and the embodiments of this application will not be repeated.

[0048] The screen video stream processing method provided in this application will be described in detail below with reference to the accompanying drawings and through some embodiments and application scenarios.

[0049] See Figure 2 , Figure 2 This is one of the flowcharts illustrating the screen video stream processing method provided in this application embodiment. The screen video stream processing method of this application embodiment can be applied to the aforementioned server (as described above). Figure 1 In the server 104), in specific implementation, the method can be executed by the server, or by the server's components, such as the server's processor, chip, or chip system, or by the logic module or software that implements all or part of the server's functions.

[0050] The screen video stream processing method described below, as exemplified in this application, is applied to the above-mentioned... Figure 1 Taking server 104 as an example, various embodiments will be described. For the various embodiments applied to a server in a system including a server and a receiving terminal, it is understood that the receiving terminal can be used simultaneously to implement... Figure 1 The method steps executed by the sending terminal 102 and the receiving terminal 106 in this application will not be described again in order to avoid repetition.

[0051] like Figure 2As shown, the screen video stream processing method includes, but is not limited to, the following steps: Step 201: In response to obtaining the first key video frame of the screen video stream, call the machine learning model to identify the first feature information of the first key video frame.

[0052] In some embodiments, the first key video frame is any key video frame of the screen video stream, that is, for any key video frame, its semantic events can be determined by the screen video stream processing method of the embodiments of this application.

[0053] In some embodiments, the machine learning model may be deployed on a server.

[0054] In some embodiments, the machine learning model has the ability to identify features of key video frames. There can be multiple models to identify different features in a targeted manner. The feature information is information about the features of the key video frames that are related to semantic events.

[0055] In some embodiments, the first feature information includes at least one of the following: user interface element information of the first key video frame, character recognition information of the first key video frame, and application category information corresponding to the first key video frame. Accordingly, the invoked machine learning model includes at least one of the following: an object detection model (such as YOLO (You Only LookOnce)) or a machine learning model based on Optical Character Recognition (OCR) technology.

[0056] It should be noted that User Interface (UI) elements are a collective term for various controls used to build user interaction interfaces, including visual components such as navigation bars, menus, and input boxes.

[0057] In some embodiments, for UI element information, the UI elements of key video frames are first identified by the corresponding object detection model, and then the interface type or UI type (such as virtual joystick) of the key video frames is identified by the corresponding object detection model through the UI elements. The interface type or UI type is the UI element information.

[0058] In some embodiments, character recognition information refers to character-related information such as words and phrases in the user interface, such as "Start Game," "Eat," and "Lecture." This character recognition information can be obtained by a machine learning model based on OCR technology.

[0059] In some embodiments, for application category information, UI elements of key video frames can be identified first using a corresponding object detection model, and then application category information (such as communication applications) can be identified based on the UI elements using the corresponding object detection model. Alternatively, application category information of key video frames can be identified directly using an object detection model.

[0060] It should be noted that, as one possible implementation, the sending terminal can also call the system API to obtain the package name of the foreground application while recording the screen. It can then establish a association between the video frame and the package name, specifically by using the package name as metadata for the video frame. Therefore, when uploading key video frames or screen video streams, the sending terminal can also upload this metadata. Thus, after identifying the first feature information of the first key video frame, the package name can be obtained by parsing the metadata, and the second feature information, i.e., application category information, can be obtained from the package name.

[0061] It's understandable that, since determining the application category information of the first key video through metadata is more efficient than calling a machine learning model, the server can determine the second feature information using metadata when it exists, without calling a machine learning model to identify the application category information of the first key video. Only when the sending terminal has not uploaded metadata will the machine learning model be called to identify the application category information of the first key video.

[0062] In some embodiments, the feature information may be carried by structured data (such as JSON data).

[0063] Step 202: Perform semantic decision-making on the first feature information, and determine the first semantic event corresponding to the first key video frame from multiple semantic events corresponding to the screen video stream.

[0064] It should be noted that screen video streams correspond to different application scenarios (such as online education supervision, automated IT support, and troubleshooting). If a screen video stream is a video stream obtained by the sending terminal recording the screen under a certain application scenario, then that video stream corresponds to that application scenario. Furthermore, application scenarios also correspond to multiple semantic events (such as games and faults), and these correspondences can be preset.

[0065] For example, in the application scenario of online education supervision, the screen video stream can show the normal operation interface of game applications, browsers, and video applications, with semantic events such as "game" and "learning"; while in the application scenario of automated IT support and troubleshooting, the screen video stream can show the abnormal operation interface of office software, with semantic events such as "error message" and "app crash".

[0066] Therefore, in the process of semantic decision-making on the first feature information, multiple semantic events that may be used in the semantic decision-making process can be determined by the target application scenario corresponding to the screen video stream, and then the first semantic event corresponding to the first key video frame can be determined from the multiple semantic events.

[0067] It should be noted that, in addition to establishing the correspondence between screen video streams and multiple semantic events through application scenarios, the correspondence between screen video streams and multiple semantic events can also be established through user accounts, etc. That is, when making semantic decisions on the first feature information, the target application scenario is first determined through the user account corresponding to the screen video stream, and then the corresponding multiple semantic events are determined through the target application scenario.

[0068] This embodiment identifies the first key video frame of the screen video stream by calling a machine learning model to obtain first feature information; then, semantic decision-making is performed on this first feature information to determine the first semantic event corresponding to the first key video frame from multiple semantic events corresponding to the screen video stream. That is, in this embodiment, the extraction of semantic events from the screen video stream is automatic and requires no manual intervention; this avoids users of receiving terminals spending a lot of time extracting semantic events from the screen video stream by simply viewing it. Furthermore, this embodiment does not determine the semantic events corresponding to all video frames of the screen video stream, but rather the semantic events corresponding to key video frames, thereby reducing the amount of data processing involved in determining semantic events. In summary, this embodiment improves information extraction efficiency.

[0069] In some embodiments, semantic decision-making can be implemented based on rule-based knowledge base matching, probabilistic reasoning, etc.

[0070] For probabilistic reasoning: Specifically, a Bayesian network can be constructed where "multiple semantic events" are parent nodes, "various feature information" are child nodes, and the connections between nodes represent dependencies. When applying this Bayesian network, the first feature information is input into the network, and probabilities are derived using Bayes' theorem to obtain the probability of each semantic event. The semantic event with the highest probability is then taken as the first semantic event corresponding to the first key video frame.

[0071] For rule-based knowledge base matching: First, a rule knowledge base corresponding to the screen video stream is obtained; second, a first rule matching the first feature information is obtained from the rule knowledge base; wherein, the matching of the first feature information with the first rule indicates that the first feature information satisfies the triggering condition corresponding to the first rule. It can be understood that this embodiment provides a feasible technical solution for semantic decision-making based on the first feature information.

[0072] It should be noted that each rule in the rule knowledge base corresponds to a semantic event. Since different application scenarios correspond to different semantic events, independent rule knowledge bases can be established for different application scenarios to determine the semantic events of key video frames in the screen video stream for the corresponding application scenario. For example, rule knowledge base A corresponds to the scenario of online education supervision, while rule knowledge base B corresponds to the scenario of automated IT support and troubleshooting.

[0073] In some embodiments, the rule knowledge base includes multiple rules, each rule corresponding to a triggering condition and a semantic event; the triggering condition corresponding to each rule is used to define the feature information that the semantic event corresponding to each rule must satisfy.

[0074] In this embodiment, the semantic events corresponding to each rule defined for each triggering condition must satisfy the feature information, which may be UI element information, character recognition information, and / or application category information.

[0075] For example, rule knowledge base A corresponds to the scenario of online education supervision. Rule knowledge base A contains rule a1 and rule a2. The triggering condition for rule a1 is that the corresponding semantic event must satisfy any one of the following features: UI element information, character recognition information, and application category information; the triggering condition for rule a2 is that the corresponding semantic event must satisfy the following features: character recognition information and application category information.

[0076] Specifically, the semantic event corresponding to rule a1 can be "game," and the corresponding triggering condition is that any one of the following is met: 1. The UI element information is a game-related UI type such as a virtual joystick or in-game button; 2. The character recognition information is game-related characters such as "Start Game," "Game Over," or "Enemy"; 3. The application category information is a game application. The semantic event corresponding to rule a2 is "professional work," and the corresponding triggering condition is that all of the following are met: 1. The character recognition information is work-related characters such as "Meeting Minutes," "Flowchart," or "Code"; 2. The application category information is a work application.

[0077] Assume the first feature information consists of the user interface element information of the first key video frame, the character recognition information of the first key video frame, and the application category information corresponding to the first key video frame. The user interface element information of the first key video frame is a virtual joystick, the character recognition information of the first key video frame is "code," and the application category information corresponding to the first key video frame is a game application. It can be understood that the first feature information satisfies the triggering conditions corresponding to rule a1, specifically satisfying the first and third items; but it does not satisfy the triggering conditions corresponding to rule a2 because the first feature information only satisfies the first item of the triggering conditions and not the second item. Therefore, the first rule matching the first feature information is rule a1, and correspondingly, the first semantic event is "game."

[0078] It should be noted that the above matching method can be word vector matching, that is, by mapping words to a low-dimensional real vector space, the semantic similarity of words or text is measured by using the geometric relationship between vectors (such as cosine similarity).

[0079] After the first rule is obtained through matching, the semantic event corresponding to the first rule is determined as the first semantic event.

[0080] It should be noted that during the matching process, since there can be many rules in the rule knowledge base, the number of candidate rules that match the first feature information is likely to be greater than 1, which may lead to ambiguity in the semantic event corresponding to the first key video frame.

[0081] In some embodiments, a resolution mechanism is provided to resolve the ambiguity, thereby enabling a first key video frame to uniquely match a certain first semantic event. Specifically, to determine the first rule from multiple candidate rules, each rule is assigned a unique priority, which in some embodiments can be represented by weights. Thus, when multiple candidate rules are matched, i.e., in response to the rule knowledge base including multiple candidate rules matching the first feature information, the candidate rule with the highest priority among the multiple candidate rules is determined as the first rule.

[0082] For example, candidate rule a1, candidate rule a2, and candidate rule a3 are obtained by matching from rule knowledge base A.

[0083] In some embodiments, if candidate rule a1 has the highest priority, then candidate rule a1 is determined as the first rule.

[0084] In some embodiments, some pop-ups may reduce the accuracy of the determined first semantic event. This is because, in the case of pop-ups such as advertising pop-ups and notification pop-ups in key video frames, the obtained first feature information may come partly from the pop-ups. These pop-ups will participate in the matching of the first rule, thereby affecting the determination of the first semantic event.

[0085] To address the interference of pop-up windows in the matching process and improve the accuracy of the determined first semantic event, this application proposes an automatic filtering mechanism tailored to the characteristics of these pop-ups: short duration and automatic closure. Specifically: The first semantic event is compared with the second semantic event corresponding to the second key video frame in the screen video stream, where the second key video frame is the previous key video frame of the acquired first key video frame. In response to the difference between the first and second semantic events, a timer is started.

[0086] In response to the timer reaching the preset duration, and if the semantic events corresponding to the key video frames of the screen video stream acquired during the timer's timing period are all first semantic events, the first semantic event is output.

[0087] In some embodiments, the preset duration can be determined based on the display duration of the pop-up from its appearance to its automatic closure. For example, if the display duration is 2 seconds, the preset duration can be slightly longer than 2 seconds, such as 3 seconds.

[0088] Furthermore, the output of the first semantic event can be either the event log corresponding to the first semantic event or the analysis report corresponding to the first semantic event. It should be noted that the event log and / or analysis report are sent from the server to the receiving terminal.

[0089] In some embodiments, the event log is generated based on the timestamp carried by the first semantic event and the first key video frame. That is, the timestamp carried by the first key video frame is appended to the first semantic event, thereby aggregating a structured log data.

[0090] In some embodiments, aggregation refers to storing timestamped semantic events in a database and forming a machine-readable, structured event log in chronological order. It can be understood that, assuming the semantic events remain unchanged, each log entry corresponds to a semantic event and its duration. Therefore, after the receiving terminal displays the event log, the user can more easily obtain the information carried by the screen video stream.

[0091] In some embodiments, for an analysis report, the process of obtaining and sending the analysis report via a first semantic event specifically includes: An event log is generated based on the timestamp carried by the first semantic event and the first key video frame; based on the event log, log analysis results for the application scenario corresponding to the screen video stream are obtained; based on the event log and the log analysis results, an analysis report is generated; and the analysis report is sent to the receiving terminal of the screen video stream.

[0092] In some embodiments, log analysis results may include the percentage of time a child spends studying, a child's concentration level score, etc.

[0093] The time percentage can be determined by comparing the duration of the "learning" session with the total duration of the event log.

[0094] For focus assessment, the duration of "learning" can be matched with multiple preset time intervals for scoring. For example, preset time intervals include 10-20 minutes, 20-30 minutes, and 30-40 minutes, corresponding to focus assessment scores of 50, 70, and 90 points, respectively.

[0095] It should be noted that, in order to avoid overwhelming the receiving terminal with event logs and / or analysis reports, this embodiment limits the timing of outputting the first semantic event. One approach is to output it once at fixed intervals (e.g., 1 hour); another approach is to output it when the semantic event corresponding to any key video frame is the target semantic event.

[0096] In some embodiments, the target semantic event can be set by the user of the receiving terminal. For example, if the user of the receiving terminal sets the target semantic event to "game" or "online learning," then the event will only be output if the first semantic event is "game" or "online learning," and will not be output if the first semantic event is another semantic event (such as "entertainment video"). Furthermore, the user of the receiving terminal can also set a filter for semantic events outside of the learning period, meaning the server will not output semantic events outside of the learning period.

[0097] It should be noted that the server can also output other content, such as primary feature information, as a basis for users to understand the server's semantic decisions.

[0098] In some embodiments, the server outputs the first semantic event when the semantic event corresponding to any key video frame is the target semantic event.

[0099] like Figure 3 As shown, Figure 3 This is one of the display interfaces of the receiving terminal provided in the embodiments of this application. The target semantic events include at least "online learning", "games" and "instant messaging"; the duration of each semantic event is calculated by timestamp; "Application: video player, content: lecture hall" and other similar information are the first feature information.

[0100] In some embodiments, the server outputs the first semantic event when the semantic event corresponding to any key video frame is the target semantic event.

[0101] like Figure 4 As shown, Figure 4This is the second display interface of the receiving terminal provided in this application embodiment. The target semantic events include at least "online learning", "games" and "instant messaging"; the duration of each semantic event is calculated using timestamps; "Application: video player, content: lecture" and other similar information are the first feature information. In addition, it also includes log analysis results, namely "learning time accounts for 70%".

[0102] It is understandable that the receiving terminal receives and displays the event logs or analysis reports sent by the server, which is equivalent to displaying the "high-value" information in the screen video stream, while not displaying the redundant information in the screen video stream.

[0103] Furthermore, the timing of outputting the first semantic event can also be that the new semantic event differs from the most recent semantic event. That is, after comparing the first semantic event with the second semantic event corresponding to the second key video frame of the screen video stream, the method further includes: In response to the first semantic event being the same as the second semantic event, the fourth key video frame of the screen video stream is obtained, and the step of calling the machine learning model to identify the first feature information of the first key video frame in response to obtaining the first key video frame of the screen video stream is returned; wherein, the fourth key video frame is the next key video frame after the first key video frame.

[0104] In other words, if the first semantic event and the second semantic event are the same, the semantic event corresponding to the next key video frame is directly determined instead of the first semantic event, thereby improving the efficiency of processing screen video streams.

[0105] It should be noted that the various embodiments described in this application can be combined with each other or implemented individually without conflict, and this application does not limit this.

[0106] For ease of understanding, a specific embodiment will be used as an example: like Figure 5 As shown, Figure 5 This is the second flowchart illustrating the screen video stream processing method provided in this application embodiment. The specific process is as follows: First, the sending terminal records and samples the screen: After the sending terminal starts, it begins to capture the screen image in real time and encodes the image for video. Simultaneously, an intelligent frame extraction strategy is used to extract key video frames. The frame extraction strategy includes: Basic strategy (fixed sampling): Set a fixed analysis rate (e.g., 1 frame per second) to sample the screen video stream at fixed time intervals. This is sufficient for most system state recognition needs.

[0107] Advanced Strategy (Key Video Frame Analysis): At the decoding unit level, perceptual hashes or structural similarities are compared between consecutive frames, or motion vectors in the screen video stream are analyzed. If multiple consecutive frames show minimal changes (e.g., remaining on a static document or video interface), they are considered redundant data and not triggered for further analysis. When a significant change in the frame is detected (e.g., application switching, pop-up, page scrolling), that frame is immediately captured and marked as a "key video frame."

[0108] The sending terminal pushes key video frames to the server.

[0109] Secondly, the server receives key video frames.

[0110] The server inputs key video frames into the Artificial Intelligence (AI) analysis module.

[0111] This is the core processing loop of the system. The AI ​​analysis module continuously receives key video frames from the sending terminal and calls the AI ​​model to extract multi-dimensional features from these key video frames. The extracted features include: ① Application identification: Based on the metadata (such as application package name) attached when the sending terminal pushes the frame, or through visual model analysis when the metadata is missing, determine the category of the application currently running in the foreground (e.g., "productivity application", "communication application" or "entertainment application").

[0112] ② Content Classification: Determine the main visual content of the screen (e.g., "game screen", "streaming video", "instant messaging interface" or "text and image data").

[0113] The server then inputs the feature information output by the AI ​​analysis module into the intelligent decision engine.

[0114] The previous AI analysis module outputs fragmented, multi-dimensional raw features (e.g., {app:"browser"}, {content: "game forum"}).

[0115] These scattered features are then sent to the intelligent decision engine. The intelligent decision engine is responsible for "judging" these scattered features as a single, high-information-value semantic event.

[0116] I. Module Positioning The intelligent decision engine is responsible for analyzing and making decisions based on the feature information processed by the preceding modules such as application identification and content classification, and providing a final semantic event conclusion. The problems that need to be solved include: 1. Resolving Ambiguity: If conflicting feature information arises during the initial processing—for example, the application recognition module identifies it as a "learning" application, but the content classification module identifies it as a "game" interface—the system needs a decision-making mechanism.

[0117] 2. Semantic transformation: The scattered and meaningless UI elements identified in the content classification (such as {UI:[input box]}, {OCR:"Report"}) are integrated and judged into a unified and identifiable "semantic event" (such as: work report).

[0118] 3. Data noise reduction: Filter out certain instantaneous and meaningless feature jitters, such as flashing pop-up ads, to prevent them from polluting the final event log.

[0119] II. Module Implementation This module consists of three core components: a knowledge base (i.e., a rule-based knowledge base), an inference engine, and a rule-based state machine. Its position and workflow within the overall system are as follows: Figure 6 As shown, Figure 6 This is a schematic diagram of a server structure provided in an embodiment of this application.

[0120] 1. Knowledge Base: This is the foundation of all decisions. It consists of a series of rules, each defining the mapping relationship between "feature combinations" and "semantic events," and includes a "weight" or "priority." Below are some example rules.

[0121] Rule 1: [Judgment: Game] (Highest priority / weight 100).

[0122] Rule description: Used to identify game behavior.

[0123] Triggering conditions (feature extraction output from the AI ​​analysis module, any one of these conditions must be met): Feature 1 (UI element detection): "Discover virtual joystick" or "Discover in-game buttons".

[0124] Feature 2 (OCR recognition): "Recognized the text 'Start Game', 'Game Over', or 'Enemy'".

[0125] Feature 3 (Application Identification): "The current application is a known game app (such as com.tencent.tmgp.sgame)".

[0126] Rule 2: [Judgment: Professional Work] (High Priority / Weight 80).

[0127] Rule description: Used to identify high-deterministic working applications.

[0128] Triggering conditions (feature extraction output from the AI ​​analysis module must all be met): Feature 1 (Application Identification): "The current application is a known working application (such as Office, WPS, integrated development tools)".

[0129] Feature 2 (OCR recognition): "Recognized text 'meeting minutes', 'flowchart', and 'code'".

[0130] Rule 3: [Decision: Instant Messaging] (Medium Priority / Weight 50).

[0131] Rule description: Used to identify chat behavior.

[0132] Triggering conditions (feature extraction output from the AI ​​analysis module must all be met): Feature 1 (UI element detection): "Discover chat bubble interface".

[0133] Feature 2 (OCR recognition): "Recognizes high-frequency chat words (such as 'hahaha', 'game', 'restaurant')".

[0134] Feature 3 (Application Identification): "The current application is a communication APP (such as WeChat, QQ)".

[0135] Rule 4: [Judgment: Online Learning] (Medium Priority / Weight 40).

[0136] Rule Description: This rule is specifically designed to resolve ambiguity—separating educational videos from entertainment videos. It must take precedence over the "Entertainment Videos" rule below.

[0137] Triggering conditions (feature extraction output from the AI ​​analysis module must all be met): Feature 1 (UI element detection): "Discover video player interface".

[0138] Feature 2 (OCR recognition): "Recognizes specific learning words (such as 'Chapter 3', 'Lecture', 'python')".

[0139] Rule 5: [Judgment: Entertainment Video] (Low Priority / Weight 20).

[0140] Rule description: This is a "general" rule used to judge "non-learning" videos or full-screen content.

[0141] Triggering conditions (feature extraction output from the AI ​​analysis module must all be met): Feature 1 (UI element detection): "Detects video player interface (and its area is more than 30% larger than the screen)".

[0142] (Exclusionary condition) Feature 2 (OCR recognition) failed to recognize the “learned vocabulary” in Rule 4.

[0143] Rule Six: [Decision: Unknown / Other] (Lowest Priority / Weight 0).

[0144] Rule description: "Catch-all" rule.

[0145] Triggering condition: None of the above rules are met.

[0146] 2. Inference Engine: This is the software logic that executes the rules. It receives all low-level features from the AI ​​pipeline (AI analysis module), then iterates through all rules in the "knowledge base" to check which rules the current feature combination matches. If multiple rules are matched simultaneously, the inference engine selects the rule with the highest priority as the "candidate event" for this instance.

[0147] 3. State Machine: Used to solve the "data interference" problem. It maintains two variables in memory: the currently confirmed event and the start time of that event.

[0148] When the inference engine sends a "candidate event", the state machine does not output immediately, but performs a "state check": Case A (State unchanged): If candidate event == Current_Event, then do nothing (to prevent log flooding).

[0149] Case B (State Change): If candidate event != Current_Event, then start a short "acknowledgment timer" (e.g., 3 seconds). If, within 3 seconds, all candidate events received are this new event, then it acknowledges the state change and officially outputs the new semantic event.

[0150] This allows us to filter out momentary distractions such as advertisements and pop-up notifications.

[0151] III. Work Process The following example illustrates the complete workflow in a specific scenario.

[0152] Scenario: The sending terminal is using a browser to view technical documents when a game advertisement pop-up suddenly appears and is closed after 2 seconds.

[0153] The following explanation uses a 1-second frame extraction interval as an example: 10:00:00 AM (has been ongoing for 10 seconds): Input to the AI ​​analysis module: {app:"Chrome", ocr:"Technical Documentation", ui:"text_area"}.

[0154] Inference engine: Matches the knowledge base and hits the "working" rule (priority 50).

[0155] State machine: Current_Event is "Working". The state remains unchanged, and there is no output.

[0156] 2. At 10:00:01 AM (an ad pop-up appears): AI analysis module input: {app:"Chrome", ocr:"Start Game", ui:"popup_ad"}.

[0157] Inference engine: Matches the knowledge base and hits both "in the game" (priority 100) and "at work" (priority 50, because the app is still a browser).

[0158] Conflict resolution: The engine selects the highest priority "in-game" event as the candidate event.

[0159] State machine: A state change (working -> playing) has been detected. Start the "Confirmation Timer" but do not output anything yet.

[0160] 3. 10:00:03 AM (Advertisement still running): AI pipeline input: {app:"Chrome", ocr:"Start Game", ui:"popup_ad"}.

[0161] Inference engine: Output "in the game" as a candidate event again.

[0162] State machine: The timer is still running (less than 3 seconds).

[0163] 4. 10:00:04 AM (Advertisement ends): AI pipeline input: {app:"Chrome", ocr:"Technical Documentation", ui:"text_area"}.

[0164] Inference engine: Output "working" as a candidate event again.

[0165] State machine: The candidate event was detected and the status changed back to "working". The "acknowledgment timer" was reset.

[0166] Final result: Based on the analysis of the AI ​​module and the judgment of the intelligent decision engine, the sending terminal was "working" at 10:00 AM. Although a "game ad pop-up" appeared during this period, it was successfully filtered out because the event did not last for more than 3 seconds, thus avoiding the incorrect generation of semantic events.

[0167] Then, the server inputs the semantic events output by the intelligent decision engine into the log generation unit.

[0168] The "semantic events" output by the decision engine are sent to the log generation unit, which stores these timestamped semantic events into the database and forms a machine-readable, structured "event log" in chronological order.

[0169] Finally, the server pushes the logs.

[0170] When preset triggering conditions are met, such as: a. reaching a fixed time period (e.g., 1 hour); b. the intelligent decision engine determines a "predefined important event"), the push service unit sends the "event log" for that time period to the receiving terminal through the mobile push channel.

[0171] After receiving the event log, the receiving terminal presents (i.e. displays) it: When a user receives a push notification and clicks to view the notification, the receiving terminal will parse and present the event log based on the received log.

[0172] See Figure 7 , Figure 7 This is the third flowchart illustrating the screen video stream processing method provided in this application embodiment. The screen video stream processing method of this application embodiment can be applied to the receiving terminal in the above embodiments (as described above). Figure 1 In the receiving terminal 106, the method can be executed by the receiving terminal, or by a component of the receiving terminal, such as the processor, chip, or chip system of the receiving terminal, or by a logic module or software that implements all or part of the functions of the receiving terminal.

[0173] like Figure 7 As shown, the screen video stream processing method includes, but is not limited to, the following steps: Step 701: In response to obtaining the event log corresponding to the first key video frame of the screen video stream, display the event log; The event log is generated from the first semantic event and the timestamp carried by the first key video frame; the first semantic event is determined from multiple semantic events corresponding to the screen video stream by performing semantic decision on the first feature information corresponding to the first key video frame; the first feature information is obtained by recognizing the first key video frame through a machine learning model; and the screen video stream is obtained by recording the screen through a receiving terminal or a sending terminal.

[0174] It should be noted that this embodiment is an embodiment applied to a receiving terminal, corresponding to the method embodiment applied to a server described above. Therefore, the relevant descriptions in the method embodiment applied to a server described above can be referred to, and the same beneficial effects can be achieved. To avoid repetition, further details will not be provided here.

[0175] This application also provides an electronic device, such as... Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 800 includes: One or more processors 810; The memory 820 stores one or more programs that, when executed by one or more processors 810, cause the one or more processors 810 to implement the screen video stream processing method described in any of the above embodiments.

[0176] The memory 820, as a non-transitory network system, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, the memory 820 may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 820 may optionally include remotely located memories 820 relative to the processor 810, which can be connected to the processor 810 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0177] The memory 820 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 820 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 820 and is called and executed by the processor 810.

[0178] The processor 810 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0179] In some embodiments, the electronic device further includes: Input / output interfaces are used to implement information input and output; The communication interface is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). The bus transmits information between various components of the device, such as the processor 810, memory 820, input / output interfaces, and communication interfaces. The processor 810, memory 820, input / output interface, and communication interface can communicate with each other within the device via a bus.

[0180] One embodiment of this application also provides a computer-readable storage medium storing computer-executable instructions for performing the screen video stream processing method provided in any embodiment of this application.

[0181] An embodiment of this application also provides a computer program product, including a computer program or computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the computer device to perform the screen video stream processing method provided in any embodiment of this application.

[0182] The system architecture and application scenarios described in this application are intended to more clearly illustrate the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. Those skilled in the art will understand that as system architectures evolve and new application scenarios emerge, the technical solutions provided in this application are also applicable to similar technical problems.

[0183] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0184] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0185] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process or execution thread, and components may be located on a single computer or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, or a network, such as the Internet interacting with other systems via signals).

[0186] The above description, with reference to the accompanying drawings, illustrates some embodiments of this application, but does not limit the scope of this application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of this application shall be within the scope of this application.

Claims

1. A screen video stream processing method, applied to a server, comprising: In response to acquiring the first key video frame of the screen video stream, a machine learning model is invoked to identify the first feature information of the first key video frame; Semantic decision-making is performed on the first feature information to determine the first semantic event corresponding to the first key video frame from multiple semantic events corresponding to the screen video stream.

2. The screen video stream processing method according to claim 1, characterized in that, The first feature information includes at least one of the following: user interface element information of the first key video frame, character recognition information of the first key video frame, and application category information corresponding to the first key video frame.

3. The screen video stream processing method according to claim 1, characterized in that, The step of performing semantic decision-making on the first feature information, and determining the first semantic event corresponding to the first key video frame from multiple semantic events corresponding to the screen video stream, includes: Obtain the rule knowledge base corresponding to the screen video stream; wherein, the rule knowledge base includes multiple rules, each rule corresponds to a trigger condition and a semantic event; the trigger condition corresponding to each rule is used to define the feature information that the semantic event corresponding to each rule must satisfy; Obtain a first rule that matches the first feature information from the rule knowledge base; wherein, the first feature information matching the first rule indicates that the first feature information satisfies the triggering condition corresponding to the first rule; The semantic event corresponding to the first rule is determined as the first semantic event.

4. The method according to claim 3, characterized in that, The step of obtaining a first rule matching the first feature information from the rule knowledge base includes: In response to the rule knowledge base including multiple candidate rules that match the first feature information, the candidate rule with the highest priority among the multiple candidate rules is determined as the first rule.

5. The method according to claim 3, characterized in that, After determining the semantic event corresponding to the first rule as the first semantic event, the method further includes: The first semantic event is compared with the second semantic event corresponding to the second key video frame of the screen video stream, wherein the second key video frame is the previous key video frame of the first key video frame. In response to the first semantic event being different from the second semantic event, a timer is started to begin counting. In response to the timer reaching a preset duration, and the semantic events corresponding to the third key video frame of the screen video stream acquired during the timer's timing period being the first semantic event, the first semantic event is output.

6. The method according to claim 5, characterized in that, The output of the first semantic event includes: An event log is generated based on the first semantic event and the timestamp carried by the first key video frame; Send the event log to the receiving terminal.

7. The screen video stream processing method according to claim 5, characterized in that, After comparing the first semantic event with the second semantic event corresponding to the second key video frame of the screen video stream, the method further includes: In response to the first semantic event being the same as the second semantic event, the fourth key video frame of the screen video stream is obtained, and the step of calling a machine learning model to identify the first feature information of the first key video frame in response to obtaining the first key video frame of the screen video stream is returned; wherein, the fourth key video frame is the next key video frame after the first key video frame.

8. A screen video stream processing method, applied to a receiving terminal, comprising: In response to acquiring the event log corresponding to the first key video frame of the screen video stream, the event log is displayed; The event log is generated from a first semantic event and a timestamp carried by the first key video frame. The first semantic event is determined from multiple semantic events corresponding to the screen video stream by performing semantic decision-making on the first feature information corresponding to the first key video frame. The first feature information is obtained by recognizing the first key video frame through a machine learning model. The screen video stream is obtained by recording the screen through the receiving terminal or the sending terminal.

9. An electronic device, comprising: One or more processors; A memory having stored one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the screen video stream processing method as described in any one of claims 1 to 7, or the screen video stream processing method as described in claim 8.

10. A computer-readable storage medium storing computer-executable instructions for performing the screen video stream processing method according to any one of claims 1 to 7, or the screen video stream processing method according to claim 8.

11. A computer program product comprising a computer program or computer instructions stored in a computer-readable storage medium, wherein a processor of an electronic device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions to cause a server to perform the screen video stream processing method of any one of claims 1 to 7, or the screen video stream processing method of claim 8.

Images