Web-based methods, devices, and computer programs for adding video effects.

Abstract

This disclosure provides a method and apparatus for adding video effects. At a target application, an original video frame can be obtained from a video source, and the original video frame can be provided to a video effects processing application, wherein the video effects processing application is a web application. A processed video frame with video effects applied can also be obtained from the video effects processing application. At the video effects processing application, the original video frame can be obtained from the target application, video effects can be applied to the original video frame to obtain a processed video frame, and the processed video frame can be provided to the target application.

Description

Background Technology

[0001] In video-related scenarios, there is a need to add video effects. Users may want to apply video effects while using video-related applications. In this article, video-related applications can refer to various applications involving operations such as video capture, transmission, and presentation, such as online conferencing applications and video streaming services. Furthermore, applying video effects can broadly refer to editing or changing pixels in a video frame, such as performing operations like deleting, replacing, adding, or modifying pixels. Taking online conferencing applications as an example, meeting participants may want to apply video effects to the raw video footage captured by their cameras, such as applying filters, to transform the raw video footage into one with the desired video effect. Summary of the Invention

[0002] This invention is provided to introduce a set of concepts, which will be further described in the following detailed description. This invention is not intended to identify key or essential features of the protected subject matter, nor is it intended to limit the scope of the protected subject matter.

[0003] Embodiments of this disclosure provide methods and apparatus for adding video effects. In some embodiments, at a target application, an original video frame can be obtained from a video source, the original video frame can be provided to a video effects processing application, wherein the video effects processing application is a web application, and a processed video frame with applied video effects can be obtained from the video effects processing application. In some embodiments, at a video effects processing application, an original video frame can be obtained from a target application, video effects can be applied to the original video frame to obtain a processed video frame, and the processed video frame can be provided to the target application.

[0004] It should be noted that one or more of the above aspects include the features specifically pointed out in the following detailed description and claims. Certain illustrative features of the one or more aspects are set forth in detail in the following specification and drawings. These features merely indicate various ways in which the principles of each aspect can be implemented, and this disclosure is intended to include all such aspects and their equivalents. Attached Figure Description

[0005] The following description will take into account several aspects disclosed, which are provided to illustrate rather than limit the aspects disclosed.

[0006] Figure 1 An exemplary architecture for implementing video effects addition is shown according to an embodiment.

[0007] Figure 2 An exemplary process for adding video effects according to an embodiment is shown.

[0008] Figure 3 An exemplary process for adding video effects is shown according to an embodiment, in the case where the target application is a web application.

[0009] Figure 4 An exemplary process for adding video effects is shown according to an embodiment, in the case where the target application is a desktop application.

[0010] Figure 5 An exemplary process for adding video effects is shown according to an embodiment, in the case where the target application is a desktop application.

[0011] Figures 6A to 6D An exemplary user interface for adding video effects according to an embodiment is shown.

[0012] Figure 7 A flowchart of an exemplary method for implementing video effects addition according to an embodiment is shown.

[0013] Figure 8 A flowchart of an exemplary method for implementing video effects addition according to an embodiment is shown.

[0014] Figure 9 An exemplary apparatus for adding video effects according to an embodiment is shown.

[0015] Figure 10 An exemplary apparatus for adding video effects according to an embodiment is shown.

[0016] Figure 11 An exemplary apparatus for adding video effects according to an embodiment is shown. Detailed Implementation

[0017] This disclosure will now be discussed with reference to various exemplary embodiments. It should be understood that this discussion of embodiments is merely intended to enable those skilled in the art to better understand and thus implement the embodiments of this disclosure, and is not intended to teach any limitation on the scope of this disclosure.

[0018] Because video effects are performance-sensitive, traditional video effects are typically implemented as binary code compiled from programming languages ​​such as C / C++. These binary codes corresponding to video effects form a separate binary code library. To provide video effect addition functionality in video-related applications, the binary code library for applying the video effect needs to be downloaded and installed beforehand. This binary code library is embedded in the video-related application and serves as the underlying code for providing video effect addition functionality. When a video effect is applied, the corresponding binary code needs to be run on the local machine. The installation and execution of the binary code require significant effort to avoid security risks. Furthermore, the distribution and upgrade process of the binary code library is complex. To provide video effect addition functionality, developers of video-related applications need to invest considerable effort in developing various application features to handle the downloading, revision, updating, and execution of the binary code library used for applying video effects.

[0019] This disclosure proposes a web-based method for adding video effects. For example, a web-based video effects processing application can be used to assist a target application in adding video effects. The target application can broadly refer to various video-related applications that require video effects, such as online conferencing applications, video streaming services, and video editing applications. The target application can be a web application, a desktop application, etc. The video effects processing application can utilize a pre-prepared video effects library and video effects algorithms to apply video effects to video frames. The video effects processing application can be a web application and can be a regular web page running within a web content container. For example, the video effects processing application can be a web page implemented using HTML, Cascading Style Sheets (CSS), Javascript, etc. Thus, the video effects processing application can be loaded at any time without installation. The web content container can refer to various web-based containers used to load or run applications, such as iframes (inline frames), Webviews (web page views), etc., where an iframe is the HTML carrier of a web page and is used to load a web page in a web page or browser environment, and a Webview is the native carrier of a web page and is used to load a web page in a native environment. When video effects need to be applied to video frames, a video effects processing application can be loaded into the target application through a web content container. This allows the video effects processing application to apply video effects to the video frames provided by the target application and return the processed video frames with applied effects to the target application. Therefore, the target application can also be considered as the host application of the video effects processing application.

[0020] Unlike existing video-related applications that include binary code libraries as part of their functionality to add video effects, the web-based video effects processing application according to embodiments of this disclosure can be a third-party application, which can be loaded by the target application at any time without installation. Therefore, the target application can easily access various video effects processing applications developed by different third parties to achieve greater scalability in adding video effects and provide users with a wider variety of video effects. Furthermore, since there is no need to install and run the binary code corresponding to the video effects on the local machine, embodiments of this disclosure can effectively avoid security risks. Moreover, the video effects adding processing of the web-based video effects processing application can be managed and executed, for example, in a browser. Browsers have their own security mechanisms to prevent malicious access to local systems and resources by the running scripts; thus, embodiments of this disclosure can further effectively avoid security risks.

[0021] With the development of Web technologies such as WebAssembly, WebGL, and WebGPU, video effects that run in browsers can be achieved efficiently. Consequently, the development, deployment, and upgrading of web-based video effects processing applications will become easier. Furthermore, the widespread use of various Web technologies will encourage more third parties to develop more video effects for target applications.

[0022] When the target application is a desktop application, shared memory can be used to efficiently transfer raw and processed video frames between the target application and the video effects processing application. Inter-process communication can be used to achieve synchronous access to the shared memory by both the target application and the video effects processing application. Furthermore, at least a custom application programming interface (API) can be provided for the video effects processing application to access the shared memory. By using shared memory, embodiments of this disclosure can achieve better video frame transfer performance, thereby effectively improving the execution efficiency of adding video effects.

[0023] When the target application is a desktop application, the video effects processing application can run in a sandbox environment, such as in a sandboxed web content container, thereby further and more effectively avoiding security risks.

[0024] When the target application is a web application, raw and processed video frames can be efficiently transferred between the target application and the video effects processing application using a transferable array buffer.

[0025] The embodiments disclosed herein can easily implement video effects addition in any target application that requires video effects, can achieve stronger scalability of video effect addition, can effectively avoid security risks, can have high execution efficiency of video effect addition, and so on.

[0026] In this paper, a video frame can be understood as a unit of video; that is, a video can include multiple video frames. Therefore, video effects can be added to a video by applying video effects to each video frame. Although the application of video effects to video frames is described in several places, it should be understood that video effects can be applied to each video frame in a similar manner to add video effects to the entire video. Therefore, processing video frames is equivalent to processing the video itself, and correspondingly, the terms "video frame" and "video" can be used interchangeably in many places throughout this paper. Furthermore, in this paper, the term "application" can broadly refer to application programs, application software, widgets, program code, etc.

[0027] Figure 1 An exemplary architecture 100 for implementing video effects addition is shown according to an embodiment.

[0028] In architecture 100, the target application 110 can be any video-related application that requires video effects to be added. The video effects processing application 120 is a web application that can be loaded by the target application 110 to apply video effects to video frames.

[0029] Target application 110 can obtain raw video frames from video source 102. Video source 102 can refer to various sources capable of providing video frames. For example, video source 102 can be a camera, and accordingly, target application 110 can obtain the video captured by the camera as raw video frames. For example, video source 102 can be a video file stored locally or on a network, and accordingly, target application 110 can obtain the video content of the video file as raw video frames. For example, video source 102 can be a video stream received during the operation of target application 110, and accordingly, target application 110 can use video frames from the video stream as raw video frames. It should be understood that, although in Figure 1 Video source 102 is shown as independent of target application 110, but video source 102 may also be included in target application 110. Furthermore, in this document, original video frame refers to the video frame obtained by target application 110 from video source 102, which may be an unedited video frame or an edited video frame.

[0030] The target application 110 may include a video acquisition module 112, a video consuming module 114, etc. The video acquisition module 112 can be used to acquire video frames required by the target application 110. In one aspect, the video acquisition module 112 can acquire raw video frames from the video source 102 and transmit the raw video frames to the video effects processing application 120. In another aspect, the video acquisition module 112 can acquire processed video frames with applied video effects from the video effects processing application 120.

[0031] The video consumption module 114 can use or process the processed video frames acquired by the video acquisition module 112 in different ways under different scenarios. For example, when the target application 110 needs to send the processed video frames over the network, the video consumption module 114 can perform encoding, uploading, and other processing on the processed video frames. For example, when the target application 110 needs to present the processed video frames to the user, the video consumption module 114 can present the processed video frames through a user interface (UI). For example, when the target application 110 needs to store the processed video frames, the video consumption module 114 can store the processed video frames in a predetermined location. For example, when the target application 110 needs to further edit the processed video frames, the video consumption module 114 can perform various editing operations on the processed video frames. For example, when the target application 110 needs to provide the processed video frames to other video processing applications, the video consumption module 114 can send the processed video frames to other video processing applications. It should be understood that the embodiments of this disclosure are not limited to how the video consumption module 114 specifically uses or processes the processed video frames.

[0032] Although not shown, the target application 110 may also include any other functional modules. For example, the target application 110 may include a UI module through which the user 104 can interact with the target application 110 in various ways.

[0033] The video effects processing application 120 may include a video effects application module 122, a UI module 124, etc. The video effects application module 122 can apply video effects to the original video frame using various video effects application techniques. In one implementation, the video effects application module 122 can utilize a pre-prepared video effects library and video effects algorithms to apply video effects to the original video frame. The video effects library includes multiple video effects materials that can be applied to the video frame. The video effects algorithm can be executed to determine the specific way to apply the video effects to the video frame. The developer of the video effects processing application 120 can pre-prepare the video effects library and video effects algorithms in any manner. The embodiments of this disclosure are not limited to any specific technique used by the video effects processing application 120 to apply video effects to the video frame. Furthermore, the present invention is not limited to any specific form of video effect, but can cover video effects obtained by arbitrarily editing or changing pixels in a video frame.

[0034] UI module 124 can be used to enable various ways of interaction between user 104 and video effects processing application 120. For example, user 104 can use UI module 124 to input instructions about video effects, indicating how user 104 wants video effects application module 122 to apply video effects.

[0035] Although not shown, the video effects processing application 120 may also include any other functional modules.

[0036] To further illustrate architecture 100, let's take an online meeting scenario as an example. The target application 110 can be an online meeting application, user 104 can be a meeting participant, and video source 102 can be the camera on user 104's terminal device. Assume the meeting participant is attending a meeting and has their camera on. The online meeting application can acquire the video image of the meeting participant captured by the camera and display it in the meeting interface. If the meeting participant requests the online meeting application to apply video effects to their own video image, such as applying a filter, the online meeting application can load video effects processing application 120. Video effects processing application 120 can receive instructions from the meeting participant regarding specific video effects, such as instructions for a specific filter, through the user interface and can apply the corresponding video effect to the meeting participant's video image. Then, the online meeting application will display the video image of the meeting participant with the applied video effect in the meeting interface. It should be understood that the above online meeting scenario is merely an exemplary implementation of architecture 100, and architecture 100 can also be implemented for any other scenario involving the addition of video effects, such as video streaming service scenarios, video editing scenarios, etc. In a video streaming service scenario, the target application 110 can be a video streaming service application, which adds video effects to video frames in the video stream by loading the video effects processing application 120. In a video editing scenario, the target application 110 can be a video editing application, which adds video effects to input video frames by loading the video effects processing application 120.

[0037] Figure 2 An exemplary process 200 for adding video effects according to an embodiment is shown. Process 200 is based on... Figure 1 The exemplary process for adding video effects is proposed in Architecture 100.

[0038] At position 202, target application 110 can obtain the original video frame from video source 102.

[0039] At point 204, the raw video frame can be transferred from the target application 110 to the video effects processing application 120. For example, the target application 110 can provide the raw video frame to the video effects processing application 120, and the video effects processing application 120 can obtain the raw video frame from the target application 110.

[0040] Optionally, at 206, the video effects processing application 120 can receive instructions about video effects via a user interface. For example, the UI module 124 can receive instructions about video effects from the user. The user can indicate which video effect they want to apply to the original video frame by inputting the instructions. In other implementations, receiving instructions about video effects at 206 can be omitted, so that the video effects processing application 120 can determine which video effect to apply to the original video frame based on any predetermined rules.

[0041] At 208, the video effects processing application 120 can apply video effects to the original video frame to obtain a processed video frame. For example, the video effects application module 122 can apply video effects to the original video frame using various video effects application techniques. If an instruction regarding a video effect is received at 206, the video effects application module 122 can apply the video effect specified by the instruction to the original video frame, at least based on the instruction.

[0042] At point 210, a processed video frame with applied video effects can be transmitted from the video effects processing application 120 to the target application 110. For example, the video effects processing application 120 can provide the target application 110 with the processed video frame with applied video effects, and the target application 110 can obtain the processed video frame with applied video effects from the video effects processing application 120.

[0043] It should be understood that Figure 2 This only illustrates an exemplary overall process 200 for adding video effects. When implemented differently in the target application 110, the specific process for adding video effects may include more details. These will be discussed later in conjunction with... Figures 3 to 5 Further explanation is provided.

[0044] Figure 3 An exemplary process 300 for adding video effects is shown according to an embodiment, in the case where the target application is a web application.

[0045] exist Figure 3 In this context, "web-based target application 110" indicates that the target application is a web application. Therefore, the target application 110 can be, for example, a web page running in a browser. The web page corresponding to the target application 110 can also be referred to as the main web page.

[0046] In this scenario, the video effects processing application 120 can be an embedded web page running within the main web page of the target application 110. In one implementation, the web content container running the video effects processing application 120 can be, for example, an iframe. By using an iframe, the video effects processing application 120 can be loaded within the target application 110, for example, by loading an embedded web page of the video effects processing application 120 within the main web page of the target application 110.

[0047] According to embodiments of this disclosure, the video effects processing application 120 can be loaded and run in the target application 110 by defining a URL associated with the video effects processing application 120 in an iframe.

[0048] In one implementation, a URL corresponding to the video effects processing application 120 can be defined in an iframe. By accessing this URL, various UI data required by the UI module 124 of the video effects processing application 120 can be downloaded from the server of the video effects processing application 120, so as to realize the UI display of the video effects processing application 120 on the local machine.

[0049] In one implementation, a separate URL for each video effect can be defined within an iframe. By accessing this URL, data related to the video effect used by the video effect application module 122 can be downloaded from the server of the video effect processing application 120 to the local machine and executed thereon. This video effect-related data may include the video effect algorithm corresponding to the video effect, video effect materials, and any other related content. Thus, the video effect processing application 120 can use the downloaded video effect-related data to apply video effects to video frames locally. It should be understood that the video effect-related data corresponding to a video effect can be cached locally and directly used to apply the video effect to multiple video frames, without needing to repeatedly download the video effect-related data for each video frame unless the video effect-related data is updated on the server. This effectively improves execution efficiency. Furthermore, whenever a new video frame needs to be applied, for example, the video effect algorithm from the video effect-related data can be loaded into the process for execution to apply the video effect to the new video frame. The video effect algorithm can be implemented using, for example, Javascript, and can therefore run in a web page and be managed by, for example, a browser.

[0050] In one implementation, a URL corresponding to the video effects processing application 120 can be defined in an iframe. By accessing this URL, various UI data required by the UI module 124 and multiple video effect-related data corresponding to multiple video effects used by the video effects application module 122 can be downloaded from the server of the video effects processing application 120. This allows the UI display of the video effects processing application 120 to be implemented on the local machine, and the multiple video effect-related data can be cached. When a certain video effect needs to be applied to a video frame, the video effect algorithm in the cached video effect-related data corresponding to that video effect can be directly loaded into the process for execution, so as to apply the video effect to that video frame.

[0051] At step 302, the target application 110 can acquire raw video frames from video source 102. For example, video acquisition module 112 can acquire raw video frames from video source 102. Step 302 is... Figure 2 An example of step 202 in the process.

[0052] At point 304, the raw video frame can be transferred from target application 110 to video effects processing application 120. For example, the raw video frame can be transferred from video acquisition module 112 to video effects application module 122. In one implementation, at point 304, the raw video frame can be transferred from target application 110 to video effects processing application 120 in a transitive ArrayBuffer manner. For example, target application 110 can provide the raw video frame to video effects processing application 120 in a transitive ArrayBuffer manner, and video effects processing application 120 can acquire the raw video frame from target application 110 in a transitive ArrayBuffer manner. To enable the transfer of video frames between the main web page of target application 110 and the embedded web page of video effects processing application 120, a transfer function such as PostMessage for transferring data between pages can be used. The transfer mode can be set to transitive ArrayBuffer in the adopted transfer function so that video frames can be transferred in a transitive ArrayBuffer manner. Compared to other transmission methods applicable to the transfer function, the ArrayBuffer transfer method does not require serialization, encoding, or decoding of video frames, thus achieving faster transmission speeds. Step 304 is... Figure 2 An example of step 204 in the process.

[0053] Optionally, at 306, the video effects processing application 120 can receive instructions about video effects through a user interface. Step 306 is... Figure 2 An example of step 206 in the process.

[0054] At 308, the video effects processing application 120 can apply video effects to the original video frame to obtain a processed video frame. For example, the video effects application module 122 can apply video effects to the original video frame to obtain a processed video frame. If an instruction regarding a video effect is received at 306, the video effects application module 122 can apply the video effect specified by the instruction to the original video frame, at least based on the instruction. Step 308 is... Figure 2 An example of step 208 in the process.

[0055] At step 310, the processed video frame with applied video effects can be transferred from the video effects processing application 120 to the target application 110. For example, the processed video frame can be transferred from the video effects application module 122 to the video acquisition module 112. In one implementation, at step 310, the processed video frame can be transferred from the video effects processing application 120 to the target application 110 in a transitive ArrayBuffer manner. For example, the video effects processing application 120 can provide the processed video frame with applied video effects to the target application 110 in a transitive ArrayBuffer manner, and the target application 110 can acquire the processed video frame with applied video effects from the video effects processing application 120 in a transitive ArrayBuffer manner. Step 310 is... Figure 2 An example of step 210 in the process.

[0056] Figure 4 An exemplary process 400 for adding video effects is shown according to an embodiment, in the case where the target application is a desktop application.

[0057] exist Figure 4 In this context, "desktop-based target application 110" indicates that the target application is a desktop application. A desktop application can broadly refer to a locally installed application or application client. Specifically, the desktop-based target application 110 in process 400 can be an application created based on, for example, Electron. Therefore, the target application 110 can correspond to, for example, an Electron rendering process.

[0058] In this scenario, the video effects processing application 120 can run within a web content container such as a WebView, for example, in an Electron WebView. By using a WebView, the video effects processing application 120 can be loaded into a desktop-based target application 110. In conjunction with the above... Figure 3Similar to the method described in which a URL is defined in an iframe to load and run the video effects processing application 120 in the target application 110, a URL associated with the video effects processing application 120 can be defined in a Webview to download UI data, implement UI display, download video effects-related data, run video effects algorithms, etc.

[0059] In process 400, raw and processed video frames can be efficiently transferred between the target application 110 and the video effects processing application 120 via shared memory. Synchronous access to the shared memory by the target application 110 and the video effects processing application 120 can be achieved through inter-process communication. Furthermore, custom APIs can be configured for both the target application 110 and the video effects processing application 120, at least for accessing the shared memory.

[0060] At step 402, target application 110 can acquire raw video frames from video source 102. For example, video acquisition module 112 can acquire raw video frames from video source 102. Step 402 is... Figure 2 An example of step 202 in the process.

[0061] At position 404, the video acquisition module 112 can provide the raw video frames to the target application 110's custom API 116. Custom API 116 can be used to create shared memory 430, enable the target application 110 to access shared memory 430, etc. Custom API 116 can be implemented using various technologies, such as the Electron API, Node.js Addon API, etc.

[0062] At 406, target application 110 can store raw video frames into shared memory 430. For example, target application 110 can use custom API 116 to store raw video frames into shared memory 430. In one implementation, the memcpy (memory copy) function can be used to store raw video frames into shared memory 430.

[0063] The video effects processing application 120 may include a custom API 126. The custom API 126 can be used to enable the video effects processing application 120 to access shared memory 430, etc. The custom API 126 can be implemented using various technologies, such as the Electron API, Node.js Addon API, etc.

[0064] After the original video frame is stored in shared memory 430, video effects processing application 120 can use custom API 126 at 408 to obtain the storage information of the original video frame in shared memory 430. This storage information may include various information used to locate the original video frame in shared memory 430, such as the memory pointer and size of the original video frame.

[0065] According to process 400, the video effects application module 122 can apply video effects to the original video frame in the shared memory 430, based at least on the stored information.

[0066] In one implementation, at 410, the custom API 126 can generate a storage information indicator based on the acquired storage information and provide the storage information indicator to the video effects application module 122. The storage information indicator may include storage space information of the original video frame in shared memory 430 that the video effects application module 122 can understand. For example, the storage information indicator may use a SharedArrayBuffer.

[0067] Optionally, at 412, the video effects processing application 120 can receive instructions about video effects through a user interface. Step 412 is... Figure 2 An example of step 206 in the process.

[0068] At 414, the video effects application module 122 can directly apply video effects to the original video frame in shared memory 430 based on the storage information instruction generated according to the storage information, to obtain a processed video frame. If an instruction regarding a video effect is received at 412, the video effects application module 122 can apply the video effect specified by the instruction to the original video frame at least based on the instruction. Step 414 is Figure 2 An example of step 208 in the process.

[0069] After applying the video effect at point 414, the video effect application module 122 can send a notification to the custom API 126 at point 416 to inform the custom API 126 that it has completed the operation of applying the video effect to the original video frame.

[0070] At 418, target application 110 can read processed video frames from shared memory 430. For example, custom API 116 can read processed video frames from shared memory 430.

[0071] At 420, custom API 116 can provide processed video frames to video acquisition module 112.

[0072] In process 400, steps 404, 406, 408, 410, etc., can correspond to Figure 2 The operation at step 204 involves transferring the original video frame from the target application 110 to the video effects processing application 120. For example, the target application 110 can provide the original video frame to the video effects processing application 120 via shared memory 430, and the video effects processing application 120 can retrieve the original video frame from the target application 110 via shared memory 430. It should be understood that, due to the use of shared memory 430, the operation of the target application 110 providing the original video frame to the video effects processing application 120 can refer to the target application 110 storing the original video frame in shared memory 430 and providing the storage information of the original video frame in shared memory 430 to the video effects processing application 120, and the operation of the video effects processing application 120 retrieving the original video frame from the target application 110 can refer to the video effects processing application 120 retrieving the storage information. In process 400, since the video effects processing application 120 does not actually receive the original video frame, but accesses the original video frame in shared memory 430 at least based on stored information (e.g., based on a storage information indication generated based on the stored information), the time and resources required to actually send the original video frame to the video effects processing application 120 can be saved, thereby improving execution efficiency.

[0073] In process 400, step 418 can correspond to Figure 2 In step 210, the processed video frame is transferred from the video effects processing application 120 to the target application 110. For example, the target application 110 can obtain the processed video frame from the video effects processing application 120 via shared memory 430. It should be understood that, due to the use of shared memory 430, the operation of the target application 110 obtaining the processed video frame from the video effects processing application 120 can refer to the target application 110 reading the processed video frame from shared memory 430.

[0074] It should be understood that process 400 may also include, at 440, inter-process communication to enable synchronous access to shared memory 430 by the target application 110 and the video effects processing application 120. The inter-process communication at 440 can refer to communication between the processes of the target application 110 and the video effects processing application 120, which can be continuously executed so that when one of the target application 110 or the video effects processing application 120 accesses shared memory 430, causing a change in the state of shared memory 430, the other of the two applications can be promptly notified and proceed with the next operation. Inter-process communication can be performed between custom API 116 and custom API 126. For example, after custom API 116 stores the original video frame into shared memory 430 at 406, custom API 116 can notify custom API 126 via inter-process communication that it has completed storing the original video frame. Thus, custom API 126 can then retrieve the storage information at 408. For example, after the video effects application module 122 has completed the video effects application operation at 414 and notified the custom API 126 at 416, the custom API 126 can notify the custom API 116 via inter-process communication that the operation of applying video effects to the original video frame has been completed. Thus, the custom API 116 can then read the processed video frame from shared memory 430 at 418. The inter-process communication at 440 can be implemented using various techniques, such as event objects, sockets, semaphores, etc. The embodiments of this disclosure are not limited to any specific technique for implementing inter-process communication.

[0075] Figure 5 An exemplary process 500 for adding video effects is shown according to an embodiment, in the case where the target application is a desktop application.

[0076] exist Figure 5 In this context, "desktop-based target application 110" indicates that the target application is a desktop application. Specifically, the desktop-based target application 110 in process 500 can be an application created using native solutions such as C / C++. Therefore, the target application 110 can directly access system resources, such as creating and accessing shared memory, without using APIs.

[0077] In this scenario, the video effects processing application 120 can run within a web content container such as a WebView, for example, in an Electron WebView. By using a WebView, the video effects processing application 120 can be loaded into a desktop-based target application 110. In conjunction with the above... Figure 3 Similar to the method described in which a URL is defined in an iframe to load and run the video effects processing application 120 in the target application 110, a URL associated with the video effects processing application 120 can be defined in a Webview to download UI data, implement UI display, download video effects-related data, run video effects algorithms, etc.

[0078] In process 500, raw and processed video frames can be efficiently transferred between the target application 110 and the video effects processing application 120 via shared memory. Synchronous access to the shared memory by the target application 110 and the video effects processing application 120 can be achieved through inter-process communication. Furthermore, a custom API can be set up for the video effects processing application 120, at least for accessing the shared memory.

[0079] At step 502, the target application 110 can acquire raw video frames from video source 102. For example, video acquisition module 112 can acquire raw video frames from video source 102. Step 502 is... Figure 2 An example of step 202 in the process.

[0080] At 504, target application 110 can create shared memory 530 and store raw video frames in shared memory 530. For example, video acquisition module 112 can store raw video frames in shared memory 530.

[0081] The video effects processing application 120 may include a custom API 126. The custom API 126 can be used to enable the video effects processing application 120 to access shared memory 530, etc. The custom API 126 can be implemented using various technologies, such as the Electron API, Node.js Addon API, etc.

[0082] After the original video frame is stored in shared memory 530, video effects processing application 120 can use custom API 126 at 506 to obtain the storage information of the original video frame in shared memory 530. This storage information may include various information used to locate the original video frame in shared memory 530, such as the memory pointer and size of the original video frame.

[0083] According to process 500, the video effects application module 122 can apply video effects to the original video frame in the shared memory 530 based at least on the stored information.

[0084] In one implementation, at 508, the custom API 126 can generate a storage information indicator based on the acquired storage information and provide the storage information indicator to the video effects application module 122. The storage information indicator may include storage space information of the original video frame in shared memory 530 that the video effects application module 122 can understand. For example, the storage information indicator may use a SharedArrayBuffer.

[0085] Optionally, at step 510, the video effects processing application 120 can receive instructions regarding video effects through a user interface. Step 510 is... Figure 2 An example of step 206 in the process.

[0086] At 512, the video effects application module 122 can directly apply video effects to the original video frame in shared memory 530 based on the storage information instruction generated according to the storage information, to obtain a processed video frame. If an instruction regarding a video effect is received at 510, the video effects application module 122 can apply the video effect specified by the instruction to the original video frame at least based on the instruction. Step 512 is Figure 2 An example of step 208 in the process.

[0087] After the video effect is applied at point 512, the video effect application module 122 can send a notification to the custom API 126 at point 514 to inform the custom API 126 that it has completed the operation of applying the video effect to the original video frame.

[0088] At 516, target application 110 can read processed video frames from shared memory 530. For example, video acquisition module 112 can read processed video frames from shared memory 530.

[0089] In process 500, steps 504, 506, 508, etc., can correspond to... Figure 2The operation at step 204 involves transferring the original video frame from the target application 110 to the video effects processing application 120. For example, the target application 110 can provide the original video frame to the video effects processing application 120 via shared memory 530, and the video effects processing application 120 can retrieve the original video frame from the target application 110 via shared memory 530. It should be understood that, due to the use of shared memory 530, the operation of the target application 110 providing the original video frame to the video effects processing application 120 can refer to the target application 110 storing the original video frame in shared memory 530 and providing the storage information of the original video frame in shared memory 530 to the video effects processing application 120, and the operation of the video effects processing application 120 retrieving the original video frame from the target application 110 can refer to the video effects processing application 120 retrieving the storage information. In process 500, since the video effects processing application 120 does not actually receive the original video frame, but accesses the original video frame in shared memory 530 at least based on stored information (e.g., based on a storage information indication generated based on the stored information), the time and resources required to actually send the original video frame to the video effects processing application 120 can be saved, thereby improving execution efficiency.

[0090] In process 500, step 516 can correspond to Figure 2 The operation at step 210 involves transferring the processed video frame from the video effects processing application 120 to the target application 110. For example, the target application 110 can obtain the processed video frame from the video effects processing application 120 via shared memory 530. It should be understood that, due to the use of shared memory 530, the operation of the target application 110 obtaining the processed video frame from the video effects processing application 120 can refer to the target application 110 reading the processed video frame from shared memory 530.

[0091] It should be understood that process 500 may also include, at 540, inter-process communication to enable synchronous access to shared memory 530 by the target application 110 and the video effects processing application 120. The inter-process communication at 540 can refer to communication between the processes of the target application 110 and the video effects processing application 120, which can be continuously executed so that when one of the target application 110 or the video effects processing application 120 accesses shared memory 530, causing a change in the state of shared memory 530, the other of the two applications can be promptly notified and proceed with the next operation. Inter-process communication can be performed between the video acquisition module 112 and the custom API 126. For example, after the video acquisition module 112 stores the original video frame into shared memory 530 at 504, the video acquisition module 112 can notify the custom API 126 via inter-process communication that it has completed storing the original video frame. Thus, the custom API 126 can then retrieve the storage information at 506. For example, after the video effects application module 122 has completed the video effects application operation at 512 and notified the custom API 126 at 514, the custom API 126 can notify the video acquisition module 112 via inter-process communication that the operation of applying video effects to the original video frame has been completed. Thus, the video acquisition module 112 can then read the processed video frame from shared memory 530 at 516. The inter-process communication at 540 can be implemented using various techniques, such as event objects, sockets, semaphores, etc. The embodiments of this disclosure are not limited to any specific technique for implementing inter-process communication.

[0092] It should be understood that the above combination Figures 3 to 5 All steps and their order described in the process of adding video effects are exemplary and can be modified in any way according to specific application scenarios and needs. For example, although in Figures 3 to 5 The diagram shows that UI module 124 can receive instructions regarding video effects; however, even if UI module 124 does not receive instructions regarding video effects, the video effects processing application can still determine which video effect to apply to a video frame based on any predetermined rules. For example, in Figure 4 and Figure 5In the case of a desktop application, the video effects processing application can run in a sandbox environment, such as within a sandboxed iframe or WebView. Since the sandbox mechanism effectively restricts an application's access to system resources, running the video effects processing application in a sandbox environment can further effectively avoid security risks. The embodiments disclosed herein are not limited to any specific implementation of running the video effects processing application in a sandbox environment.

[0093] Figures 6A to 6D An exemplary user interface for adding video effects according to an embodiment is shown. Figures 6A to 6D In this context, we assume the target application is a web-based or desktop-based online conferencing application, and the user of this online conferencing application, Betty, wants to add video effects to her video image. Figures 6A to 6D The exemplary user interface is presented on user Betty's terminal device.

[0094] Figure 6A The user interface 600a of an online conferencing application is shown. User interface 600a displays an interface for making audio and video settings. Audio and video settings may include, for example, whether to enable the microphone / speaker, volume adjustment, whether to enable the camera, and whether to enable video effects. Assume Betty has already enabled her camera, and the video image of Betty captured by the camera is displayed in box 602. In user interface 600a, control 604 is in a closed state, indicating that video effects are not yet enabled.

[0095] Suppose Betty activates control 604 in user interface 600a to indicate her desire to add video effects to the video image in box 602. Activating control 604 will trigger the loading and running of the video effects processing application in the online conferencing application. For example, it may trigger the downloading of UI data for the video effects processing application and implement UI display, and optionally, it may trigger the downloading and caching of video effects-related data. Accordingly, the online conferencing application will then present, as shown below... Figure 6B The user interface shown is 600b.

[0096] In user interface 600b, control 604 is already in the open state. User interface 600b includes user interface 610 of a video effects processing application. For example, user interface 610 is generated by the UI module of the video effects processing application. User interface 610 includes several exemplary video effect icons 612 to 618. Video effect icon 612 indicates adding sunglasses to the face of a person in the original image frame. Video effect icon 614 indicates adding regular glasses to the face of a person in the original image frame and changing the person's hairstyle. Video effect icon 616 indicates adding star decorations to the hair of a person in the original image frame. Video effect icon 618 indicates adding leaf headdresses to the hair of a person in the original image frame. User interface 610 may also include a "More" button, clicking which will trigger loading more video effect icons.

[0097] Suppose Betty selects video effect icon 616 in user interface 600b to indicate that she wants to add star decorations to the hair in the video image presented in box 602. Selecting video effect icon 616 will trigger the downloading of video effect-related data corresponding to video effect icon 616 and the execution of the corresponding video effect algorithm, or it will trigger the direct execution of the locally cached video effect algorithm corresponding to video effect icon 616. The video effect application's video effect application module will use this video effect algorithm to apply the video effect material "star decorations" to the hair of the person in the original video frame within box 602. Accordingly, the online conferencing application will then present, as shown below. Figure 6C The user interface shown is 600c.

[0098] In user interface 600c, video effect icon 616 is highlighted to indicate that the video effect corresponding to that icon is currently in use. Box 622 displays a video image of Betty with the "star decoration" video effect added.

[0099] Let's say that after the video effects were added, Betty entered an online meeting. Figure 6D The diagram illustrates the meeting interface 600d of an online conferencing application. Box 632 displays a video image of meeting participant Tom, and box 634 displays a video image of meeting participant Jimmy. Box 636 displays Betty's own video image, which has been enhanced with a "star decoration" video effect. It should be understood that in the meeting interfaces displayed on Tom and Jimmy's respective terminal devices, Betty's video image will be the one with the added video effect shown in box 636, not the one displayed on Tom's and Jimmy's devices. Figure 6A and Figure 6B The original video image presented in box 602.

[0100] Furthermore, assuming Betty did not set up video effects before the online meeting began, she could still set them up during the meeting. For example, the multiple operation buttons at the top of user interface 600d could include buttons for audio and video settings or buttons for adding video effects, to navigate to user interfaces 600a or 600b, and then set up the video effects.

[0101] It should be understood that Figures 6A to 6D All elements and their layouts are exemplary, and various variations may exist in different target applications and different real-world scenarios. The embodiments disclosed herein are not limited to any specific details of the user interface. Furthermore, it should be understood that, although... Figure 6B and Figure 6C Only the user interface 610 corresponding to one video effects processing application is shown. However, if the target application supports multiple video effects processing applications, multiple user interfaces corresponding to the multiple video effects processing applications can be displayed simultaneously. Thus, the user can select the desired video effect in the user interface of any video effects processing application.

[0102] Figure 7 A flowchart is shown of an exemplary method 700 for implementing video effect addition according to an embodiment. Method 700 may be performed at a target application.

[0103] At position 710, the original video frame can be obtained from the video source.

[0104] At 720, the original video frame can be provided to a video effects processing application, which is a web application.

[0105] At 730, the processed video frame with applied video effects can be obtained from the video effects processing application.

[0106] In one implementation, the target application can be a web application.

[0107] Providing the original video frame may include: providing the original video frame to the video effects processing application in a passable ArrayBuffer manner. Obtaining the processed video frame may include: obtaining the processed video frame from the video effects processing application in a passable ArrayBuffer manner.

[0108] In one implementation, the target application can be a desktop application.

[0109] Providing the original video frame may include: providing the original video frame to the video effects processing application via shared memory. Obtaining the processed video frame may include: obtaining the processed video frame from the video effects processing application via the shared memory.

[0110] Providing the original video frame may include storing the original video frame in the shared memory. Obtaining the processed video frame may include reading the processed video frame from the shared memory.

[0111] Storing the original video frame may include: storing the original video frame in the shared memory using a custom API. Reading the processed video frame may include: reading the processed video frame from the shared memory using the custom API.

[0112] Inter-process communication can be used to enable synchronous access to the shared memory by the target application and the video effects processing application.

[0113] It should be understood that method 700 may also include any steps / processes performed at the target application according to the embodiments of the present disclosure described above for implementing the addition of video effects.

[0114] Figure 8 A flowchart is shown for an exemplary method 800 for implementing video effects addition according to an embodiment. Method 800 may be performed at a video effects processing application, which may be a web application.

[0115] At position 810, the raw video frame can be obtained from the target application.

[0116] At 820, video effects can be applied to the original video frame to obtain a processed video frame.

[0117] At 830, the processed video frame can be provided to the target application.

[0118] In one implementation, the video effects processing application may run within a web content container.

[0119] In one implementation, the target application may be a web application, and the web content container may be an iframe.

[0120] Obtaining the raw video frame may include: obtaining the raw video frame from the target application using a passable ArrayBuffer. Providing the processed video frame may include: providing the processed video frame to the target application using a passable ArrayBuffer.

[0121] In one implementation, the target application may be a desktop application, and the web content container may be a Webview.

[0122] Obtaining the raw video frame may include: obtaining the raw video frame from the target application via shared memory. Providing the processed video frame may include: providing the processed video frame to the target application via the shared memory.

[0123] Obtaining the original video frame may include: using a custom API to obtain the storage information of the original video frame in the shared memory. Applying the video effect may include: applying the video effect to the original video frame in the shared memory, at least based on the storage information.

[0124] Inter-process communication can be used to enable synchronous access to the shared memory by the video effects processing application and the target application.

[0125] The video effects processing application can run in a sandbox environment.

[0126] In one implementation, method 800 may further include receiving an instruction regarding the video effect via a user interface. The video effect may be applied at least based on the instruction.

[0127] It should be understood that method 800 may also include any steps / processes performed at the video effects processing application according to the above embodiments of this disclosure for implementing the addition of video effects.

[0128] Figure 9 An exemplary apparatus 900 for implementing video effects addition according to an embodiment is shown. The apparatus 900 may be implemented at a target application.

[0129] The apparatus 900 may include: an original video frame acquisition module 910 for acquiring original video frames from a video source; an original video frame providing module 920 for providing the original video frames to a video effects processing application, which is a web application; and a processed video frame acquisition module 930 for acquiring processed video frames with applied video effects from the video effects processing application. Furthermore, the apparatus 900 may also include any other modules implemented at the target application that perform the steps of the method for adding video effects according to the above embodiments of this disclosure.

[0130] Figure 10 An exemplary apparatus 1000 for implementing video effects addition according to an embodiment is shown. The apparatus 1000 may be implemented at a video effects processing application, which may be a web application.

[0131] The apparatus 1000 may include: a raw video frame acquisition module 1010 for acquiring raw video frames from a target application; a video effects application module 1020 for applying video effects to the raw video frames to obtain processed video frames; and a processed video frame providing module 1030 for providing the processed video frames to the target application. Furthermore, the apparatus 1000 may also include any other modules implemented at the video effects processing application that perform the steps of the method for adding video effects according to the above embodiments of this disclosure.

[0132] Figure 11 An exemplary apparatus 1100 for implementing video effects addition is shown according to an embodiment.

[0133] The apparatus 1100 may include: at least one processor 1110; and a memory 1120 storing computer-executable instructions. When the computer-executable instructions are executed, the at least one processor 1110 may perform any step / process of the method for adding video effects according to the embodiments of the present disclosure described above.

[0134] Embodiments of this disclosure provide a computer program product for implementing video effects addition, including a computer program that is run by at least one processor to perform any step / process of the method for implementing video effects addition according to the embodiments of this disclosure described above.

[0135] Embodiments of this disclosure can be implemented in a non-transitory computer-readable medium. This non-transitory computer-readable medium may include instructions that, when executed, cause one or more processors to perform any step / process of the method for adding video effects according to the embodiments of this disclosure described above.

[0136] It should be understood that all operations in the methods described above are merely exemplary, and this disclosure is not limited to any operation in the methods or the order of such operations, but should cover all other equivalent transformations under the same or similar concept.

[0137] Furthermore, unless otherwise specified or clearly indicated from the context that the application is to the singular form, the articles “a” and “an” as used in this specification and the appended claims should generally be interpreted as meaning “one” or “one or more”.

[0138] It should also be understood that all modules in the apparatus described above can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. Furthermore, any of these modules can be further functionally divided into sub-modules or combined together.

[0139] Processors have been described in conjunction with various devices and methods. These processors can be implemented using electronic hardware, computer software, or any combination thereof. Whether these processors are implemented as hardware or software will depend on the specific application and the overall design constraints imposed on the system. As an example, the processors, any portions of processors, or any combinations of processors given in this disclosure can be implemented as microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gate logic, discrete hardware circuits, and other suitable processing units configured to perform the various functions described in this disclosure. The functionality of the processors, any portions of processors, or any combinations of processors given in this disclosure can be implemented as software executed by a microprocessor, microcontroller, DSP, or other suitable platform.

[0140] Software should be broadly considered as representing instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, procedures, functions, etc. Software may reside on a computer-readable medium. Computer-readable media may include, for example, memory, which may be, for example, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical disks, smart cards, flash memory devices, random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), registers, or removable disks. Although memory is shown as separate from the processor in several aspects set forth in this disclosure, memory may also reside within the processor (e.g., in caches or registers).

[0141] The above description is provided to enable any person skilled in the art to implement the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents of the elements of the various aspects described in this disclosure that are known or about to be known to those skilled in the art shall be covered by the claims.

Claims

1. A method for adding video effects, the method being executed at a target application, and the method comprising: Obtain the original video frames from the video source; The original video frames are provided to a video effects processing application, which is a web application. as well as Obtain the processed video frames with applied video effects from the video effects processing application.

2. The method as described in claim 1, wherein, The target application is a web application.

3. The method as described in claim 2, wherein, Providing the original video frame includes: providing the original video frame to the video effects processing application in the form of a transferable array buffer, and The process of obtaining the processed video frame includes: obtaining the processed video frame from the video effects processing application in a transferable ArrayBuffer manner.

4. The method of claim 1, wherein, The target application is a desktop application.

5. The method of claim 4, wherein, Providing the original video frames includes: providing the original video frames to the video effects processing application via shared memory, and The process of obtaining the processed video frame includes: obtaining the processed video frame from the video effects processing application through the shared memory.

6. The method of claim 5, wherein, Providing the original video frame includes: storing the original video frame in the shared memory, and The step of obtaining the processed video frame includes: reading the processed video frame from the shared memory.

7. The method of claim 6, wherein, The storage of the original video frames includes: storing the original video frames in the shared memory using a custom API, and Reading the processed video frame includes: using the custom API to read the processed video frame from the shared memory.

8. The method of claim 5, further comprising: Inter-process communication enables the target application and the video effects processing application to access the shared memory synchronously.

9. A method for adding video effects, the method being executed at a video effects processing application, the video effects processing application being a web application, and the method comprising: Obtain raw video frames from the target application; Video effects are applied to the original video frames to obtain processed video frames; as well as The processed video frame is provided to the target application.

10. The method of claim 9, wherein, The video effects processing application runs within a web content container.

11. The method of claim 10, wherein, The target application is a web application, and the web content container is an iframe.

12. The method of claim 11, wherein, The acquisition of the original video frames includes: acquiring the original video frames from the target application using a transferable array buffer, and Providing the processed video frame includes providing the processed video frame to the target application in a transferable ArrayBuffer manner.

13. The method of claim 10, wherein, The target application is a desktop application, and the web content container is a Webview.

14. The method of claim 13, wherein, The acquisition of the original video frames includes: acquiring the original video frames from the target application via shared memory, and Providing the processed video frame includes providing the processed video frame to the target application via the shared memory.

15. The method of claim 14, wherein, The process of obtaining the original video frame includes: using a custom API to obtain the storage information of the original video frame in the shared memory, and The application of video effects includes: applying the video effects to the original video frames in the shared memory, based at least on the stored information.

16. The method of claim 14, further comprising: Inter-process communication enables the video effects processing application and the target application to access the shared memory synchronously.

17. The method of claim 13, wherein, The video effects processing application runs in a sandbox environment.

18. The method of claim 9, further comprising: Receive instructions regarding the video effects through the user interface, and The video effect is applied at least based on the instruction.

19. An apparatus for adding video effects, comprising: At least one processor; as well as A memory that stores computer-executable instructions that, when executed, cause the at least one processor to perform the steps of the method as claimed in any one of claims 1 to 18.

20. A computer program product comprising a computer program that is executed by at least one processor to perform the steps of the method as claimed in any one of claims 1 to 18.

Images