Video stream processing method and device and electronic equipment

Abstract

The invention provides a video stream processing method and device and electronic equipment, and can be applied to the technical field of video processing and the technical field of artificial intelligence. The video stream processing method comprises the following steps: in response to a video marking request, obtaining to-be-processed video data from a server, the to-be-processed video data being composed of a plurality of video frames of a video stream; for each video frame, calling a drawing interface of a browser, and drawing the video frame on a drawing board; and calling a drawing interface of the browser, drawing the to-be-marked graphic element on the drawing board, fusing the graphic element with the video frame, obtaining a processed video frame, and obtaining processed video data.

Description

Technical Field

[0001] The present disclosure relates to the fields of video processing technology and artificial intelligence technology, and more specifically, to a video stream processing method, device and electronic device. Background Art

[0002] The current video stream tagging function mainly revolves around the core characteristics of the video stream. This process starts with the front-end device, which uses advanced sensors and other equipment to accurately collect data on the detected object, and then converts the collected raw data into digital signals. Once the system receives these precise digital signals, it immediately processes the current video data stream and cleverly integrates the relevant labeled data into it. This fusion process not only ensures the accuracy of the data, but also greatly improves the richness and readability of the video information. The system will then re-render the fused data to generate a new video stream containing the labeled information for users to watch or further analyze.

[0003] In the process of implementing the concept of the present disclosure, the inventors found that there are at least the following problems in the related art: slow processing speed and poor video quality during video stream marking. Summary of the invention

[0004] In view of this, the present disclosure provides a video stream processing method, device and electronic device.

[0005] One aspect of the present disclosure provides a video stream processing method, comprising: in response to a video marking request, obtaining to-be-processed video data from a server, wherein the to-be-processed video data is composed of a plurality of video frames of a video stream; for each video frame, calling a drawing interface of a browser to draw the video frame on a drawing board; and calling a drawing interface of the browser to draw a graphic element to be marked on the drawing board to merge the graphic element with the video frame to obtain a processed video frame to obtain processed video data.

[0006] According to an embodiment of the present disclosure, the above-mentioned calling of the drawing interface of the above-mentioned browser to draw the graphic elements to be marked on the above-mentioned drawing board to merge the above-mentioned graphic elements with the above-mentioned video frame includes: calling the drawing interface of the above-mentioned browser to generate a marking layer on the above-mentioned drawing board based on the size of the above-mentioned video frame, wherein the above-mentioned marking layer is an upper layer of the layer where the above-mentioned video frame is located; and drawing the above-mentioned graphic elements in the above-mentioned marking layer to merge the above-mentioned graphic elements with the above-mentioned video frame.

[0007] According to an embodiment of the present disclosure, the method further includes: acquiring the graphic element in response to the video tagging request.

[0008] According to an embodiment of the present disclosure, the above-mentioned response to the above-mentioned video marking request and obtaining the above-mentioned graphic element include: determining marker requirement information based on the above-mentioned video marking request; and in the case that there is a first material matching the above-mentioned marker requirement information in the material library, obtaining the above-mentioned graphic element based on the above-mentioned first material.

[0009] According to an embodiment of the present disclosure, the above-mentioned graphic element is obtained based on the above-mentioned first material, including: determining an adjustment ratio based on the size of the above-mentioned video frame and the size of the above-mentioned drawing board; and adjusting the size of the above-mentioned first material based on the above-mentioned adjustment ratio to obtain the above-mentioned graphic element.

[0010] According to an embodiment of the present disclosure, the method further includes: generating a second material based on a user operation when there is no material matching the marker requirement information in the material library; and obtaining the graphic element based on the second material.

[0011] According to an embodiment of the present disclosure, the generation of the second material based on user operation includes: calling the drawing interface of the above-mentioned browser to generate a drawing window on the above-mentioned browser; generating a material drawing script based on the drawing operation of the user controlling the cursor on the above-mentioned drawing window; and running the above-mentioned material drawing script to generate the above-mentioned second material.

[0012] According to an embodiment of the present disclosure, the method further includes: calling a drawing interface of the browser, refreshing the drawing board based on the frame rate of the video stream, so as to sequentially display a plurality of processed video frames on the drawing board to display the marked video stream.

[0013] Another aspect of the present disclosure provides a video stream processing device, including:

[0014] An acquisition module, configured to acquire to-be-processed video data from a server in response to a video tag request, wherein the to-be-processed video data is composed of a plurality of video frames of a video stream;

[0015] A calling module, used for calling a drawing interface of a browser for each video frame to draw the video frame on a drawing board; and

[0016] The drawing module is used to call the drawing interface of the above-mentioned browser to draw the graphic elements to be marked on the above-mentioned drawing board, so as to merge the above-mentioned graphic elements with the above-mentioned video frames to obtain processed video frames and to obtain processed video data.

[0017] Another aspect of the present disclosure provides an electronic device, comprising:

[0018] one or more processors;

[0019] a memory for storing one or more programs,

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0021] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0022] Another aspect of the present disclosure provides a computer program product, the computer program product comprising computer executable instructions, and the instructions are used to implement the above method when executed.

[0023] According to an embodiment of the present disclosure, by responding to a video marking request, the to-be-processed video data consisting of a plurality of video frames of a video stream from a server is obtained, and the video stream format is converted into an image format for transmission, and the video quality during the transmission process changes slightly. For each video frame, the drawing interface of the browser is called to draw the video frame on a drawing board; the drawing interface of the browser is called to draw the graphic element to be marked on the drawing board to merge the graphic element with the video frame to obtain the processed video frame to obtain the processed video data. Since the drawing interface of the browser is called to mark the video frame, there is no need for complex optimization algorithm fusion, thereby reducing the computational burden and improving the processing speed of the video stream marking process. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0025] Figure 1 A system architecture diagram to which a video stream processing method according to an embodiment of the present disclosure can be applied is schematically shown;

[0026] Figure 2 The flowchart of the video stream processing method according to the embodiment of the present disclosure is schematically shown;

[0027] Figure 3 A flowchart of a video stream processing method according to another embodiment of the present disclosure is schematically shown;

[0028] Figure 4 A block diagram schematically shows a video stream processing device according to an embodiment of the present disclosure;

[0029] Figure 5 The block diagram schematically shows an electronic device suitable for implementing a video stream processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0031] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0032] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0033] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0034] In the embodiments of the present disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security, and national security.

[0035] In the embodiments of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0036] An embodiment of the present disclosure provides a video stream processing method including: in response to a video marking request, obtaining video data to be processed from a server, wherein the video data to be processed is composed of multiple video frames of a video stream; for each video frame, calling a drawing interface of a browser to draw the video frame on a drawing board; and calling a drawing interface of the browser to draw a graphic element to be marked on the drawing board to merge the graphic element with the video frame to obtain a processed video frame to obtain processed video data.

[0037] Figure 1 The system architecture diagram to which the video stream processing method according to an embodiment of the present disclosure can be applied is schematically shown.

[0038] It should be noted that Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0039] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a client 101, a network 102 and a server 103. The network 102 is used to provide a medium for a communication link between the client 101 and the server 103. The network 102 may include various connection types, such as wired and / or wireless communication links, etc.

[0040] A user may use the client 101 to interact with the server 103 via the network 102 to receive or send messages, etc.

[0041] The client 101 may be any electronic device with a display screen and supporting web browsing, including but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. The browser is the main part of the client 101, and the user accesses the web application through the browser. The browser is responsible for sending an HTTP (Hypertext Transfer Protocol) request to the server 102 and presenting the web page returned by the server 103.

[0042] The server 103 may be a server that provides various services, such as processing client requests, executing application logic, storing data, etc. For example, a user initiates an HTTP request to the server 103 through the browser of the client 101. After receiving the request, the server 103 forwards the request to the application server, which processes the request according to the business logic, generates response content, and returns the response content to the browser of the client 101 through the server 103. The browser receives the response content, renders the page according to the response content, and displays it to the user.

[0043] That is, the client 101 and the server 103 are a B / S architecture (Browser / Server architecture) in a computer system. The client does not need to install additional software, only a browser is needed.

[0044] It should be noted that the video stream processing method provided in the embodiment of the present disclosure can generally be executed by the client 101. Accordingly, the video stream processing device provided in the embodiment of the present disclosure can generally be set in the client 101.

[0045] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0046] Figure 2 The flowchart of the video stream processing method according to the embodiment of the present disclosure is schematically shown.

[0047] like Figure 2 As shown, the method includes operations S210 to S230. The method can be executed by a client.

[0048] In operation S210 , in response to a video marking request, to-be-processed video data is acquired from a server, wherein the to-be-processed video data is composed of a plurality of video frames of a video stream.

[0049] According to an embodiment of the present disclosure, the video marking request may be a request from the client to the server. For example, the video marking request may be an HTTP request.

[0050] The server is used to receive the video tag request, parse the video tag request, and obtain video tag information. The video tag information includes a unique identifier of the video data to be processed. According to the unique identifier of the video data to be processed, the data to be processed is queried from the database or the server locally.

[0051] According to an embodiment of the present disclosure, the video stream can be video data that can be transmitted in real time through a network or a medium. The server is also used to extract multiple video frames from the video stream; compress the multiple video frames, etc., to obtain the video data to be processed.

[0052] The client is used to obtain the video data to be processed from the server.

[0053] In operation S220, for each video frame, a drawing interface of the browser is called to draw the video frame on a drawing board.

[0054] According to an embodiment of the present disclosure, the browser may be a main part of the client, and the application program may be accessed through the browser. Therefore, the client does not need to install additional software.

[0055] According to an embodiment of the present disclosure, a browser may provide a drawing interface for drawing images, animations, etc. in a web page. For example, the drawing interface may be an interface for drawing provided by the browser without the need to install additional software on the client.

[0056] According to an embodiment of the present disclosure, the drawing board may be a drawing web page connected to the drawing interface.

[0057] For example, the artboard can be used for rendering, filling colors, drawing various graphics, drawing text, image operations, etc.

[0058] In operation S230, a drawing interface of the browser is called to draw the graphic element to be marked on the drawing board, so as to merge the graphic element with the video frame to obtain a processed video frame, so as to obtain processed video data.

[0059] According to an embodiment of the present disclosure, the graphic element to be marked may be a point, a line, a circle, an arc, a polygon, or the like.

[0060] According to an embodiment of the present disclosure, a video frame is rendered onto a drawing board, the video frame rendered onto the drawing board is used as a background, and is merged with a graphic element on the background, thereby obtaining a processed video frame.

[0061] For example, the positions or colors of graphic elements of a plurality of processed video frames are adjusted in time sequence, so that the video presented by the processed video data is more dynamic.

[0062] For example, based on the video content, the graphic elements to be marked corresponding to each of the multiple video frames are determined. For each video frame, the graphic element is merged with the video frame to obtain a processed video frame. The multiple processed video frames are integrated according to the playback order to obtain processed video data.

[0063] According to an embodiment of the present disclosure, by responding to a video marking request, the to-be-processed video data consisting of a plurality of video frames of a video stream from a server is obtained, and the video stream format is converted into an image format for transmission, and the video quality during the transmission process changes slightly. For each video frame, the drawing interface of the browser is called to draw the video frame on a drawing board; the drawing interface of the browser is called to draw the graphic element to be marked on the drawing board to merge the graphic element with the video frame to obtain the processed video frame to obtain the processed video data. Since the drawing interface of the browser is called to mark the video frame, there is no need for complex optimization algorithm fusion, thereby reducing the computational burden and improving the processing speed of the video stream marking process.

[0064] According to an embodiment of the present disclosure, a drawing interface of a browser is called to draw a graphic element to be marked on a drawing board to merge the graphic element with a video frame, including: calling a drawing interface of a browser to generate a marking layer on a drawing board based on the size of a video frame, wherein the marking layer is an upper layer of a layer where a video frame is located; and drawing the graphic element in the marking layer to merge the graphic element with the video frame.

[0065] According to an embodiment of the present disclosure, the size of a video frame is generally expressed in the form of width×height, with the unit being pixels. When drawing, the aspect ratio of the video frame can be maintained to avoid picture deformation.

[0066] For example, based on the size of the video frame, a marker layer with the same aspect ratio as the video frame can be created. When scaling the video frame, the aspect ratio of the original video frame can be maintained to avoid image deformation.

[0067] For example, the marker layer's dimensions are the same as the video frame's dimensions, preserving the clear proportions of the video frame.

[0068] According to the embodiments of the present disclosure, by calling the drawing interface of the browser, a markup layer can be generated on the drawing board based on the size of the video frame, and the graphic elements can be drawn in the markup layer, thereby merging the graphic elements with the video frame, avoiding complex conversion and fusion processing procedures (such as decoding, fusion processing, encoding), reducing time delays, and improving user experience.

[0069] According to an embodiment of the present disclosure, the method further includes: acquiring a graphic element in response to a video tag request.

[0070] According to an embodiment of the present disclosure, the image element may be obtained from the server in response to a video tag request. The video tag information obtained by parsing the video tag request includes the image element. For example, the image element may be a circle, a texture map, etc.

[0071] According to an embodiment of the present disclosure, in response to a video tagging request, obtaining a graphic element includes: determining marker requirement information based on the video tagging request; and obtaining a graphic element based on the first material when there is a first material matching the marker requirement information in the material library.

[0072] For example, the drawn images are stored by utilizing the cache mechanism of the drawing interface of the browser to obtain a material library, and the drawn images are reused from the material library.

[0073] For example, the material library can also be accessed by a server. The server is used to match the marker requirement information with multiple materials in the material library to obtain a matching result; if the matching result indicates that there is a first material matching the marker requirement information, the server sends the first material to the browser of the client.

[0074] According to an embodiment of the present disclosure, the marker requirement information includes markers of multiple video frames and first materials corresponding to the markers. The markers of the video frames are marked by drawing using the first material.

[0075] According to an embodiment of the present disclosure, image processing may be performed on the first material to obtain image elements.

[0076] For example, the first material is cropped to obtain image elements.

[0077] For example, the first material is rotated to obtain image elements.

[0078] For example, color rendering is performed on the first material to obtain an image element.

[0079] According to an embodiment of the present disclosure, obtaining a graphic element based on a first material includes: determining an adjustment ratio based on a size of a video frame and a size of a drawing board; and adjusting a size of the first material based on the adjustment ratio to obtain the graphic element.

[0080] For example, the adjustment ratio may be a scaling ratio, and a suitable scaling ratio is calculated according to the size of the video frame and the size of the drawing board, while maintaining the aspect ratio of the video frame, and the video frame is not stretched or compressed. In order to adapt the size of the first material to the video frame, the size of the first material is adjusted based on the scaling ratio to obtain the graphic element.

[0081] For example, when the aspect ratio of the canvas is greater than that of the video frame, the video frame is scaled by height or width to obtain an adjustment ratio, and then the width and height of the first material are adjusted according to the adjustment ratio to obtain a graphic element.

[0082] For example, the adjustment ratio may be a fill ratio. In order to fully utilize the canvas space, the video frame may fill the entire canvas, thereby calculating the fill ratio of the video frame. The width and height of the first material are then adjusted according to the fill ratio, so that the obtained graphic element is more compatible with the video frame.

[0083] According to an embodiment of the present disclosure, the method further includes: generating a second material based on a user operation when there is no material matching the marker requirement information in the material library; and obtaining a graphic element based on the second material.

[0084] According to an embodiment of the present disclosure, the user operation may be a drawing operation of the user on the marking layer.

[0085] According to an embodiment of the present disclosure, the second material represents a drawing drawn by the user on the markup layer.

[0086] According to an embodiment of the present disclosure, an adjustment ratio is determined based on the size of the video frame and the size of the drawing board; and the size of the second material is adjusted based on the adjustment ratio to obtain a graphic element.

[0087] According to an embodiment of the present disclosure, generating a second material based on user operations includes: calling a drawing interface of a browser to generate a drawing window on the browser; generating a material drawing script based on a drawing operation of a user controlling a cursor on the drawing window; and running the material drawing script to generate the second material.

[0088] According to an embodiment of the present disclosure, the user controls the cursor to perform a drawing operation in the drawing window, and the initial position of the cursor can be captured, that is, the user starts drawing. The initial position of the cursor is continuously captured until the user ends drawing. The drawing path can be obtained according to the change of the cursor position, thereby generating a material drawing script.

[0089] According to an embodiment of the present disclosure, the method further includes: calling a drawing interface of the browser, refreshing the drawing board based on the frame rate of the video stream, so as to sequentially display a plurality of processed video frames on the drawing board to display the marked video stream.

[0090] According to an embodiment of the present disclosure, the frame rate of a video stream indicates the number of video frames displayed per second. The higher the frame rate, the larger the amount of video data. In particular, when transmitting or storing high-quality video streams, video encoding and decoding are often used to reduce the amount of transmission data, but there is a decrease in video quality and more network bandwidth required.

[0091] The client obtains the video data to be processed from the server. The video data to be processed is composed of multiple video frames of the video stream, so the video stream format is converted into an image format.

[0092] The video frame is drawn on the drawing board by calling the drawing interface of the browser, and the image element to be marked is drawn on the drawing board to merge the image element with the video frame to obtain the processed video frame. The drawing board is refreshed based on the frame rate of the video stream to display multiple processed video frames on the drawing board in sequence to display the marked video stream.

[0093] When there is no need to transmit high-quality video streams, the video stream format is converted into an image format for transmission, which can reduce the degradation of video quality and can also mark the video to display the marked high-quality video stream.

[0094] By deeply calling the browser's drawing interface, the video stream fusion processing is pushed to a new level. In this process, it is no longer limited to traditional video stream processing technology, but fully utilizes the flexibility and powerful functions of the browser's drawing interface to cleverly combine the video stream with various markers and graphic elements in the presentation layer.

[0095] This deep integration not only solves the common performance bottleneck problems in the video stream fusion process, such as slow processing speed and poor rendering effect, but also significantly improves the efficiency and smoothness of the fusion process by optimizing algorithms and reducing unnecessary computing burdens. Now, whether it is the decoding and rendering of video streams, or the drawing and superposition of markers, it can be completed efficiently in a short time, bringing users a smoother, clearer and more accurate visual experience.

[0096] Figure 3 The flowchart of a video stream processing method according to another embodiment of the present disclosure is schematically shown.

[0097] like Figure 3 As shown, the method includes operations S310 to S350.

[0098] In operation S310 , in response to a video marking request, to-be-processed video data is acquired from a server, wherein the to-be-processed video data is composed of a plurality of video frames of a video stream.

[0099] In operation S310 , for each video frame, a drawing interface of the browser is called to draw the video frame on a drawing board.

[0100] In operation S310, a drawing interface of the browser is called to generate a markup layer on a drawing board based on the size of the video frame.

[0101] In operation S310, a graphic element is drawn in a markup layer to merge the graphic element with a video frame to obtain a processed video frame.

[0102] In operation S310, a drawing interface of the browser is called to refresh the drawing board based on the frame rate of the video stream, so as to sequentially display a plurality of processed video frames on the drawing board to display the marked video stream.

[0103] The process of video stream conversion and fusion plays an important role in video processing, but this process also has some obvious disadvantages, mainly including the following points:

[0104] The computing resources are consumed greatly. For example, the conversion and fusion of video streams requires decoding and processing the original video streams, and then re-encoding them. This process consumes a lot of computing resources, including CPU (Central Processing Unit) and GPU (Graphic Processing Unit). This consumption is particularly significant when processing high-resolution and high-frame-rate videos.

[0105] Time delay, such as because the conversion and fusion process involves multiple steps, including decoding, processing, encoding, etc., which will inevitably lead to a certain time delay. In application scenarios with high real-time requirements, such as live broadcasting and video conferencing, this delay may have a negative impact on user experience.

[0106] Video quality is degraded. For example, in the process of video stream conversion and fusion, the video quality may be lost due to encoding and decoding. Especially when the compression rate is high, the details, color and contrast of the video may be affected, resulting in a decrease in video quality.

[0107] Network bandwidth usage: For example, the converted and fused video stream usually has a higher bit rate, which means more network bandwidth is required for transmission. In the case of limited bandwidth, it may cause problems such as reduced transmission speed and freezes, affecting user experience.

[0108] Technical complexity, such as video stream code conversion and fusion, involves multiple technical fields, including video encoding and decoding, image processing, network communication, etc. Developers are required to have a high level of technical skills and rich experience, otherwise various technical difficulties and failures may occur.

[0109] Compatibility issues, such as compatibility issues between different video codec standards and formats. If multiple codec standards and formats are involved in the conversion and fusion process, problems such as decoding failure and format errors may occur, affecting the normal playback of the video.

[0110] In the process of video stream processing, the code stream conversion and fusion process may often cause a series of problems. In order to avoid these potential problems, the overall strategy is adjusted to cleverly combine the basic video stream with markers in the presentation layer (complete the interaction between users and the background) of the B / S architecture (browser / server architecture).

[0111] On the presentation layer, the browser's drawing interface is fully utilized for drawing, making the drawing board a powerful creation platform. Drawing various image elements on the drawing board, whether it is points, lines, circles, arcs or polygons, can be easily achieved. At the same time, the drawing board also supports text rendering and image overlay, which greatly enriches the expressiveness of the picture.

[0112] Based on this feature, the video stream code stream is drawn on the bottom layer of the drawing board, and the video stream and the marker are seamlessly integrated by using the browser's drawing interface. In order to ensure that the proportion of the image elements is consistent with the video content, the ratio of the length and width of the original video to the length and width of the drawing board is calculated, and the size of the image elements is adjusted synchronously accordingly, realizing the modernization of traditional video stream marking technology. This method not only improves the efficiency of video processing, but also ensures the accuracy and consistency of marking, bringing users a better user experience.

[0113] In the scenario of video stream fusion, the challenges include but are not limited to the reduction of output stream quality, delay accumulation, and even the risk of system crash. In order to cope with these challenges, we are committed to reducing CPU load, reducing memory usage, optimizing network bandwidth utilization, and alleviating disk I / O (input / output) pressure to ensure the stability and efficiency of video stream fusion.

[0114] By finely optimizing the drawing logic, unnecessary calculations and drawing operations are significantly reduced, further improving processing efficiency. For example, the cache mechanism of the browser's drawing interface is cleverly used to store and reuse the drawn images, thus avoiding unnecessary repeated drawing and greatly improving overall performance.

[0115] Figure 4 The block diagram schematically shows a video stream processing device according to an embodiment of the present disclosure.

[0116] like Figure 4 As shown, the video stream processing device 400 includes a first acquisition module 410 , a calling module 420 and a drawing module 430 .

[0117] A first acquisition module 410 is used to acquire to-be-processed video data from a server in response to a video tag request, wherein the to-be-processed video data is composed of a plurality of video frames of a video stream;

[0118] A calling module 420 is used to call the drawing interface of the browser for each video frame to draw the video frame on the drawing board; and

[0119] The drawing module 430 is used to call the drawing interface of the browser to draw the graphic element to be marked on the drawing board to merge the graphic element with the video frame to obtain the processed video frame to obtain the processed video data.

[0120] According to an embodiment of the present disclosure, the drawing module 430 includes a calling submodule and a drawing submodule; the calling submodule is used to call the drawing interface of the browser, and generate a marking layer on the drawing board based on the size of the video frame, wherein the marking layer is an upper layer of the layer where the video frame is located; the drawing submodule is used to draw graphic elements in the marking layer to merge the graphic elements with the video frame.

[0121] According to an embodiment of the present disclosure, the above-mentioned device further includes a second acquisition module, and the second acquisition module is used to acquire the graphic element in response to the video marking request.

[0122] According to an embodiment of the present disclosure, the second acquisition module includes a determination submodule and an acquisition submodule. The determination submodule is used to determine the marker requirement information based on the video marking request; the acquisition submodule is used to obtain the graphic element based on the first material when there is a first material matching the marker requirement information in the material library.

[0123] According to an embodiment of the present disclosure, the acquisition submodule includes a determination unit and an adjustment unit. The determination unit is used to determine the adjustment ratio based on the size of the video frame and the size of the drawing board; the adjustment unit is used to adjust the size of the first material based on the adjustment ratio to obtain the graphic element.

[0124] According to an embodiment of the present disclosure, the above-mentioned device also includes a generation module and an acquisition module; the generation module is used to generate a second material based on user operation when there is no material matching the marker requirement information in the material library; the acquisition module is used to obtain a graphic element based on the second material.

[0125] According to an embodiment of the present disclosure, the generation module includes a first generation submodule, a second generation submodule and an operation submodule. The first generation submodule is used to call the drawing interface of the browser to generate a drawing window on the browser; the second generation submodule is used to generate a material drawing script based on the drawing operation of the user controlling the cursor on the drawing window; the operation submodule is used to run the material drawing script to generate the second material.

[0126] According to an embodiment of the present disclosure, the device further comprises a refresh module. The refresh module is used to call the drawing interface of the browser to refresh the drawing board based on the frame rate of the video stream, so as to sequentially display multiple processed video frames on the drawing board to display the marked video stream.

[0127] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging circuits, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0128] For example, any multiple of the first acquisition module 410, the calling module 420, and the drawing module 430 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the first acquisition module 410, the calling module 420, and the drawing module 430 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in a suitable combination of any of them. Alternatively, at least one of the first acquisition module 410 , the calling module 420 , and the drawing module 430 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be executed.

[0129] It should be noted that the video stream processing device part in the embodiment of the present disclosure corresponds to the video stream processing method part in the embodiment of the present disclosure. The description of the video stream processing device part specifically refers to the video stream processing data processing method part, which will not be repeated here.

[0130] Figure 5The block diagram schematically shows an electronic device suitable for implementing a video stream processing method according to an embodiment of the present disclosure.

[0131] Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0132] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0133] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0134] According to an embodiment of the present disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0135] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0136] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0137] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0138] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 502 and / or the RAM 503 described above and / or one or more memories other than the ROM 502 and the RAM 503 .

[0139] The embodiments of the present disclosure also include a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiments of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the video stream processing method provided by the embodiments of the present disclosure.

[0140] When the computer program is executed by the processor 501, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0141] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0142] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0144] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A video stream processing method, comprising: In response to the video tagging request, obtaining to-be-processed video data from a server, wherein the to-be-processed video data is composed of a plurality of video frames of a video stream; For each video frame, calling a drawing interface of the browser to draw the video frame on a drawing board; and The drawing interface of the browser is called to draw the graphic element to be marked on the drawing board, so as to merge the graphic element with the video frame to obtain the processed video frame and the processed video data.

2. The method according to claim 1, wherein: The calling of the drawing interface of the browser to draw the graphic element to be marked on the drawing board to merge the graphic element with the video frame includes: Calling the drawing interface of the browser to generate a marking layer on the drawing board based on the size of the video frame, wherein the marking layer is an upper layer of the layer where the video frame is located; and The graphic element is drawn in the markup layer to merge the graphic element with the video frame.

3. The method according to claim 1, further comprising: In response to the video marking request, the graphic element is obtained.

4. The method according to claim 3, wherein: The step of obtaining the graphic element in response to the video tag request includes: Determining marker requirement information based on the video marking request; and In the case that there is a first material matching the marker requirement information in the material library, the graphic element is obtained based on the first material.

5. The method according to claim 4, wherein: The obtaining the graphic element based on the first material includes: Determining an adjustment ratio based on the size of the video frame and the size of the drawing board; and The size of the first material is adjusted based on the adjustment ratio to obtain the graphic element.

6. The method according to claim 4, further comprising: If there is no material matching the marker requirement information in the material library, generating a second material based on a user operation; as well as Based on the second material, the graphic element is obtained.

7. The method according to claim 6, wherein: The generating the second material based on the user operation includes: Calling the drawing interface of the browser to generate a drawing window on the browser; Generate a material drawing script based on a drawing operation of a user-controlled cursor on the drawing window; and Run the material drawing script to generate the second material.

8. The method according to any one of claims 1 to 7, further comprising: The drawing interface of the browser is called to refresh the drawing board based on the frame rate of the video stream, so as to sequentially display a plurality of processed video frames on the drawing board to display the marked video stream.

9. A video stream processing device, comprising: An acquisition module, configured to acquire to-be-processed video data from a server in response to a video tag request, wherein the to-be-processed video data is composed of a plurality of video frames of a video stream; A calling module, for calling a drawing interface of a browser for each video frame, and drawing the video frame on a drawing board; and The drawing module is used to call the drawing interface of the browser to draw the graphic elements to be marked on the drawing board, so as to merge the graphic elements with the video frame to obtain the processed video frame and the processed video data.

10. An electronic device comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

Images