Data processing method and apparatus, electronic device

Abstract

The application relates to the technical field of electronic whiteboards, in particular to a data processing method and device and electronic equipment. The data processing method comprises the following steps: receiving input data corresponding to the input of a user on a device and identifying the input data as a graphic element; drawing the graphic element to generate drawing data and writing the drawing data into an off-screen frame buffer area; displaying the drawing data stored in the off-screen frame buffer area on the device; encoding the drawing data stored in the off-screen frame buffer area to generate encoded data; and sending the encoded data to a terminal outside the device. According to the embodiment of the application, all graphic elements of an electronic whiteboard are rendered to an off-screen frame buffer area, the off-screen frame buffer area is rendered to the screen of the electronic whiteboard and is also encoded into a video code stream and sent to a video conference system at the other end, the system efficiency of the electronic whiteboard is improved, and the system performance overhead caused by multiple renderings is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronic whiteboards, and in particular to a data processing method and device and electronic equipment. BACKGROUND

[0002] Electronic conference systems have undergone several generations of technical evolution, including wired telegraphy, wireless telegraphy, wired voice, wireless voice, telecommunication video conferencing, and Internet video conferencing. A video conference system allows several participants to access a video conference from several locations, and through modern communication links and video conference equipment, audio, video, and file materials are transmitted to each other for instant interaction, so as to achieve the purpose of remote information communication. In today's world of deep integration of global interconnection, video conference systems have become increasingly important information means for improving business communication, social communication, and family communication. Video conference systems have become a standard for business communication, and more and more devices (electronic whiteboards / voice assistants / digital people, etc.) want to be integrated into video conference systems to enrich the function matrix of video conference systems.

[0003] For the existing electronic whiteboard joining a video conference, many people focus on how to perform off-screen rendering to improve the system efficiency of the electronic whiteboard, and the single mode of the electronic whiteboard integrated into the video conference system. SUMMARY

[0004] In view of the above problems, the present application provides a data processing method and device and electronic equipment, which can perform efficient off-screen rendering when an electronic whiteboard joins a video conference, and enrich the mode of the electronic whiteboard integrated into the video conference system.

[0005] In a first aspect, a data processing method is provided. The method includes receiving input data corresponding to a user's input on a device and identifying the input data as a primitive; rendering the primitive to generate rendering data and writing the rendering data to an off-screen frame buffer area; displaying the rendering data stored in the off-screen frame buffer area on the device; encoding the rendering data stored in the off-screen frame buffer area to generate encoded data; and sending the encoded data to a terminal outside the device.

[0006] In some embodiments, displaying the rendering data stored in the off-screen frame buffer area on the device includes writing the rendering data stored in the off-screen frame buffer area to a screen frame buffer area corresponding to a display screen of the device; and displaying the rendering data stored in the screen frame buffer area on the display screen.

[0007] In some embodiments, encoding the drawing data stored in the off-screen framebuffer to generate encoded data comprises encoding the drawing data based on a video encoding format specified by a video conference handshake protocol to generate the encoded data, and transmitting the encoded data to a terminal outside the device comprises transmitting the encoded data as a video stream to a terminal accessing a video conference through the video conference handshake protocol.

[0008] In some embodiments, receiving input data corresponding to a user's input on a device and identifying the input data as a primitive comprises receiving input data corresponding to a user's input on a touch panel and identifying the input data as at least one base primitive, and drawing the primitive to generate drawing data and writing the drawing data to an off-screen framebuffer comprises drawing the at least one base primitive to generate the drawing data and writing the drawing data to the off-screen framebuffer continuously.

[0009] In some embodiments, drawing the primitive to generate drawing data and writing the drawing data to an off-screen framebuffer comprises rendering the primitive to the off-screen framebuffer; calculating image features of a current frame and calculating feature differences between image features of the current frame and image features of a previous frame; determining isolated feature regions as noise regions; filling noise regions of the current frame with a system color; and inversely locating the primitive in which the noise regions are located and deleting the noise regions of the primitive.

[0010] In a second aspect, a data processing apparatus is provided. The data processing apparatus comprises: an identifying module configured to receive input data corresponding to a user's input on a device and identify the input data as a primitive; a drawing module configured to draw the primitive to generate drawing data and write the drawing data to an off-screen framebuffer; a display processing module configured to display the drawing data stored in the off-screen framebuffer on the device; an encoding module configured to encode the drawing data stored in the off-screen framebuffer to generate encoded data; and a transmission module configured to transmit the encoded data to a terminal outside the device.

[0011] In a third aspect, an electronic device is provided. The electronic device comprises: a memory configured to store a computer program; and a processor configured to execute the computer program to perform the above-mentioned data processing method.

[0012] In some embodiments, the electronic device comprises an electronic whiteboard.

[0013] In some embodiments, when the electronic whiteboard is in a ready mode, an application request for joining a current video conference is actively initiated by the electronic whiteboard, and the electronic whiteboard joins the video conference in different working modes.

[0014] In some embodiments, the video conference is initiated by a user in different modes, including: face-to-face mode, face-to-whiteboard mode.

[0015] According to embodiments of the present disclosure, all graphics of the electronic whiteboard are rendered to an off-screen frame buffer area, which is rendered to the screen of the electronic whiteboard and encoded into a video stream to be sent to a video conference system at the other end, improving the system efficiency of the electronic whiteboard and avoiding system performance overhead caused by multiple renderings.

[0016] Further, the enhanced rendering process based on image features improves the quality of graphics data and indirectly improves the rendering effect.

[0017] Further, the electronic whiteboard can join the video conference in different working modes, which is conducive to improving the user experience of the electronic whiteboard.

[0018] The above summary is only a summary of the technical solutions of the present application. In order to enable those skilled in the art to more clearly understand the technical solutions of the present application, and then implement the content recorded in the specification and drawings, and in order to enable the above and other purposes, characteristics and advantages of the present application to be more easily understood, the following will be described in combination with the specific embodiments of the present application and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are only used to show the principles, implementation manners, applications, characteristics and effects of the specific embodiments of the present application and other related contents, and cannot be considered as limitations of the present application.

[0020] Figure 1 A flowchart of a data processing method according to an embodiment of the present disclosure is shown;

[0021] Figure 2 An application flowchart of a data processing method according to an embodiment of the present disclosure is shown;

[0022] Figure 3 A flowchart of rendering graphics to an off-screen frame buffer area and displaying the off-screen frame buffer area with rendered graphics to the screen of an electronic whiteboard according to an embodiment of the present disclosure is shown;

[0023] Figure 4 A module schematic diagram of a data processing device according to an embodiment of the present disclosure is shown;

[0024] Figure 5A module schematic diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0025] Figure 6 A schematic diagram of an electronic whiteboard joining a video conference according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0026] To explain possible application scenarios, technical principles, specific schemes that can be implemented, and purposes and effects that can be achieved of the present application in detail, the following will be described in detail in combination with specific embodiments listed and with reference to the drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0027] In this article, the term "embodiment" means that the specific features, structures or characteristics described in combination with the embodiment can be included in at least one embodiment of the present application. The term "embodiment" appearing at various places in the specification does not necessarily refer to the same embodiment, and does not particularly limit the independence or association between other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, each technical feature mentioned in each embodiment can be combined in any way to form a corresponding implementable technical solution.

[0028] Unless otherwise defined, the meanings of the technical terms used herein are the same as those commonly understood by those skilled in the art to which the present application belongs; the use of related terms herein is only for the purpose of describing specific embodiments, and is not intended to limit the present application.

[0029] In the description of the present application, the phrase "and / or" is a description of the logical relationship between objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally represents that the associated objects before and after are a "or" logical relationship.

[0030] In the present application, terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual quantity, primary and secondary or order relationship between the entities or operations.

[0031] Unless otherwise specified, the use of terms such as “comprising,” “including,” “having,” or other similar expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0032] In this application, expressions such as "greater than", "less than", and "exceeding" are understood to exclude the stated number; expressions such as "above", "below", and "within" are understood to include the stated number. Furthermore, in the description of the embodiments of this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times", unless otherwise explicitly specified.

[0033] As mentioned in the background section, there are several technical issues that urgently need to be addressed when integrating electronic whiteboards into video conferencing systems, such as how to perform off-screen rendering to improve the system efficiency of electronic whiteboards and the limited range of modes available for integrating electronic whiteboards into video conferencing systems.

[0034] The following detailed description of embodiments according to the present disclosure will be made with reference to exemplary models and in conjunction with the accompanying drawings.

[0035] Figure 1 This is a flowchart illustrating a data processing method according to an embodiment of the present disclosure. (As shown) Figure 1 As shown, a data processing method includes the following steps S101 to S105.

[0036] In step S101, input data corresponding to user input on the device is received and the input data is identified as a graphic element. In some embodiments, input data corresponding to user input on a touch panel is received and the input data is identified as at least one basic graphic element. In some embodiments, the user can make different inputs on the electronic whiteboard in various ways, including drawing or writing, etc., and the various ways include drawing or writing or performing other operations on the electronic whiteboard with their own finger, a special stylus, or any other available tool. In some embodiments, the whiteboard program collects user input and identifies the user input as a basic graphic element.

[0037] In step S102, the graphics primitive is drawn to generate drawing data and the drawing data is written into an off-screen frame buffer. In some embodiments, the at least one base graphics primitive is drawn to generate the drawing data, and the drawing data is written into the continuous off-screen frame buffer. In some embodiments, the drawing engine is then invoked by the whiteboard program to draw the graphics primitive queue to the off-screen frame buffer.

[0038] In step S103, the drawing data stored in the off-screen frame buffer is displayed on the device. In some embodiments, the drawing data stored in the off-screen frame buffer is written into a screen frame buffer corresponding to the display screen of the device, and the drawing data stored in the screen frame buffer is displayed on the display screen.

[0039] In step S104, the drawing data stored in the off-screen frame buffer is encoded to generate encoded data. In some embodiments, the drawing data is encoded to generate the encoded data based on a video encoding format specified by a video conference handshake protocol. In some embodiments, the off-screen frame buffer with the drawn graphics primitives is encoded by the whiteboard program in a video encoding format specified by the video conference handshake protocol, such as H264 / H265 / VP9, etc.

[0040] In step S105, the encoded data is sent to a terminal outside the device. In some embodiments, the encoded data is sent as a video stream to a terminal accessing the video conference through the video conference handshake protocol. In some embodiments, the encoded data is sent by the whiteboard program to different video conference terminals according to the protocol of the video conference.

[0041] In the above steps, compared with the prior art graphics primitive which needs to be rendered twice independently (graphics primitive -> buffer -> display, graphics primitive -> buffer -> encoding), all graphics primitives of the electronic whiteboard of the present application are rendered to an off-screen frame buffer, which is rendered to the screen of the electronic whiteboard and also encoded into a video stream to be sent to the opposite video conference system. The rendering method only involves: graphics primitive -> buffer -> display / encoding, which improves the system efficiency of the electronic whiteboard and avoids the system performance overhead caused by multiple renderings.

[0042] Figure 2 An application flow diagram of a data processing method is shown. As shown in FIG. 1, the data processing method includes the following steps: Figure 2As shown, when a user draws or writes on an electronic whiteboard with a finger or a stylus, the smart whiteboard program collects user input and recognizes the most basic primitives. Next, the smart whiteboard program calls the drawing engine to draw the primitive queue into continuous memory (off-screen buffer). The smart whiteboard program displays the continuous memory with drawn primitives to the screen of the electronic whiteboard. The smart whiteboard program encodes the continuous memory with drawn primitives into a video encoding format specified by the video conference handshake protocol, such as H264 / H265 / VP9, etc. The smart whiteboard program sends the drawn primitives to the near-end and far-end video conference participants according to the video conference protocol. All primitives are rendered to the off-screen frame buffer, which is rendered to the screen of the electronic whiteboard and also encoded into a video stream to be sent to the opposite video conference system, improving the system efficiency of the electronic whiteboard and avoiding the system performance overhead caused by multiple renderings.

[0043] Figure 3 A flowchart of drawing the primitives to generate drawing data and writing the drawing data to an off-screen frame buffer is shown. As shown, it includes steps S301 to S305. Figure 3

[0044] In step S301, the primitives are rendered to the off-screen frame buffer.

[0045] In step S302, the image features of the current frame are calculated, and the feature difference between the image features of the current frame and the image features of the previous frame is calculated. Calculating the image features of the current frame mainly includes judging the image mode, correlation, directionality, etc. The mutation in the image features is generally noise, and in the subsequent judgment of noise regions, the isolated feature region is preferably a noise region.

[0046] In step S303, the isolated feature region is determined as a noise region.

[0047] In step S304, the noise region of the current frame is filled with a system color.

[0048] In step S305, the primitives where the noise region is located are reversely located, and the noise region of the primitives is deleted. In the above step S303, the isolated region is preferably a noise region, but in actual applications, the primitives may also contain non-isolated noise regions, which also need to be deleted.

[0049] In the above steps S301 to S305, the enhanced rendering process based on image features can improve the quality of primitive data and indirectly improve the rendering effect. The image features are preferably low-level image features: point, line, and surface features, and the image feature extraction algorithm is preferably SIFT, BRISK, ORB, or an AI-based image feature extraction algorithm.

[0050] Figure 4 ​A module schematic diagram of a data processing apparatus 400 according to an embodiment of the present disclosure is shown. The data processing apparatus 400 comprises an identification module 401, a rendering module 402, a display processing module 403, an encoding module 404, and a transmission module 405.

[0051] The identification module 401 is configured to receive input data corresponding to a user's input on a device and identify the input data as a graphic primitive. The rendering module 402 is configured to render the graphic primitive to generate rendering data and write the rendering data into an off-screen frame buffer. The display processing module 403 is configured to display the rendering data stored in the off-screen frame buffer on the device. The encoding module 404 is configured to encode the rendering data stored in the off-screen frame buffer to generate encoded data. In addition, the transmission module 405 is configured to send the encoded data to a terminal outside the device.

[0052] In some embodiments, the data processing apparatus 400 comprises an electronic whiteboard, all graphic primitives of which are rendered to an off-screen frame buffer by the above-mentioned processing apparatus 400, the off-screen frame buffer being rendered both to a screen of the electronic whiteboard and encoded into a video code stream to be sent to a peer video conference system, thereby improving system efficiency of the electronic whiteboard and avoiding system performance overhead caused by multiple renderings.

[0053] Figure 5 A module schematic diagram of an electronic device 500 according to an embodiment of the present disclosure is shown. The electronic device 500 comprises a memory 501 configured to store a computer program, and a processor 502 configured to execute the computer program to perform the above-mentioned data processing method.

[0054] In some embodiments, the electronic device 501 comprises an electronic whiteboard.

[0055] In some embodiments, when the electronic whiteboard is in a ready mode, an application request for joining a current video conference is actively initiated by the electronic whiteboard, and the electronic whiteboard joins the video conference in different working modes.

[0056] In some embodiments, the video conference is initiated by a user in different modes, the different modes comprising: a face-to-face mode and a face-to-whiteboard mode.

[0057] According to an embodiment of the present disclosure, all graphic primitives of an electronic whiteboard are rendered to an off-screen frame buffer, the off-screen frame buffer being rendered both to a screen of the electronic whiteboard and encoded into a video code stream to be sent to a peer video conference system, thereby improving system efficiency of the electronic whiteboard and avoiding system performance overhead caused by multiple renderings.

[0058] Further, based on the image feature enhanced rendering process, the quality of the image data is improved, and the rendering effect is indirectly improved.

[0059] Further, the electronic whiteboard can join the video conference in different working modes, which is conducive to improving the user experience of the electronic whiteboard.

[0060] In some embodiments, when the electronic whiteboard is in the Ready mode, the electronic whiteboard initiatively initiates a joining current video conference application. The video conference is initiated by a user in different modes, including Face-To-Face mode, Face-To-Whiteboard mode, and Face-To-Fusion (Face & Whiteboard) mode. In the Face-To-Fusion (Face & Whiteboard) mode, the Face and the Whiteboard are encoded in a single video (left-right symmetrical / asymmetrical split screen / asymmetrical superimposition), and a main identity ID (Face) is used for video call.

[0061] Figure 6 A schematic diagram of an electronic whiteboard joining a video conference is shown according to an embodiment of the present disclosure. As shown in Figure 6 When the electronic whiteboard system is ready (system / network / handwriting pen / conference service, etc.). The electronic whiteboard system initiatively applies to join the current video conference. The video conference is initiated by the user in the Face-To-Face mode. The video conference is initiated by the user in the Face-To-Whiteboard mode. The electronic whiteboard joins the video conference system, and the mode combination of user interaction is more. For example, assuming that A and B are video conference devices, and W is an electronic whiteboard device. The interaction mode includes but is not limited to A-B (W does not participate), AW-B (A and W have equal weight), and A|W-B (A and W both participate in voice, and only A or W picture is displayed). The electronic whiteboard has multiple working modes for joining the video conference, which is conducive to improving the user experience of the electronic whiteboard.

[0062] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of the present application, they do not limit the patent protection scope of the present application. Any equivalent structure or equivalent process substitution or modification based on the essential concept of the present application, using the content described in the specification and drawings of the present application, and directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, etc., are all included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized by, comprises: receiving input data corresponding to a user's input on a device and identifying the input data as a primitive; rendering the primitive to the off-screen framebuffer; calculating image features of the current frame, and calculating feature differences between the image features of the current frame and the image features of a previous frame; determining isolated feature regions as noise regions; filling the noise regions of the current frame with a system color; and back-locating the primitive in which the noise regions are located, and deleting the noise regions of the primitive.

2. The data processing method according to claim 1, characterized in that, comprises: an identifying module configured to receive input data corresponding to a user's input on a device and identify the input data as a primitive; a rendering module configured to render the primitive to the off-screen framebuffer; a display processing module configured to display the primitive stored in the off-screen framebuffer on the device; an encoding module configured to encode the primitive stored in the off-screen framebuffer to generate encoded data; and 4. The data processing method of claim 1, wherein, a transmission module configured to transmit the encoded data to terminals outside the device, wherein the encoding module is configured to encode the primitive to generate the encoded data based on a video encoding format specified by a video conference handshake protocol, wherein the transmission module is configured to transmit the encoded data to terminals accessing a video conference in a video stream through the video conference handshake protocol. comprises: a memory configured to store a computer program; and a processor configured to execute the computer program.

5. A data processing apparatus, characterized by, ​ ​ ​ ​ ​ ​ ​ ​ ​ 6. An electronic device, comprising: ​ ​ ​ A processor configured to execute the computer program to perform the method according to any one of claims 1-4.

7. The electronic device of claim 6, wherein, The electronic device comprises an electronic whiteboard.

8. The electronic device of claim 7, wherein, When the electronic whiteboard is in a ready mode, an application request for joining a current video conference is actively initiated by the electronic whiteboard, and the electronic whiteboard joins the video conference in different working modes.

9. The electronic device of claim 8, wherein, The video conference is initiated by a user in different modes, including a face-to-face mode and a face-to-whiteboard mode.

Images