A live broadcast interaction method, an electronic device, and a storage medium

Abstract

The application provides a live interaction method and related equipment, the method comprising: displaying a first drawing surface, the first drawing surface being a live video screen of a host end; displaying a second drawing surface on an upper layer of the first drawing surface based on a drawing interaction instruction, the second drawing surface being a transparent surface; generating drawing information based on contact interaction event information of a screen area to which the second drawing surface belongs; and displaying the drawing information on the live video screen through the second drawing surface. In this way, by combining the drawing function with the live technology, the user draws on the screen, and the drawing information is transmitted to other user terminals, the real-time sharing of user handwriting information can be realized, and the efficiency and quality of live communication and interaction are improved.

Description

Technical Field

[0001] This invention relates to the field of live streaming technology, and in particular to a live streaming interactive method and related equipment. Background Technology

[0002] Live streaming involves setting up independent signal acquisition equipment on-site, importing the captured audio and video into a broadcasting terminal, uploading it to a server via the network, and then publishing it to a website for viewers. Live streaming leverages and continues the advantages of the internet, using video conferencing for live online broadcasts. It allows for the real-time distribution of audio and video content to the internet, utilizing the internet's intuitiveness, speed, superior presentation, rich content, strong interactivity, lack of geographical restrictions, and segmentable audience to enhance the promotional effect of the event.

[0003] Current online live streaming interaction methods include text comments, voice chat, and video chat. However, in actual application scenarios, there may be situations where the host encounters operational problems and needs guidance from the audience. However, providing guidance through text or voice may have limitations, such as unclear meaning, difficulty in understanding, and low communication efficiency. Summary of the Invention

[0004] This invention provides a live streaming interaction method to address the problem that communication between the streamer and the user through text and voice alone is insufficient and may lead to misunderstandings during live streaming.

[0005] In a first aspect, the present invention provides a live streaming interactive method for a client, comprising:

[0006] Displaying a first drawing surface, the first drawing surface being the live video screen of the broadcaster's terminal;

[0007] Based on the drawing interaction command, a second drawing surface is displayed on top of the first drawing surface, and the second drawing surface is a transparent surface;

[0008] Drawing information is generated based on the touch interaction event information of the screen area to which the second drawing surface belongs;

[0009] The drawing information is displayed on the live video screen via the second drawing surface.

[0010] Optionally, the live video frame is associated with at least two second drawing surfaces, and the drawing information is generated jointly based on the touch interaction event information of the screen areas to which the at least two second drawing surfaces belong.

[0011] Optionally, the second drawing surface includes a broadcast drawing surface associated with the broadcaster client and a user drawing surface associated with the viewer client. The drawing information is jointly generated by the touch interaction event information of the screen area to which the broadcaster drawing surface and at least one user drawing surface belong.

[0012] Optionally, the touch interaction event information includes handwriting coordinate information, and the step of generating drawing information based on the touch interaction event information of the screen area to which the second drawing surface belongs includes:

[0013] Based on the handwriting coordinate information and the size information of the screen area to which the second drawing surface belongs, the handwriting coordinate ratio information is determined;

[0014] The drawing information is generated based on the stroke coordinate ratio information.

[0015] Optionally, the handwriting coordinate ratio information is a percentage of the handwriting coordinate information to the size information of the screen area. Generating the drawing information based on the handwriting coordinate ratio information includes:

[0016] The drawing information is generated based on the percentage data and color value data;

[0017] The step of displaying the drawing information on the live video screen through the second drawing surface includes:

[0018] The drawing information is sent to the server, so that the server sends the drawing information to other clients, so that it can be displayed on the live video screen on the other clients through the second drawing surface.

[0019] Optionally, displaying the drawing information on the live video screen via the second drawing surface includes:

[0020] When the client is a viewer client, the first handwriting coordinate information corresponding to the drawing information already present in the received live screen interface is obtained;

[0021] Obtain the second handwriting coordinate information recorded on the second drawing surface of the audience client, which is associated with the first handwriting coordinate information;

[0022] If the first handwriting coordinate information and the second handwriting coordinate information do not match, modify the first handwriting coordinate information and / or the second handwriting coordinate information so that the handwriting image corresponding to the second handwriting coordinate information covers the handwriting image corresponding to the first handwriting coordinate information.

[0023] Optionally, after the step of displaying the drawing information on the live video screen via the second drawing surface, the method further includes:

[0024] The corresponding target content is determined from the live video frame based on the handwriting coordinate information;

[0025] When the current coordinate information of the target content on the first drawing surface is detected to be inconsistent with the handwriting coordinate information on the second drawing surface, the handwriting coordinate information is adjusted to the coordinate position of the current coordinate information on the second drawing surface, so that the current note coordinates of the drawing information are synchronized with the current coordinate position of the target content.

[0026] Optionally, generating drawing information based on the touch interaction event information of the screen area to which the second drawing surface belongs includes:

[0027] In response to a drawing start command, listen for touch interaction event information of the screen area to which the second drawing surface belongs;

[0028] All of the aforementioned contact interaction event information will be used as drawing information to avoid triggering other application operations through the contact interaction event information.

[0029] Optionally, the drawing start command is obtained based on a floating window control, and the floating window control and / or the second drawing surface are displayed on top of the corresponding display content of all applications in the smart terminal to which the client belongs.

[0030] Optionally, the live streaming interaction method further includes:

[0031] Upon receiving drawing information generated by other clients from the server, a prompt message is generated;

[0032] Upon receiving the response instruction of the prompt information, the drawing information is displayed on the live video screen through the second drawing surface.

[0033] Secondly, the present invention also provides a live interactive device, comprising:

[0034] The first imaging module is used to display the first drawing surface, which is the live video screen of the broadcaster's terminal;

[0035] A drawing module is used to display a second drawing surface on top of the first drawing surface based on drawing interaction commands, wherein the second drawing surface is a transparent surface;

[0036] The interaction module is used to generate drawing information based on the touch interaction event information of the screen area to which the second drawing surface belongs;

[0037] The second imaging module is used to display the drawing information on the live video screen through the second drawing surface.

[0038] Thirdly, the present invention also provides a live interactive system, the system comprising a server and a client, the client comprising a broadcaster client and a viewer client;

[0039] The broadcaster's terminal is used to display a first drawing surface, which is used to display the live video screen of the broadcaster's terminal; based on the broadcaster's drawing interaction command, a second drawing surface is displayed on top of the first drawing surface; in response to the broadcaster's first touch interaction event information, first drawing information is generated, which is the touch interaction event information generated based on the screen area to which the second drawing surface of the broadcaster's terminal belongs; the first drawing information is displayed on the live video screen through the second drawing surface of the broadcaster's terminal to obtain a first video stream; and the first video stream is transmitted to the server.

[0040] The server is used to receive the first video stream and forward the first video stream to the viewer client;

[0041] A viewer client is configured to receive and play the first video stream; display a second drawing surface on top of the first drawing surface displaying the first video stream based on the viewer user's drawing interaction commands; generate second drawing information in response to the viewer user's second touch interaction event information, wherein the second touch interaction event information is based on the screen area to which the second drawing surface of the viewer client belongs; display the second drawing information on the video frame of the first video stream through the second drawing surface of the viewer client to obtain a second video stream; and feed the second video stream back to the server.

[0042] The server is also used to receive the second video stream and forward the second video stream to the broadcaster's client and other viewer clients.

[0043] Fourthly, the present invention also provides an electronic device, including a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory to implement the steps of the live interactive method as described in any of the first aspects above.

[0044] Fifthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the live interactive method as described in any of the first aspects.

[0045] As can be seen from the above technical solutions, this application provides a live streaming interaction method and related equipment. The method includes: displaying a first drawing surface, wherein the first drawing surface is the live video screen of the broadcaster; displaying a second drawing surface on top of the first drawing surface based on drawing interaction instructions, wherein the second drawing surface is a transparent surface; generating drawing information based on touch interaction event information of the screen area to which the second drawing surface belongs; and displaying the drawing information on the live video screen through the second drawing surface. Current online live streaming interaction methods include text comments, voice chat, and video chat. However, in actual application scenarios, there may be situations where the broadcaster encounters operational problems and needs guidance from the audience. However, providing guidance through text or voice may have problems such as unclear meaning, difficulty in understanding, and low communication efficiency, which have certain limitations. This application embodiment displays the live video screen as the first drawing surface and displays a transparent second drawing surface on top of the live video screen. Users can draw through touch interaction of the screen area to which the second drawing surface belongs, and finally the drawing information is displayed on the live video screen through the second drawing surface, which can realize real-time sharing of user handwriting information and improve the efficiency and quality of live streaming interaction. Attached Figure Description

[0046] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A schematic flowchart illustrating a live interactive method provided in an embodiment of this application;

[0048] Figure 2 A schematic diagram illustrating an embodiment of a live interactive method provided in this application, jointly drawn by the broadcaster's client and the viewer's client;

[0049] Figure 3 A schematic structural diagram of a live interactive device provided in an embodiment of this application;

[0050] Figure 4 A schematic flowchart illustrating a handwriting coordinate synchronization scheme for a live interactive method provided in this application embodiment;

[0051] Figure 5a A schematic diagram illustrating an embodiment of a live interactive method provided in this application, showing handwriting coordinate information and determined target content generated during drawing on the broadcaster's end;

[0052] Figure 5bA schematic diagram of an embodiment of a live interactive method provided in this application, showing that the target content moves, causing the current coordinate information to be inconsistent with the handwriting coordinate information;

[0053] Figure 5c A schematic diagram illustrating an embodiment of a live interactive method provided in this application, which adjusts handwriting coordinate information to the coordinate position of the current coordinate information;

[0054] Figure 6 A schematic diagram illustrating an embodiment of an electronic device provided in this application;

[0055] Figure 7 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided in this application. Detailed Implementation

[0056] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims. In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways, and the apparatus embodiments described below are merely exemplary.

[0057] This application provides a first embodiment of a live interactive method, such as... Figure 1 As shown, it includes:

[0058] Step S110: Display the first drawing surface, which is the live video screen of the broadcaster.

[0059] For example, before the step of displaying the first drawing surface, the broadcaster can create and join a live streaming channel. After successfully joining, the broadcaster can request microphone access from its smart terminal and enable local audio capture. The local audio can be captured directly through the microphone of the smart terminal or through an external device connected to the smart terminal. The broadcaster can perform special transcoding on the captured audio in real time and transmit the transcoded audio signal to the server through a designated channel. The server can receive the transcoded audio signal and forward it to other live streaming users, thereby enabling real-time sharing of audio content.

[0060] Before the step of displaying the first drawing surface, the broadcaster can enable screen sharing. After screen sharing is enabled, the broadcaster can request screen recording permission from its smart terminal and start capturing the screen. The screen can be directly captured from the smart terminal or video content captured by connecting an external device to the smart terminal. The broadcaster can perform special transcoding on the captured screen in real time and transmit the transcoded video signal to the server through a designated channel. The server can receive the transcoded video signal and forward it to other users, thus enabling real-time sharing of the broadcaster's screen content.

[0061] Users can apply to join a live chat channel created by the broadcaster. The joining process is the same as that on the broadcaster's end, enabling real-time audio sharing among users.

[0062] In this implementation, a SurfaceView can be added to the user's device to decode the screen-shared content from the broadcaster's side and draw it onto this SurfaceView. It should be noted that the SurfaceView can control the drawing format, size, and drawing position.

[0063] Step S120: Based on the drawing interaction command, display a second drawing surface on top of the first drawing surface. The second drawing surface is a transparent surface.

[0064] For example, the second drawing surface mentioned above is a transparent surface on top of the first drawing surface, used for image drawing. A transparent surface can be drawn using the Canvas function on the SurfaceView. The method for drawing a transparent surface can be to call the Draw Color method in the Canvas function, pass the parameter Color.Transparent to the drawing method to control the drawing color to be transparent, and complete the drawing of the transparent surface.

[0065] Step S130: Generate drawing information based on the touch interaction event information of the screen area to which the second drawing surface belongs.

[0066] For example, the aforementioned touch interaction event information refers to screen touch event information performed by the user on the screen area of ​​the second drawing surface, including tap touch and swipe touch. Handwriting information can be determined based on the touch interaction event information of the screen area of ​​the second drawing surface. A storage module for storing the handwriting information can be established, including a Path module and a Paint module. The Path module can be used to store handwriting coordinate information determined based on the touch interaction event information of the screen area of ​​the second drawing surface, and the Paint module can be used to store handwriting color values ​​and width information determined based on the touch interaction event information of the screen area of ​​the second drawing surface.

[0067] According to some embodiments, the above-mentioned generation of drawing information based on the touch interaction event information of the screen area to which the second drawing surface belongs includes:

[0068] In response to the drawing start command, listen for touch interaction event information of the screen area to which the second drawing surface belongs;

[0069] All of the above-mentioned contact interaction event information will be used as drawing information to avoid triggering other application operations through the above-mentioned contact interaction event information.

[0070] For example, the aforementioned drawing initiation instruction is an instruction to initiate the receipt of touch interaction event information of the screen area to which the second drawing surface belongs. A listener can be injected into the SurfaceView, wherein the listener can be used to listen for touch interaction event information generated by the client user on the screen area of ​​the smart terminal. The listener can also inform the system of the smart terminal to use all the aforementioned touch interaction event information as drawing information, which will be consumed by the second drawing surface, without having to send the aforementioned touch interaction event information to other applications.

[0071] By listening to the touch interaction event information of the screen area to which the second drawing surface belongs, and using all of the above-mentioned touch interaction event information as drawing information, it is possible to avoid opening other applications during the operation, which would affect the smoothness and accuracy of the operation and thus affect the user experience.

[0072] According to some embodiments, the above-mentioned drawing start instruction is obtained based on a floating window control, and the above-mentioned floating window control and / or the above-mentioned second drawing surface are displayed on top of the corresponding display content of all applications in the smart terminal to which the client belongs.

[0073] For example, the aforementioned floating window control can receive the aforementioned drawing start command and receive the aforementioned contact interaction event information according to the aforementioned drawing start command. A "Start Drawing" button can be set as a floating window control. The aforementioned "Start Drawing" button can be on the same layer as the aforementioned second drawing surface, or it can be on the upper layer of the aforementioned second drawing surface. When the aforementioned "Start Drawing" button is on the same layer as the aforementioned second drawing surface, the aforementioned "Start Drawing" button can be displayed together with the aforementioned second drawing surface on the upper layer of the corresponding display content of all applications in the aforementioned smart terminal. When the aforementioned "Start Drawing" button is on the upper layer of the aforementioned second drawing surface, the aforementioned second drawing surface is displayed on the upper layer of the corresponding display content of all applications in the aforementioned smart terminal. When a user clicks the "Start Graffiti" button, a color palette is opened for the user to select colors. Alternatively, after screen sharing is enabled on the broadcaster's end, the second drawing surface can always be displayed on the first drawing surface. By clicking the "Start Graffiti" button, the system can begin listening for touch interaction events in the screen area of ​​the second drawing surface. Conversely, after screen sharing is enabled on the broadcaster's end, the second drawing surface can be hidden from the first drawing surface, and the "Start Graffiti" button can be used to activate the second drawing surface, simultaneously listening for touch interaction events in the screen area of ​​the second drawing surface.

[0074] Since the second drawing surface is transparent, displaying it on the first drawing surface will not affect the presentation of the video content on the first drawing surface. The broadcaster can stream games via mobile and PC, but some game applications do not support PC operation and can only run on mobile. When a broadcaster uses a mobile device for game streaming, full-screen content is usually shared, so other applications need to be in the background. Therefore, the floating window control and / or the second drawing surface need to be displayed on top of the corresponding display content of all applications on the user's smart terminal. When the broadcaster is sharing their screen, the floating window control can be used to directly open the second drawing surface, avoiding the need for the broadcaster to switch applications to open the canvas function, simplifying the user's operation, and protecting the user's privacy.

[0075] According to some embodiments, the aforementioned touch interaction event information includes handwriting coordinate information, and the generation of drawing information based on the touch interaction event information of the screen area to which the second drawing surface belongs includes:

[0076] Based on the above handwriting coordinate information and the size information of the screen area to which the second drawing surface belongs, the handwriting coordinate ratio information is determined;

[0077] The above drawing information is generated based on the stroke coordinate ratio information.

[0078] For example, the coordinate ratio information [a,b] of the handwriting can be determined by using a = x / m and b = y / n, based on the coordinate information [x,y] of the handwriting and the width and height of the second drawing surface, where m is the width of the second drawing surface and n is the height of the second drawing surface; the width coordinate ratio information c of the handwriting can also be determined by using c = w / m, based on the width coordinate information w of the handwriting and the width of the second drawing surface, and the drawing information can be determined based on the coordinate ratio information of the handwriting and the width coordinate ratio information of the handwriting.

[0079] Since users watch live streams on a variety of devices with different screen sizes and resolutions, the same handwriting coordinate information may appear differently on different smart terminals, leading to communication errors. By using coordinate mapping, based on the handwriting coordinate information and the size information of the screen area to which the second drawing surface belongs, the handwriting coordinate ratio information can be determined, which can avoid the above differences, improve the accuracy of interaction, and thus enhance the user experience.

[0080] According to some embodiments, the above-mentioned handwriting coordinate ratio information is a percentage of the handwriting coordinate information to the size information of the screen area. Generating the drawing information based on the handwriting coordinate ratio information includes:

[0081] The above drawing information is generated based on the percentage data and color value data mentioned above.

[0082] The above-mentioned display of the drawing information on the live video screen via the second drawing surface includes:

[0083] The above drawing information is sent to the server so that the server can send the drawing information to other clients so that it can be displayed on the live video screen on the second drawing surface in the other clients.

[0084] For example, when a user opens a color palette, the color value data of the aforementioned color can be determined based on the color selected by the user. For instance, when drawing is performed by the user, the user's smart terminal can transcode the drawing information into a byte array and send this byte array to the server via a special channel at a frequency of once every 30 milliseconds. The server can then send the byte array to the broadcaster's end, restoring the handwriting on the broadcaster's second drawing surface. The broadcaster's screen sharing function enables screen sharing with other users. Alternatively, the server can send the byte array to other clients, allowing those clients to directly restore the handwriting on the user's second drawing surface.

[0085] The above drawing information is generated based on the percentage data and color value data. The drawing information is then sent to other clients via the server. On these other clients, the drawing information is displayed on the live video screen via the second drawing surface. The handwriting information can be adapted to the screen size and resolution of different users. The display is based on the color value data, which can ensure the accuracy of color presentation and avoid color difference, thereby ensuring the accuracy of interactive information.

[0086] Step S140: Display the above drawing information on the above live video screen through the above second drawing surface.

[0087] For example, upon receiving the corresponding instruction of the above prompt information, the drawing information can be extracted from the Path module and the Paint module, the Draw Path stroke drawing method in the Canvas function can be called, and the drawing information can be passed to the Canvas so that the drawing information is presented on the second drawing surface.

[0088] According to some embodiments, upon receiving drawing information generated by other clients from the server, a prompt message is generated;

[0089] Upon receiving the response instruction of the above prompt information, the above drawing information is displayed on the above live video screen through the above second drawing surface.

[0090] For example, the user can proactively send an interaction request to the broadcaster, or the broadcaster can proactively authorize the user to perform interactive drawing. Upon receiving the interaction request, the server can generate a notification indicating that another user needs to interact. The broadcaster can receive the notification from the server. If the broadcaster accepts the notification and agrees to interact, the user draws on the viewer's screen area using a second drawing surface. The server then sends the drawing information from the user to the broadcaster. The broadcaster can then recreate the drawing on the broadcaster's second drawing surface and display it on top of the live video feed.

[0091] According to some embodiments, the above-mentioned live video frame is associated with at least two second drawing surfaces, and the above-mentioned drawing information is jointly generated based on the contact interaction event information of the screen areas to which the at least two second drawing surfaces belong.

[0092] For example, the broadcaster can grant drawing permissions to only two viewer clients, and the multiple viewer clients granted drawing permissions can draw together.

[0093] This allows multiple viewer clients to jointly identify and resolve issues encountered during the broadcast on the streamer's end, improving problem-solving efficiency.

[0094] According to some embodiments, the second drawing surface mentioned above includes a broadcaster drawing surface associated with the broadcaster client and a user drawing surface associated with the viewer client. The drawing information is jointly generated by the touch interaction event information of the screen area to which the broadcaster drawing surface and at least one user drawing surface belong.

[0095] For example, the live video feed from the broadcaster can be displayed on multiple clients, including the broadcaster and viewer clients. The broadcaster, while ensuring it has drawing permissions, can grant drawing permissions to at least one viewer client, allowing both the broadcaster and viewer clients to jointly draw. If the broadcaster grants drawing permissions to viewer client A, then viewer client A and the broadcaster can perform drawing operations on their respective second drawing surfaces. For example, as... Figure 2 As shown, the streamer, unsure how to configure game character skills, grants drawing permissions to themselves and viewer client A. The user client can draw a circle on icon 1 (the entry point to the skill configuration interface) in the live stream interface using a second drawing surface. The streamer, believing icon 2 is the correct entry point, can draw a circle on icon 2. This allows for real-time interaction between the streamer and user clients, avoiding issues like prolonged communication times, low efficiency, and disruptions to the live stream caused by allowing only one client to draw.

[0096] For example, while ensuring that the broadcaster has drawing permissions, it can also grant drawing permissions to two or more viewer clients, allowing multiple viewer clients and the broadcaster to draw together. This saves time that would otherwise be needed to communicate with different users individually on the same issue, further improving the level of interaction and communication efficiency between the broadcaster and the user.

[0097] This makes communication between the broadcaster and the viewer more interactive, especially when communication needs to be combined with the content displayed on the live broadcast interface, resulting in higher communication efficiency and faster problem resolution.

[0098] According to some embodiments, displaying the above-mentioned drawing information on the above-mentioned live video screen through the above-mentioned second drawing surface includes:

[0099] When the aforementioned client is a viewer client, the first handwriting coordinate information corresponding to the aforementioned drawing information already present in the received live screen interface is obtained.

[0100] Obtain the second handwriting coordinate information recorded on the second drawing surface of the aforementioned audience client, which is associated with the first handwriting coordinate information;

[0101] If the first handwriting coordinate information and the second handwriting coordinate information do not match, modify the first handwriting coordinate information and / or the second handwriting coordinate information so that the handwriting image corresponding to the second handwriting coordinate information covers the handwriting image corresponding to the first handwriting coordinate information.

[0102] For example, the first handwriting coordinate information is the drawing information received by the broadcaster from the viewer's client. This drawing information is then sent to the server, which in turn sends it to other clients to display the handwriting information on the live video screen in the viewer's client. The second handwriting coordinate information is the coordinate information of notes drawn by the user on the user's second drawing surface on the viewer's client. The first handwriting coordinate information corresponding to the drawing information on the live screen interface can be obtained, as well as the handwriting information associated with the first handwriting coordinate information on the user's second drawing surface on the viewer's client, used as the second handwriting coordinate information. If the first and second handwriting coordinate information do not match, the second handwriting coordinate information can be modified based on the first handwriting coordinate information, or the first handwriting coordinate information can be modified based on the second handwriting coordinate information, or both can be modified simultaneously.

[0103] When a user draws on the second drawing surface, the user's drawing information will be displayed on the live broadcast screen on the broadcaster's side. The live broadcast screen interface on the broadcaster's side will be transmitted to the user's side in real time. This may cause the handwriting on the user's second drawing surface to not completely overlap with the handwriting displayed on the live broadcast screen interface, affecting the user's experience. By correcting the handwriting coordinate information, the above-mentioned problem of incomplete handwriting overlap can be avoided.

[0104] like Figure 3 As shown, Figure 3 This is a schematic structural diagram of a live interactive device provided in an embodiment of this application.

[0105] This application provides a live interactive device 300, including a first imaging module 301, a drawing module 302, an interactive module 303, and a second imaging module 304, wherein:

[0106] The first imaging module 301 is used to display the first drawing surface, which is the live video screen of the broadcaster terminal;

[0107] The drawing module 302 is used to display a second drawing surface on top of the first drawing surface based on drawing interaction commands, wherein the second drawing surface is a transparent surface;

[0108] Interactive module 303 generates drawing information based on the touch interaction event information of the screen area to which the second drawing surface belongs;

[0109] The second imaging module 304 displays the above-mentioned drawing information on the above-mentioned live video screen through the second drawing surface.

[0110] A live interactive device 300 can achieve Figure 1 The various processes implemented in the method embodiments are not described in detail here to avoid repetition. Furthermore, the live interactive device 300 can combine drawing functions with live streaming technology, allowing users to draw on the screen and transmit the drawing information to other user terminals. This enables real-time sharing of user handwriting information, improving the efficiency and quality of live communication and interaction.

[0111] Furthermore, the present invention also proposes a live interactive system, the system comprising a server and a client, the client comprising a broadcaster client and a viewer client;

[0112] In a specific implementation, both the broadcaster's client and the viewer's client can be equipped with the aforementioned live interactive device 300;

[0113] The broadcaster's terminal is used to display a first drawing surface, which is used to display the live video screen of the broadcaster's terminal; based on the broadcaster's drawing interaction command, a second drawing surface is displayed on top of the first drawing surface; in response to the broadcaster's first touch interaction event information, first drawing information is generated, which is the touch interaction event information generated based on the screen area to which the second drawing surface of the broadcaster's terminal belongs; the first drawing information is displayed on the live video screen through the second drawing surface of the broadcaster's terminal to obtain a first video stream; and the first video stream is transmitted to the server.

[0114] The server is used to receive the first video stream and forward the first video stream to the viewer client;

[0115] A viewer client is used to receive and play the first video stream; based on the viewer's drawing interaction instructions, a second drawing surface is displayed on top of the first drawing surface displaying the first video stream; in response to the viewer's second touch interaction event information, second drawing information is generated, the second touch interaction event information being generated based on the screen area to which the second drawing surface of the viewer client belongs; the second drawing information is displayed on the video frame of the first video stream through the second drawing surface of the viewer client to obtain a second video stream; and the second video stream is fed back to the server.

[0116] The server is also used to receive the second video stream and forward the second video stream to the broadcaster's client and other viewer clients.

[0117] Specifically, this embodiment can be applied to scenarios where broadcasters and users interact via live stream. For example, when a broadcaster is live streaming a game, they draw a trajectory (i.e., the generated drawing information) on the live video screen displayed on the broadcaster's end. Users watch the broadcaster's live video screen in the broadcaster's live room and draw a trajectory on the live video screen played on the viewer's client. The trajectories drawn by the broadcaster and the user on the same live video screen are synchronously rendered by channel transmission.

[0118] Furthermore, this application also provides a second embodiment of a live interactive method, such as... Figure 4 As shown, it includes:

[0119] In this embodiment, after step S140, which displays the drawing information on the live video screen via the second drawing surface, the method further includes:

[0120] Step S410: Determine the corresponding target content from the live video frame based on the handwriting coordinate information.

[0121] Understandably, before generating the initial drawing information, the live interactive device 300 captures the coordinate information of the handwriting generated by the user's touch on the mobile phone screen. This handwriting coordinate information is the coordinate A of the initially generated drawing information, such as... Figure 5a As shown, the broadcaster user draws a circle around person 1 in the live video frame at coordinate A on the terminal screen. This circle is the drawing information located on the second drawing surface, and person 1 is the target content located on the first drawing surface (live video frame).

[0122] Step S420: When it is detected that the current coordinate information of the target content on the first drawing surface is inconsistent with the handwriting coordinate information on the second drawing surface, the handwriting coordinate information is adjusted to the coordinate position of the current coordinate information on the second drawing surface for display, so that the current note coordinates of the drawing information are synchronized with the current coordinate position of the target content.

[0123] Understandably, the target content (person 1) in the live video frame will continuously move within the live video frame. The live interactive device 300 will capture the current coordinate information of the target content (person 1) on the screen in real time, and compare the current coordinate information of the target content (person 1) with the handwriting coordinate information of the user on the second drawing surface (i.e., Figure 5a The coordinates (A) of the circle drawn on person 1 are compared to determine whether the target content in the live video has moved. If the comparison result shows that the current coordinates of person 1 are different from the coordinates (A) of the handwriting, then it is determined that the target content (person 1) in the live video has moved. Figure 5b As shown, the current coordinates of Person 1 have moved to coordinate B; the live interactive device 300 transmits the current coordinates of the target content (Person 1) after its movement on the screen to the second drawing surface in real time. On the second drawing surface, the current coordinates of the drawing information (the circle drawn by the user for Person 1) are adjusted from the original coordinate A to the coordinate position of coordinate B (e.g., ...). Figure 5c As shown in the diagram, the drawing information with adjusted note coordinates on the second drawing surface is encoded with the live video feed to form a third video stream. This third video stream is then transmitted to the server, which broadcasts it to other viewer clients in the live stream room. This allows all viewers in the live stream room to see the effect where the note coordinates of the drawing information are synchronized with the current coordinates of the target content as it moves across the screen. This achieves the goal that once the broadcaster / viewer client circles and marks the target content on the live video feed, even if the target content moves, the broadcaster's and viewers' note markings are synchronized with the target content's movement trajectory within the live video feed.

[0124] like Figure 6 As shown, Figure 6 This is a schematic structural diagram of an electronic device provided in an embodiment of this application.

[0125] This application provides an electronic device 600, including a memory 610, a processor 620, and a computer program 611 stored in the memory 610 and executable on the processor 620. When the processor 620 executes the computer program 611, it performs the following steps:

[0126] Displaying a first drawing surface, wherein the first drawing surface is the live video screen of the broadcaster's terminal;

[0127] Based on the drawing interaction command, a second drawing surface is displayed on top of the first drawing surface, and the second drawing surface is a transparent surface;

[0128] Based on the touch interaction event information of the screen area to which the second drawing surface belongs, drawing information is generated;

[0129] The above drawing information is displayed on the above live video screen through the above second drawing surface.

[0130] In practical implementation, when the processor 620 executes the computer program 611, it can achieve... Figure 1 Any of the corresponding implementation methods in the embodiments.

[0131] Since the electronic device described in this embodiment is a device used to implement a device in the embodiments of this application, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the method described in the embodiments of this application. Therefore, how the electronic device implements the method in the embodiments of this application will not be described in detail here. Any device used by those skilled in the art to implement the method in the embodiments of this application falls within the scope of protection of this application.

[0132] like Figure 7 As shown, Figure 7 This is a schematic structural diagram of a computer-readable storage medium provided in an embodiment of this application.

[0133] This embodiment provides a computer-readable storage medium 700 on which a computer program 711 is stored. When the computer program 711 is executed by a processor, it performs the following steps:

[0134] Displaying a first drawing surface, wherein the first drawing surface is the live video screen of the broadcaster's terminal;

[0135] Based on the drawing interaction command, a second drawing surface is displayed on top of the first drawing surface, and the second drawing surface is a transparent surface;

[0136] Based on the touch interaction event information of the screen area to which the second drawing surface belongs, drawing information is generated;

[0137] The above drawing information is displayed on the above live video screen through the above second drawing surface.

[0138] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0139] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0141] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0142] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0143] This application also provides a computer program product, which includes computer software instructions that, when executed on a processing device, cause the processing device to perform actions such as... Figure 1 The process of the live interactive method in the corresponding embodiment.

[0144] The aforementioned computer program product includes one or more computer instructions. When the aforementioned computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The aforementioned computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The aforementioned computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0145] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0147] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0148] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0149] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0150] In summary, the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A live interaction method, characterized in that, For a client, comprising: displaying a first drawing surface for displaying a live video picture of a host end; based on a drawing interaction instruction, displaying a second drawing surface on an upper layer of the first drawing surface, the second drawing surface being a transparent surface; generating drawing information based on contact interaction event information of a screen area to which the second drawing surface belongs; displaying the drawing information on the live video picture through the second drawing surface; the live video picture is associated with at least two second drawing surfaces, and the drawing information is generated based on contact interaction event information of screen areas to which at least two second drawing surfaces belong; the second drawing surface includes a host drawing surface associated with a host side client and a user drawing surface associated with a viewer client, and the drawing information is generated based on contact interaction event information of screen areas to which the host drawing surface and at least one user drawing surface belong; the contact interaction event information includes handwriting coordinate information, and the generation of the drawing information based on the contact interaction event information of the screen area to which the second drawing surface belongs includes: determining handwriting coordinate proportion information based on the handwriting coordinate information and size information of the screen area to which the second drawing surface belongs; generating the drawing information according to the handwriting coordinate proportion information; the handwriting coordinate proportion information is percentage data of the handwriting coordinate information and the size information of the screen area, and the generation of the drawing information according to the handwriting coordinate proportion information includes: generating the drawing information according to the percentage data and color value data; the displaying of the drawing information on the live video picture through the second drawing surface includes: sending the drawing information to a server, so that the server sends the drawing information to other clients, and displays the drawing information on the live video picture through the second drawing surface in other clients; the displaying of the drawing information on the live video picture through the second drawing surface includes: in the case that the client is a viewer client, obtaining first handwriting coordinate information corresponding to the drawing information already existing in a received live screen interface; obtaining second handwriting coordinate information associated with the first handwriting coordinate information recorded by the second drawing surface of the viewer client; in the case that the first handwriting coordinate information and the second handwriting coordinate information do not match, modifying the first handwriting coordinate information and / or the second handwriting coordinate information, so that a handwriting picture corresponding to the second handwriting coordinate information covers a handwriting picture corresponding to the first handwriting coordinate information; after the step of displaying the drawing information on the live video picture through the second drawing surface, the method further includes: determining corresponding target content from the live video picture according to the handwriting coordinate information; adjust the handwriting coordinate information on the second drawing surface to the coordinate position of the current coordinate information of the target content when it is detected that the current coordinate information of the target content in the first drawing surface is inconsistent with the handwriting coordinate information on the second drawing surface, so that the current note coordinate of the drawing information is synchronized with the current coordinate position of the target content; generating drawing information based on the contact interaction event information of the screen area to which the second drawing surface belongs, comprising: listening to the contact interaction event information of the screen area to which the second drawing surface belongs in response to a drawing start instruction; using all the contact interaction event information as drawing information to avoid triggering other application operations through the contact interaction event information; the drawing start instruction is obtained based on a floating window control, and the floating window control and / or the second drawing surface is displayed above the display content corresponding to all application programs in the intelligent terminal to which the client belongs.

2. The method of claim 1, wherein, Further comprising: generating prompt information in the case of receiving drawing information generated by other clients from the server; displaying the drawing information on the live video picture through the second drawing surface in the case of receiving a response instruction to the prompt information.

3. An electronic device comprising a memory, a processor, characterized in that, The processor is used to implement the steps of the live interaction method according to any one of claims 1 to 2 when executing the computer program stored in the memory.

4. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is used to implement the steps of the live interaction method according to any one of claims 1 to 2 when executed by the processor.

Images