Method for processing live streaming, device, and storage medium

Abstract

Embodiments of the present disclosure provide a method for processing live streaming, a device, and a storage medium. A streamer end device adds supplemental enhancement information (SEI) to live stream data, the SEI comprising first layout information of a livestream interface; meanwhile, preset coding information is added to a livestream interface image of the live stream data, and the preset coding information can comprise second layout information of the livestream interface, so that an audience end device has two ways to acquire the layout information of the livestream interface; and when the SEI is lost, the second layout information of the livestream interface is acquired on the basis of the preset coding information in the livestream interface image, ensuring the audience end device can stably acquire the layout information of the livestream interface, promptly perceive a change in the livestream interface layout, and perform corresponding layout control, and ensuring user experience.

Description

Live broadcast processing method, device and storage medium

[0001] This application claims priority to Chinese Patent Application No. 202311599932.8 filed on November 27, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby cited in their entirety as a part of this application. Technical Field

[0002] The embodiments of the present disclosure relate to a live broadcast processing method, device, and storage medium. Background Art

[0003] In online live streaming scenarios, the layout of the live studio interface may change. For example, in a live broadcast with microphones connected, the host and the connected guests in the live studio can have real-time audio and video calls, and the content of the audio and video calls can be pushed to the viewer's device after multiple streams are merged. However, as the number of connected hosts and guests changes, the layout of the live studio interface will also change, and the screen content of the viewer's device will also change accordingly.

[0004] Since the viewer-end device needs to load some interface components according to the live broadcast room interface layout, it needs to perceive the changes in the live broadcast room interface layout, for example, by carrying Supplemental Enhancement Information (SEI) in the live broadcast stream data to convey the changes in the live broadcast room interface layout.

[0005] However, the supplementary enhancement information carried in the live streaming data is easily lost, resulting in the viewer's end device being unable to perceive the changes in the live broadcast room interface layout, affecting the user experience. Summary of the Invention

[0006] The embodiments of the present disclosure provide a live broadcast processing method, device, and storage medium to ensure that the viewer-end device can stably obtain the layout information of the live broadcast room interface and promptly perceive changes in the live broadcast room interface layout.

[0007] In a first aspect, an embodiment of the present disclosure provides a live broadcast processing method, applied to a viewer terminal device, the method comprising:

[0008] In the case where the live streaming data does not include the supplementary enhancement information, obtaining a live studio interface image; wherein the supplementary enhancement information includes first layout information of the live studio interface;

[0009] Acquire second layout information of the live broadcast room interface according to preset coding information included in the live broadcast room interface image;

[0010] The live broadcast room interface is layout controlled according to the second layout information.

[0011] In a second aspect, an embodiment of the present disclosure provides a live broadcast processing method, which is applied to a host terminal device, and the method includes:

[0012] Adding supplementary enhancement information to the live streaming data of the live broadcast room, wherein the supplementary enhancement information includes first layout information of the live broadcast room interface;

[0013] Adding preset coding information to each frame of the live broadcast room interface image of the live broadcast stream data, wherein the preset coding information includes the second layout information of the live broadcast room interface;

[0014] The live streaming data is pushed to the server, so that the viewer's terminal device can obtain the live streaming data from the server for playback.

[0015] In a third aspect, an embodiment of the present disclosure provides a live broadcast processing device, including:

[0016] An image acquisition unit is configured to acquire a live studio interface image when the live streaming data does not include supplementary enhancement information; wherein the supplementary enhancement information includes first layout information of the live studio interface;

[0017] A parsing unit, configured to obtain second layout information of the live broadcast room interface according to preset coding information included in the live broadcast room interface image;

[0018] A layout control unit is used to control the layout of the live broadcast room interface according to the second layout information.

[0019] In a fourth aspect, an embodiment of the present disclosure provides a live broadcast processing device, including:

[0020] A data processing unit, configured to add supplementary enhancement information to the live streaming data of the live broadcast room, wherein the supplementary enhancement information includes first layout information of the live broadcast room interface;

[0021] A synthesis unit, configured to add preset coding information to each frame of the live studio interface image of the live streaming data, wherein the preset coding information includes second layout information of the live studio interface;

[0022] The communication unit is used to push the live streaming data to the server, so that the viewer device can obtain the live streaming data from the server for playback.

[0023] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0024] The memory stores computer-executable instructions;

[0025] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the live broadcast processing method described in the first aspect and various possible designs of the first aspect.

[0026] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the live broadcast processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0027] In the seventh aspect, an embodiment of the present disclosure provides a computer program product, including computer execution instructions. When a processor executes the computer execution instructions, it implements the live broadcast processing method described in the first aspect and various possible designs of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, a brief introduction will be given below to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0029] Figure 1 is an example diagram of a live broadcast room interface layout;

[0030] FIG2 is a flow chart of a live broadcast processing method provided by an embodiment of the present disclosure;

[0031] FIG3 is a flow chart of a live broadcast processing method provided by another embodiment of the present disclosure;

[0032] FIG4 is a flow chart of a live broadcast processing method provided by another embodiment of the present disclosure;

[0033] FIG5 is a structural block diagram of a live broadcast processing device provided by an embodiment of the present disclosure;

[0034] FIG6 is a structural block diagram of a live broadcast processing device provided by another embodiment of the present disclosure; and

[0035] FIG7 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0037] The blind watermarking function adds an invisible watermark to an image. This watermark does not affect the image's appearance while ensuring its originality. By extracting the blind watermark, the invisible watermark can be extracted to verify the image's ownership. One method of adding blind watermarks is frequency-domain digital watermarking. This involves transforming the image into the frequency domain (wavelet domain) through some kind of transformation (Fourier transform, discrete cosine transform, wavelet transform, etc.), adding the watermark to the image in the frequency domain, and then converting the image to the spatial domain through an inverse transform. Frequency-domain blind watermarks are more invisible and resistant to attacks. The watermark content can still be parsed even after the image has been compressed, cropped, or scaled.

[0038] Supplemental Enhancement Information (SEI) is text data inserted into the audio and video stream to convey additional information. It can be precisely synchronized with the relevant audio and video content and reach the client through the transmission link.

[0039] In a live broadcast scenario, the interface layout of the live broadcast room may change. For example, in a live broadcast scenario with connected microphones, the host and the connected guests in the live broadcast room can have real-time audio and video calls, and the content of the audio and video calls can be pushed to the viewer's device after multiple streams are merged. However, as the number of connected hosts and guests changes, the interface layout of the live broadcast room will also change, and the screen content of the viewer's device will also change accordingly. As shown in Figure 1, different live broadcast room interface layouts can be presented as the number of connected hosts and guests changes.

[0040] Because viewer devices need to load certain interface components based on the live studio layout, such as the avatars, names, and live stream PK scores of each connected host, they need to be able to detect changes in the live studio layout. This can be achieved by including supplementary enhancement information (SEI) in the live stream data to communicate changes to the live studio layout to viewer devices. SEI is aligned with the video frame, making it the most timely and accurate way to detect changes in the video image.

[0041] SEI, as supplementary enhancement information in the H.264 and H.265 encoding formats, is easily lost during streaming media transmission links, especially during transcoding. As a data transmission link, it is relatively unstable, and can lead to problems such as complete SEI loss and SEI truncation, resulting in parsing failures. Currently, live broadcast and live-streaming services rely heavily on SEI to transmit information and status. The poor stability of the SEI link prevents viewers from perceiving changes in the live studio interface layout, impacting the user experience.

[0042] In order to solve the above technical problems, the present disclosure provides a live broadcast processing method, in which the anchor-end device adds supplementary enhancement information to the live broadcast stream data, and the supplementary enhancement information includes the first layout information of the live broadcast room interface. At the same time, preset coding information is added to the live broadcast room interface image of the live broadcast stream data, and the preset coding information may include the second layout information of the live broadcast room interface. In this way, there are two ways for the audience-end device to obtain the layout information of the live broadcast room interface. It is preferred to obtain the layout information of the live broadcast room interface from the supplementary enhancement information. When the supplementary enhancement information is lost, the layout information of the live broadcast room interface can be obtained through the preset coding information in the live broadcast room interface image. In this way, it is ensured that the audience-end device can stably obtain the layout information of the live broadcast room interface, and timely perceive the changes in the layout of the live broadcast room interface, and then perform corresponding layout control, thereby ensuring user experience.

[0043] The live broadcast processing method disclosed herein is applicable to live broadcast scenarios, which include a host-end device, a server-end device, and an audience-end device. The host-end device pushes the live stream data to the server-end, and the audience-end device pulls and plays the live stream data from the server-end. The specific process will not be repeated here.

[0044] It should be noted that the user information and data involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0045] The live broadcast processing method disclosed herein will be described in detail below with reference to specific embodiments.

[0046] Refer to Figure 2, which is a flow chart of a live broadcast processing method provided by an embodiment of the present disclosure. The method of this embodiment can be applied to a viewer terminal device in a live broadcast scenario. The live broadcast processing method includes:

[0047] S201. When the live streaming data does not include supplementary enhancement information, obtain a live studio interface image; wherein the supplementary enhancement information includes first layout information of the live studio interface.

[0048] In this embodiment, the live streaming data of the live broadcast room will carry supplementary enhancement information, and the supplementary enhancement information includes the first layout information of the live broadcast room interface. Since the supplementary enhancement information is synchronized with the live streaming data, the first layout information of the live broadcast room interface can be the layout information of the live broadcast room interface of the current frame of the live streaming data. In the live broadcast room connected microphone scenario, the layout information of the live broadcast room interface includes the position information of the video screen of each connected microphone anchor, such as the coordinate information of the video screen of each connected microphone anchor (such as the coordinates of the four corners of the video screen). The audience-end device can load interface components on the live broadcast room interface according to the layout information, for example, loading the anchor avatar, name, connected microphone PK score and other components on the video screen of each connected microphone anchor according to the coordinate information of the video screen of each connected microphone anchor.

[0049] However, the supplementary enhancement information of the live stream data may be lost during the transcoding process. For example, when the viewer-end device switches the live broadcast clarity, the Blu-ray live stream data will be transcoded into high-definition live stream data or low-definition live stream data, which may cause the loss of supplementary enhancement information. Or when the viewer-end device performs other compression, cropping, scaling and other operations on the live stream data, the supplementary enhancement information may also be lost, making it impossible for the viewer-end device to perceive the changes in the live broadcast room interface layout, especially the changes in the number of connected hosts in the live broadcast room, the changes in the position and size of the video images of each connected host in the live broadcast room interface, etc., resulting in errors in the live broadcast room interface layout of the viewer-end device.

[0050] Therefore, in this embodiment, the audience-end device can detect whether the live stream data of the live broadcast room includes supplementary enhancement information, where the supplementary enhancement information includes the first layout information of the live broadcast room interface. If the live stream data includes supplementary enhancement information, the first layout information of the live broadcast room interface can be obtained from the supplementary enhancement information to perform layout control of the live broadcast room interface according to the first layout information, including loading interface components, updating interface components, etc.; if the live stream data does not include supplementary enhancement information, S201 can be executed.

[0051] It should be noted that, in this embodiment, the audience-end device can detect in real time whether the live stream data of the live broadcast room includes supplementary enhancement information; or, considering that the live stream data loses supplementary enhancement information due to transcoding and other reasons, the supplementary enhancement information will never exist in the live stream data. Therefore, as long as the audience-end device detects that the live stream data does not include the supplementary enhancement information, there is no need to detect whether the live stream data includes supplementary enhancement information subsequently; or, the audience-end device can detect whether the live stream data includes supplementary enhancement information once after entering the live broadcast room, or detect whether the live stream data includes supplementary enhancement information once after each switching of clarity.

[0052] In this embodiment, if it is detected that the live streaming data does not include the supplemental enhancement information, the live studio interface image is acquired in real time or periodically. Optionally, the live studio interface image can be acquired by taking a screenshot of the player. Of course, other methods can also be used for acquisition, which are not limited here. In addition, when periodically acquiring the live studio interface image, the acquisition period can be set according to actual needs. For example, the acquisition period can be 1 second. The shorter the acquisition period, the more sensitive the perception of the live studio interface layout changes, but it occupies more resources on the viewer's end device.

[0053] S202. Obtain second layout information of the live broadcast room interface according to preset coding information included in the live broadcast room interface image.

[0054] In this embodiment, preset coding information can be pre-encoded into each frame (or every few frames) of the live studio interface image, where the preset coding information includes the second layout information of the live studio interface, or the second layout information of the live studio interface can be obtained based on the preset coding information. The preset coding information is not visible and does not affect the user's viewing of the live broadcast.

[0055] Optionally, the preset coding information is pre-encoded into the live broadcast room interface image, and a blind watermark may be added to each frame of the live broadcast room interface image, or a blind watermark may be added every few frames. The blind watermark includes the second layout information of the live broadcast room interface, and the anchor end device adds a blind watermark to each frame of the live broadcast room interface image.

[0056] Of course, other known methods for adding invisible information to an image may also be used in this embodiment, and are not limited to the blind watermark method.

[0057] The second layout information may be the layout information of the live broadcast room interface of the current frame (that is, the same as the first layout information); considering that the preset coding information (such as a blind watermark) can carry limited information, the second layout information may also be the identification information corresponding to the layout information of the live broadcast room interface of the current frame, and the mapping relationship between the identification information and the layout information is maintained on the server side. The audience-end device may query the corresponding layout information from the server side according to the identification information. Optionally, when the host-end device determines that the layout information of the live broadcast room interface of the current frame has changed, it may update the layout information of the live broadcast room interface of the current frame and update the corresponding identification information, and then upload the updated layout information of the live broadcast room interface of the current frame and the corresponding identification information to the server side for query by the audience-end device.

[0058] S203. Perform layout control on the live broadcast room interface according to the second layout information.

[0059] In this embodiment, after obtaining the second layout information of the live broadcast room interface included in the preset coding information, the viewer-end device can control the UI layout of the live broadcast room interface according to the second layout information.

[0060] Optionally, if the layout information of the second interface changes, it means that the layout of the live broadcast room interface has changed. For example, the number of connected hosts in the live broadcast room has changed, or the position and size of the video images of each connected host in the live broadcast room have changed. In this case, the layout of the live broadcast room interface can be changed according to the changed layout information of the second interface, such as loading the host's avatar, name, PK score and other components of the new connected host, deleting the host's avatar, name, PK score and other components of the host who has exited the live broadcast room, changing the position of the host's avatar, name, PK score and other components, etc.

[0061] Among them, if the second layout information is the identification information corresponding to the layout information of the live broadcast room interface of the current frame, then when the second interface layout information changes, the audience-end device can request the layout information corresponding to the changed identification information from the server, thereby obtaining the layout information of the changed live broadcast room interface, that is, the latest layout information of the live broadcast room interface, and realizing timely perception of changes in the live broadcast room interface layout.

[0062] Optionally, if the second interface layout information has not changed, it means that the layout of the live broadcast room interface has not changed, and the audience-end device can control the live broadcast room interface layout to remain unchanged.

[0063] The live broadcast processing method provided in this embodiment is as follows: the viewer-end device detects whether the live broadcast stream data of the live broadcast room includes supplementary enhancement information, wherein the supplementary enhancement information includes the first layout information of the live broadcast room interface; if the live broadcast stream data does not include the supplementary enhancement information, the viewer-end device obtains the live broadcast room interface image; obtains the second layout information of the live broadcast room interface based on the preset coding information included in the live broadcast room interface image; and performs layout control on the live broadcast room interface based on the second layout information. In this embodiment, the second layout information of the live broadcast room interface is obtained through the preset coding information included in the live broadcast room interface image, which serves as a backup link for the supplementary enhancement information. When the supplementary enhancement information is lost, the preset coding information in the live broadcast room interface image can be parsed to obtain the second layout information of the live broadcast room interface. This ensures that the viewer-end device can stably obtain the layout information of the live broadcast room interface, promptly perceive changes in the live broadcast room interface layout, perform corresponding layout control, and ensure user experience.

[0064] Based on any of the above embodiments, the preset coding information can be pre-encoded into a preset area of ​​the live broadcast room interface image. In particular, if the preset coding information uses a blind watermark, the blind watermark is synthesized into the preset area of ​​the live broadcast room interface image. The preset area can be the background area of ​​the live broadcast room interface image, such as the background area above and / or below the live broadcast room interface image in Figure 1 (the gray area above and below). The video screen of any anchor is not displayed in the background area, but a blind watermark can be synthesized therein.

[0065] Furthermore, parsing the blind watermark included in the live studio interface image to obtain the second layout information of the live studio interface included in the blind watermark may specifically include:

[0066] The blind watermark included in the preset area of ​​the live broadcast room interface image is parsed to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0067] In this embodiment, since the blind watermark is synthesized into a preset area of ​​the live studio interface image, when parsing, only the preset area of ​​the live studio interface image needs to be parsed to obtain the second layout information included in the blind watermark. Alternatively, an image of the preset area can be intercepted from the live studio interface image and parsed to obtain the second layout information included in the blind watermark; optionally, when acquiring the live studio interface image, only the image of the preset area of ​​the live studio interface image can be acquired.

[0068] On the basis of any of the above embodiments, considering that adding a blind watermark in the image frequency domain has higher anti-attack properties and better stability, and the blind watermark content can still be parsed after compression, cropping, and scaling, in this embodiment, the blind watermark can be synthesized into the live studio interface image in the frequency domain. Therefore, parsing the blind watermark included in the live studio interface image to obtain the second layout information of the live studio interface included in the blind watermark may specifically include:

[0069] The live broadcast room interface image is transformed into the frequency domain, and the blind watermark included in the live broadcast room interface image is parsed in the frequency domain to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0070] In this embodiment, there is no limitation on the transformation method used to transform the live broadcast room interface image into the frequency domain. It can be Fourier transform, discrete cosine transform, wavelet transform, etc., and can be consistent with the transformation method used when adding blind watermarks.

[0071] Optionally, if the blind watermark is synthesized into a preset area of ​​the live broadcast room interface image, it is only necessary to transform the preset area image of the live broadcast room interface image into the frequency domain, and parse the blind watermark included in the preset area image in the frequency domain to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0072] Based on any of the above embodiments, the live broadcast processing method provided in this embodiment, as shown in FIG3 , specifically includes the following steps:

[0073] S301. When entering the live broadcast room, obtain the latest layout information of the live broadcast room interface from the server, and load interface components in the live broadcast room interface according to the latest layout information of the live broadcast room interface to perform initial layout of the live broadcast room interface;

[0074] In this embodiment, when the audience device enters the live broadcast room, it can first obtain the latest layout information of the live broadcast room interface from the server. The latest layout information of the live broadcast room interface can be included in the live broadcast room information (Room information), or it can be the latest layout information requested from the server separately.

[0075] According to the latest layout information of the live broadcast room interface, interface components are loaded in the live broadcast room interface, including but not limited to loading the anchor avatar, name, live broadcast PK score and other components of each live broadcast room in the appropriate position, thereby completing the initial layout of the live broadcast room interface.

[0076] S302: Detect whether the live streaming data of the live broadcast room includes supplementary enhancement information, wherein the supplementary enhancement information includes first layout information of the live broadcast room interface.

[0077] Please refer to the above embodiment, which will not be repeated here. If it is detected that the live streaming data includes supplementary enhancement information, S303 is executed, otherwise S304 is executed. Optionally, since the viewer end device has just entered the live broadcast room, the downloading and decoding of the live streaming data takes a certain amount of time, so it is possible to wait for a preset time (e.g., N seconds) before detecting whether the live streaming data in the live broadcast room includes supplementary enhancement information.

[0078] S303: If the live streaming data of the live broadcast room includes supplementary enhancement information, obtain first layout information of the live broadcast room interface included in the supplementary enhancement information;

[0079] Further, according to the first layout information, S306 is executed to perform layout control on the live broadcast room interface;

[0080] S304: if it is detected that the live streaming data does not include the supplementary enhancement information, then acquiring a live studio interface image in real time or periodically;

[0081] S305: Parse the blind watermark included in the live studio interface image to obtain second layout information of the live studio interface included in the blind watermark;

[0082] Further, according to the second layout information, S306 is executed to perform layout control on the live broadcast room interface;

[0083] S306: Perform layout control on the live broadcast room interface.

[0084] It should be noted that if other known methods are used to add preset coding information to the live broadcast room interface image, the process of parsing the blind watermark in the above steps will be replaced accordingly.

[0085] The live broadcast processing method provided in this embodiment has two ways for the viewer-end device to obtain the layout information of the live broadcast room interface. The first layout information of the live broadcast room interface is preferably obtained from the supplementary enhancement information. When the supplementary enhancement information is lost, the blind watermark in the live broadcast room interface image can be parsed to obtain the second layout information of the live broadcast room interface in the blind watermark. This ensures that the viewer-end device can stably obtain the layout information of the live broadcast room interface, timely perceive the changes in the layout of the live broadcast room interface, and then perform corresponding layout control, thereby ensuring user experience.

[0086] Refer to Figure 4, which is a flow chart of a live broadcast processing method provided by an embodiment of the present disclosure. The method of this embodiment can be applied to a host terminal device in a live broadcast scenario. The live broadcast processing method includes:

[0087] S401. Add supplementary enhancement information to the live streaming data of the live broadcast room, wherein the supplementary enhancement information includes first layout information of the live broadcast room interface.

[0088] In this embodiment, the host-end device can obtain the first layout information of the live broadcast room interface and add the first layout information to the live broadcast stream data as supplementary enhancement information. Optionally, since the supplementary enhancement information is synchronized with the live broadcast stream data, the first layout information of the live broadcast room interface can be the layout information of the live broadcast room interface of the current frame of the live broadcast stream data. Optionally, in the live broadcast room connected microphone scenario, the layout information of the live broadcast room interface includes the position information of the video screen of each connected microphone host, such as the coordinate information of the video screen of each connected microphone host (such as the coordinates of the four corners of the video screen).

[0089] Optionally, the host-end device can add supplementary enhancement information to each frame of the live stream data, or add supplementary enhancement information at intervals of several frames, or add supplementary enhancement information to the live stream data only when the layout of the live broadcast room interface changes and the first layout information is updated.

[0090] S402: Add preset coding information to each frame of the live broadcast room interface image of the live broadcast stream data, wherein the preset coding information includes second layout information of the live broadcast room interface.

[0091] In this embodiment, the anchor end device can add the second layout information of the live broadcast room interface to each frame of the live broadcast room interface image of the live broadcast stream data through preset coding information.

[0092] Optionally, the host device can synthesize the second layout information of the live studio interface into each frame of the live studio interface image of the live stream data through a blind watermark (or add a blind watermark every few frames). Of course, other known methods for adding invisible information to images can also be used, and are not limited in this embodiment.

[0093] The second layout information may be the layout information of the live broadcast room interface of the current frame (that is, the same as the first layout information); considering that the preset coding information (such as a blind watermark) can carry limited information, the second layout information may also be the identification information corresponding to the layout information of the live broadcast room interface of the current frame.

[0094] Optionally, in this embodiment, the second layout information of the live studio interface can be synthesized into a preset area of ​​the live studio interface image of each frame of the live stream data through a blind watermark. The preset area can be the background area of ​​the live studio interface image, such as the background area above and / or below the live studio interface image in Figure 1 (the gray area above and below). No video image of any anchor is displayed in the background area, but a blind watermark can be synthesized therein.

[0095] Specifically, you can first obtain the preset area image of the live broadcast room interface image (if the preset area image has not changed, there is no need to repeatedly obtain the preset area image), and synthesize the second layout information into the preset area image through a blind watermark (if the second layout information has not changed, there is no need to repeat the above process of synthesizing the blind watermark into the preset area image), and then synthesize the synthesized preset area image into each frame of the live broadcast room interface image.

[0096] Optionally, considering that adding a blind watermark in the image frequency domain has higher anti-attack and better stability, in this embodiment, the blind watermark can be synthesized into the live studio interface image in the frequency domain. Specifically, each frame of the live studio interface image can be transformed into the frequency domain, and the second layout information can be synthesized into each frame of the live studio interface image in the frequency domain through the blind watermark, and then each synthesized live studio interface image frame can be converted to the spatial domain. There is no limitation on the transformation method used to transform the live studio interface image into the frequency domain, and it can be Fourier transform, discrete cosine transform, wavelet transform, etc.; and the inverse transformation of the above transformation method can be used to transform each synthesized live studio interface image frame into the spatial domain.

[0097] Based on the above embodiment, if the blind watermark is synthesized into a preset area of ​​the live broadcast room interface image, the preset area image of the live broadcast room interface image can be obtained first (if the preset area image has not changed, there is no need to repeatedly obtain the preset area image); the preset area image is transformed into the frequency domain, and the second layout information is synthesized into the preset area image through the blind watermark in the frequency domain, and then the synthesized preset area image is converted to the spatial domain (if the second layout information has not changed, there is no need to repeat the above process of synthesizing the blind watermark into the preset area image); the synthesized preset area image is synthesized into each frame of the live broadcast room interface image in the spatial domain.

[0098] Taking the background area above and / or below the live broadcast room interface image as an example, the background area image can be obtained first (if the background area has not changed, there is no need to repeatedly obtain the background area image); the background area image is transformed into the frequency domain, and the second layout information is synthesized into the background area image through a blind watermark in the frequency domain, and then the synthesized background area image is converted to the spatial domain (if the second layout information has not changed, there is no need to repeat the above process of synthesizing the blind watermark into the background area image); the synthesized background area image is synthesized into each frame of the live broadcast room interface image in the spatial domain.

[0099] It should be noted that the execution order of S401 and S402 is not limited. For example, S401 and S402 may be executed simultaneously.

[0100] S403: Push the live streaming data to the server, so that the viewer device can obtain the live streaming data from the server for playback.

[0101] In this embodiment, after the host-end device completes the above process, it performs the push process of the live stream data, which will not be repeated here; the audience-end device performs the pull process of the live stream data, and after pulling the stream, the live broadcast processing method on the audience-end device side in the above embodiment can be executed.

[0102] The live broadcast processing method provided in this embodiment is as follows: the host-side device adds supplementary enhancement information to the live broadcast stream data of the live broadcast room, wherein the supplementary enhancement information includes the first layout information of the live broadcast room interface; adds preset coding information to each frame of the live broadcast room interface image of the live broadcast stream data, wherein the preset coding information includes the second layout information of the live broadcast room interface; and pushes the live broadcast stream data to the server, so that the viewer-side device can obtain the live broadcast stream data from the server for playback. The host-side device provides two ways for the viewer-side device to obtain the layout information of the live broadcast room interface, ensuring that the viewer-side device can stably obtain the layout information of the live broadcast room interface, timely perceive changes in the live broadcast room interface layout, and then perform corresponding layout control, thereby ensuring user experience.

[0103] Based on any of the above embodiments, when the layout of the live broadcast room interface of the current frame changes, for example, when the number of connected anchors in the live broadcast room changes, or the position and size of the video images of each connected anchor in the live broadcast room interface changes, the layout information of the live broadcast room interface of the current frame changes, and the anchor-end device needs to update the above-mentioned first layout information and second layout information to ensure that the supplementary enhancement information and preset encoding information (such as blind watermarks) in the live broadcast stream data also change synchronously, so that the audience-end device can perceive the layout change of the live broadcast room interface.

[0104] Among them, if the second layout information is the identification information corresponding to the layout information of the live broadcast room interface of the current frame, then when the layout information of the live broadcast room interface of the current frame changes, the second layout information is updated, that is, the identification information is updated, and then the updated second layout information and the corresponding changed layout information of the live broadcast room interface of the current frame are uploaded to the server, and the mapping relationship between the identification information and the layout information is maintained on the server, so that the audience-end device can query the corresponding layout information from the server according to the identification information after obtaining the identification information from the preset coding information (such as a blind watermark).

[0105] Corresponding to the live broadcast processing method on the viewer-side device side of the above embodiment, Figure 5 is a block diagram of the live broadcast processing device provided by the present embodiment. For ease of illustration, only the portions relevant to the present embodiment are shown. Referring to Figure 5 , the live broadcast processing device 500 includes: an image acquisition unit 501, a parsing unit 502, and a layout control unit 503.

[0106] The image acquisition unit 501 is configured to acquire a live studio interface image when the live streaming data does not include supplementary enhancement information; wherein the supplementary enhancement information includes first layout information of the live studio interface;

[0107] The parsing unit 502 is configured to obtain second layout information of the live broadcast room interface according to preset coding information included in the live broadcast room interface image;

[0108] The layout control unit 503 is used to control the layout of the live broadcast room interface according to the second layout information.

[0109] In one or more embodiments of the present disclosure, the device may further include a detection unit for detecting whether the live streaming data of the live broadcast room includes supplementary enhancement information, wherein the supplementary enhancement information includes first layout information of the live broadcast room interface.

[0110] In one or more embodiments of the present disclosure, when the layout control unit 503 performs layout control on the live broadcast room interface according to the second layout information, it is configured to:

[0111] If the second interface layout information changes, the layout of the live broadcast room interface is changed according to the changed second interface layout information.

[0112] In one or more embodiments of the present disclosure, the first layout information is layout information of the live broadcast room interface of the current frame;

[0113] The second layout information is the layout information of the live broadcast room interface of the current frame, or the second layout information is the identification information corresponding to the layout information of the live broadcast room interface of the current frame.

[0114] In one or more embodiments of the present disclosure, if the second layout information is identification information corresponding to the layout information of the live studio interface of the current frame, then if the second interface layout information changes, the layout control unit 503, when changing the layout of the live studio interface according to the changed second interface layout information, is used to:

[0115] If the identification information changes, the server is requested to provide layout information corresponding to the changed identification information; wherein the server stores a mapping relationship between different identification information and different layout information;

[0116] The layout of the live broadcast room interface is changed according to the layout information corresponding to the changed identification information.

[0117] In one or more embodiments of the present disclosure, the preset coding information included in the live broadcast room interface image is a blind watermark synthesized in the live broadcast room interface image, and the blind watermark includes the second layout information of the live broadcast room interface.

[0118] When the parsing unit 502 obtains the second layout information of the live broadcast room interface according to the preset coding information included in the live broadcast room interface image, it is used to:

[0119] The blind watermark included in the live broadcast room interface image is parsed to obtain the second layout information of the live broadcast room interface.

[0120] In one or more embodiments of the present disclosure, when parsing the blind watermark included in the live studio interface image and obtaining the second layout information of the live studio interface included in the blind watermark, the parsing unit 502 is configured to:

[0121] The live broadcast room interface image is transformed into the frequency domain, and the blind watermark included in the live broadcast room interface image is parsed in the frequency domain to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0122] In one or more embodiments of the present disclosure, when parsing the blind watermark included in the live studio interface image and obtaining the second layout information of the live studio interface included in the blind watermark, the parsing unit 502 is configured to:

[0123] The blind watermark included in the preset area of ​​the live broadcast room interface image is parsed to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0124] In one or more embodiments of the present disclosure, the layout control unit 503 is further configured to:

[0125] When entering the live broadcast room, the latest layout information of the live broadcast room interface is obtained from the server, and the interface components are loaded in the live broadcast room interface according to the latest layout information of the live broadcast room interface to perform initial layout of the live broadcast room interface.

[0126] The live broadcast processing device provided in this embodiment can be used to execute the technical solution of the method embodiment on the above-mentioned viewer-end device side. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0127] Corresponding to the live broadcast processing method on the host device side of the above embodiment, Figure 6 is a block diagram of the live broadcast processing device provided in the embodiment of the present disclosure. For ease of illustration, only the portions relevant to the embodiment of the present disclosure are shown. Referring to Figure 6, the live broadcast processing device 600 includes: a data processing unit 601, a synthesis unit 602, and a communication unit 603.

[0128] The data processing unit 601 is configured to add supplementary enhancement information to the live streaming data of the live broadcast room, wherein the supplementary enhancement information includes first layout information of the live broadcast room interface;

[0129] A synthesis unit 602 is configured to add preset coding information to each frame of the live studio interface image of the live streaming data, wherein the preset coding information includes second layout information of the live studio interface;

[0130] The communication unit 603 is used to push the live streaming data to the server, so that the viewer device can obtain the live streaming data from the server for playback.

[0131] In one or more embodiments of the present disclosure, the first layout information is layout information of the live broadcast room interface of the current frame;

[0132] The second layout information is the layout information of the live broadcast room interface of the current frame, or the second layout information is the identification information corresponding to the layout information of the live broadcast room interface of the current frame.

[0133] In one or more embodiments of the present disclosure, if the second layout information is identification information corresponding to the layout information of the live broadcast room interface of the current frame;

[0134] The data processing unit 601 is further configured to update the second layout information when the layout information of the live broadcast room interface of the current frame changes;

[0135] The communication unit 603 is further configured to upload the updated second layout information and the corresponding changed layout information of the live broadcast room interface of the current frame to the server.

[0136] In one or more embodiments of the present disclosure, when the layout information of the live broadcast room interface of the current frame changes, the data processing unit 601 updates the second layout information, and is used to:

[0137] When the number of connected hosts in the live broadcast room changes, or when the position and size of the video images of each connected host in the live broadcast room interface changes, it is determined that the layout information of the live broadcast room interface of the current frame has changed, and the second layout information is updated.

[0138] In one or more embodiments of the present disclosure, when the synthesis unit 602 adds the preset coding information to each frame of the live broadcast room interface image of the live broadcast stream data, it is used to:

[0139] The second layout information of the live broadcast room interface is synthesized into each frame of the live broadcast room interface image of the live broadcast stream data through a blind watermark.

[0140] In one or more embodiments of the present disclosure, when synthesizing the second layout information of the live studio interface into each frame of the live studio interface image of the live streaming data through a blind watermark, the synthesizing unit 602 is configured to:

[0141] The second layout information of the live broadcast room interface is synthesized into a preset area of ​​the live broadcast room interface image of each frame of the live broadcast stream data through a blind watermark.

[0142] In one or more embodiments of the present disclosure, when the synthesis unit 602 synthesizes the second layout information of the live studio interface into each frame of the live studio interface image of the live streaming data through a blind watermark, it is used to:

[0143] Each frame of the live broadcast room interface image is transformed into the frequency domain, the second layout information is synthesized into each frame of the live broadcast room interface image through a blind watermark in the frequency domain, and then each synthesized frame of the live broadcast room interface image is converted to the spatial domain.

[0144] In one or more embodiments of the present disclosure, the synthesis unit 602 is used to transform each frame of the live studio interface image into the frequency domain, synthesize the second layout information into each frame of the live studio interface image through a blind watermark in the frequency domain, and then convert each synthesized frame of the live studio interface image into the spatial domain:

[0145] Obtain a preset area image of the live broadcast room interface image, transform the preset area image into the frequency domain, synthesize the second layout information into the preset area image through a blind watermark in the frequency domain, and then convert the synthesized preset area image into the spatial domain, and synthesize the synthesized preset area image into each frame of the live broadcast room interface image in the spatial domain.

[0146] The live broadcast processing device provided in this embodiment can be used to execute the technical solution of the method embodiment on the above-mentioned anchor-end device side. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0147] Referring to FIG7 , a schematic diagram of the structure of an electronic device 700 suitable for implementing an embodiment of the present disclosure is shown. The electronic device 700 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG7 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0148] As shown in FIG7 , the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0149] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 7 shows an electronic device 700 having various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0150] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the live broadcast processing method of the above-mentioned viewer-end device or anchor-end device defined in the method of the embodiment of the present disclosure is executed.

[0151] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0152] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0153] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0154] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0156] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0157] The functions described above in the present disclosure may be performed at least in part by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0158] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0159] In a first aspect, according to one or more embodiments of the present disclosure, a live broadcast processing method is provided, including:

[0160] In the case where the live streaming data does not include the supplementary enhancement information, obtaining a live studio interface image; wherein the supplementary enhancement information includes first layout information of the live studio interface;

[0161] Acquire second layout information of the live broadcast room interface according to preset coding information included in the live broadcast room interface image;

[0162] The live broadcast room interface is layout controlled according to the second layout information.

[0163] According to one or more embodiments of the present disclosure, controlling the layout of the live broadcast room interface according to the second layout information includes:

[0164] If the second interface layout information changes, the layout of the live broadcast room interface is changed according to the changed second interface layout information.

[0165] According to one or more embodiments of the present disclosure, the first layout information is layout information of the live broadcast room interface of the current frame;

[0166] The second layout information is the layout information of the live broadcast room interface of the current frame, or the second layout information is the identification information corresponding to the layout information of the live broadcast room interface of the current frame.

[0167] According to one or more embodiments of the present disclosure, if the second layout information is identification information corresponding to the layout information of the live studio interface of the current frame, then if the second interface layout information changes, the layout of the live studio interface is changed according to the changed second interface layout information, including:

[0168] If the identification information changes, the server is requested to provide layout information corresponding to the changed identification information; wherein the server stores a mapping relationship between different identification information and different layout information;

[0169] The layout of the live broadcast room interface is changed according to the layout information corresponding to the changed identification information.

[0170] According to one or more embodiments of the present disclosure, the preset encoding information included in the live studio interface image is a blind watermark synthesized in the live studio interface image, and the blind watermark includes the second layout information of the live studio interface;

[0171] The acquiring the second layout information of the live broadcast room interface according to the preset coding information included in the live broadcast room interface image includes:

[0172] The blind watermark included in the live broadcast room interface image is parsed to obtain the second layout information of the live broadcast room interface.

[0173] According to one or more embodiments of the present disclosure, parsing the blind watermark included in the live studio interface image to obtain the second layout information of the live studio interface included in the blind watermark includes:

[0174] The live broadcast room interface image is transformed into the frequency domain, and the blind watermark included in the live broadcast room interface image is parsed in the frequency domain to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0175] According to one or more embodiments of the present disclosure, parsing the blind watermark included in the live studio interface image to obtain the second layout information of the live studio interface included in the blind watermark includes:

[0176] The blind watermark included in the preset area of ​​the live broadcast room interface image is parsed to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0177] According to one or more embodiments of the present disclosure, before detecting whether the live streaming data of the live broadcast room includes supplementary enhancement information, the method further includes:

[0178] When entering the live broadcast room, the latest layout information of the live broadcast room interface is obtained from the server, and the interface components are loaded in the live broadcast room interface according to the latest layout information of the live broadcast room interface to perform initial layout of the live broadcast room interface.

[0179] In a second aspect, according to one or more embodiments of the present disclosure, a live broadcast processing method is provided, which is applied to a host terminal device, and the method includes:

[0180] Adding supplementary enhancement information to the live streaming data of the live broadcast room, wherein the supplementary enhancement information includes first layout information of the live broadcast room interface;

[0181] Adding preset coding information to each frame of the live broadcast room interface image of the live broadcast stream data, wherein the preset coding information includes the second layout information of the live broadcast room interface;

[0182] The live streaming data is pushed to the server, so that the viewer's terminal device can obtain the live streaming data from the server for playback.

[0183] According to one or more embodiments of the present disclosure, the first layout information is layout information of the live broadcast room interface of the current frame;

[0184] The second layout information is the layout information of the live broadcast room interface of the current frame, or the second layout information is the identification information corresponding to the layout information of the live broadcast room interface of the current frame.

[0185] According to one or more embodiments of the present disclosure, if the second layout information is identification information corresponding to the layout information of the live broadcast room interface of the current frame, the method further includes:

[0186] When the layout information of the live broadcast room interface of the current frame changes, updating the second layout information;

[0187] The updated second layout information and the corresponding changed layout information of the live broadcast room interface of the current frame are uploaded to the server.

[0188] According to one or more embodiments of the present disclosure, when the layout information of the live broadcast room interface of the current frame changes, updating the second layout information includes:

[0189] When the number of connected hosts in the live broadcast room changes, or when the position and size of the video images of each connected host in the live broadcast room interface changes, it is determined that the layout information of the live broadcast room interface of the current frame has changed, and the second layout information is updated.

[0190] According to one or more embodiments of the present disclosure, adding preset coding information to each frame of the live room interface image of the live streaming data includes:

[0191] The second layout information of the live broadcast room interface is synthesized into each frame of the live broadcast room interface image of the live broadcast stream data through a blind watermark.

[0192] According to one or more embodiments of the present disclosure, synthesizing the second layout information of the live studio interface into each frame of the live studio interface image of the live streaming data through a blind watermark includes:

[0193] The second layout information of the live broadcast room interface is synthesized into a preset area of ​​the live broadcast room interface image of each frame of the live broadcast stream data through a blind watermark.

[0194] According to one or more embodiments of the present disclosure, synthesizing the second layout information of the live studio interface into each frame of the live studio interface image of the live streaming data through a blind watermark includes:

[0195] Each frame of the live broadcast room interface image is transformed into the frequency domain, the second layout information is synthesized into each frame of the live broadcast room interface image through a blind watermark in the frequency domain, and then each synthesized frame of the live broadcast room interface image is converted to the spatial domain.

[0196] According to one or more embodiments of the present disclosure, the method of transforming each frame of the live studio interface image into the frequency domain, synthesizing the second layout information into each frame of the live studio interface image through a blind watermark in the frequency domain, and then converting each synthesized frame of the live studio interface image into the spatial domain includes:

[0197] Obtain a preset area image of the live broadcast room interface image, transform the preset area image into the frequency domain, synthesize the second layout information into the preset area image through a blind watermark in the frequency domain, and then convert the synthesized preset area image into the spatial domain, and synthesize the synthesized preset area image into each frame of the live broadcast room interface image in the spatial domain.

[0198] In a third aspect, according to one or more embodiments of the present disclosure, a live broadcast processing device is provided, including:

[0199] An image acquisition unit is configured to acquire a live studio interface image when the live streaming data does not include supplementary enhancement information; wherein the supplementary enhancement information includes first layout information of the live studio interface;

[0200] A parsing unit, configured to obtain second layout information of the live broadcast room interface according to preset coding information included in the live broadcast room interface image;

[0201] A layout control unit is used to control the layout of the live broadcast room interface according to the second layout information.

[0202] According to one or more embodiments of the present disclosure, when the layout control unit performs layout control on the live broadcast room interface according to the second layout information, it is configured to:

[0203] If the second interface layout information changes, the layout of the live broadcast room interface is changed according to the changed second interface layout information.

[0204] According to one or more embodiments of the present disclosure, the first layout information is layout information of the live broadcast room interface of the current frame;

[0205] The second layout information is the layout information of the live broadcast room interface of the current frame, or the second layout information is the identification information corresponding to the layout information of the live broadcast room interface of the current frame.

[0206] According to one or more embodiments of the present disclosure, if the second layout information is identification information corresponding to the layout information of the live studio interface of the current frame, then if the second interface layout information changes, the layout control unit is used to change the layout of the live studio interface according to the changed second interface layout information:

[0207] If the identification information changes, the server is requested to provide layout information corresponding to the changed identification information; wherein the server stores a mapping relationship between different identification information and different layout information;

[0208] The layout of the live broadcast room interface is changed according to the layout information corresponding to the changed identification information.

[0209] According to one or more embodiments of the present disclosure, the preset coding information included in the live broadcast room interface image is a blind watermark synthesized in the live broadcast room interface image, and the blind watermark includes the second layout information of the live broadcast room interface.

[0210] The acquiring the second layout information of the live broadcast room interface according to the preset coding information included in the live broadcast room interface image includes:

[0211] The blind watermark included in the live broadcast room interface image is parsed to obtain the second layout information of the live broadcast room interface.

[0212] According to one or more embodiments of the present disclosure, when the parsing unit parses the blind watermark included in the live studio interface image and obtains the second layout information of the live studio interface included in the blind watermark, it is configured to:

[0213] The live broadcast room interface image is transformed into the frequency domain, and the blind watermark included in the live broadcast room interface image is parsed in the frequency domain to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0214] According to one or more embodiments of the present disclosure, when the parsing unit parses the blind watermark included in the live studio interface image and obtains the second layout information of the live studio interface included in the blind watermark, it is configured to:

[0215] The blind watermark included in the preset area of ​​the live broadcast room interface image is parsed to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0216] According to one or more embodiments of the present disclosure, the layout control unit is further configured to:

[0217] When entering the live broadcast room, the latest layout information of the live broadcast room interface is obtained from the server, and the interface components are loaded in the live broadcast room interface according to the latest layout information of the live broadcast room interface to perform initial layout of the live broadcast room interface.

[0218] In a fourth aspect, according to one or more embodiments of the present disclosure, a live broadcast processing device is provided, including:

[0219] A data processing unit, configured to add supplementary enhancement information to the live streaming data of the live broadcast room, wherein the supplementary enhancement information includes first layout information of the live broadcast room interface;

[0220] A synthesis unit, configured to add preset coding information to each frame of the live studio interface image of the live streaming data, wherein the preset coding information includes second layout information of the live studio interface;

[0221] The communication unit is used to push the live streaming data to the server, so that the viewer device can obtain the live streaming data from the server for playback.

[0222] According to one or more embodiments of the present disclosure, the first layout information is layout information of the live broadcast room interface of the current frame;

[0223] The second layout information is the layout information of the live broadcast room interface of the current frame, or the second layout information is the identification information corresponding to the layout information of the live broadcast room interface of the current frame.

[0224] According to one or more embodiments of the present disclosure, if the second layout information is identification information corresponding to the layout information of the live broadcast room interface of the current frame;

[0225] The data processing unit is further configured to update the second layout information when the layout information of the live broadcast room interface of the current frame changes;

[0226] The communication unit is also used to upload the updated second layout information and the corresponding changed layout information of the live broadcast room interface of the current frame to the server.

[0227] According to one or more embodiments of the present disclosure, when the layout information of the live broadcast room interface of the current frame changes, the data processing unit updates the second layout information, and is used to:

[0228] When the number of connected hosts in the live broadcast room changes, or when the position and size of the video images of each connected host in the live broadcast room interface changes, it is determined that the layout information of the live broadcast room interface of the current frame has changed, and the second layout information is updated.

[0229] According to one or more embodiments of the present disclosure, when the synthesis unit adds preset coding information to each frame of the live broadcast room interface image of the live broadcast stream data, it is used to:

[0230] The second layout information of the live broadcast room interface is synthesized into each frame of the live broadcast room interface image of the live broadcast stream data through a blind watermark.

[0231] According to one or more embodiments of the present disclosure, when the synthesis unit synthesizes the second layout information of the live studio interface into each frame of the live studio interface image of the live streaming data through a blind watermark, it is used to:

[0232] The second layout information of the live broadcast room interface is synthesized into a preset area of ​​the live broadcast room interface image of each frame of the live broadcast stream data through a blind watermark.

[0233] According to one or more embodiments of the present disclosure, when the synthesis unit synthesizes the second layout information of the live studio interface into each frame of the live studio interface image of the live streaming data through a blind watermark, it is used to:

[0234] Each frame of the live broadcast room interface image is transformed into the frequency domain, the second layout information is synthesized into each frame of the live broadcast room interface image through a blind watermark in the frequency domain, and then each synthesized frame of the live broadcast room interface image is converted to the spatial domain.

[0235] According to one or more embodiments of the present disclosure, when the synthesis unit transforms each frame of the live studio interface image into the frequency domain, synthesizes the second layout information into each frame of the live studio interface image through a blind watermark in the frequency domain, and then converts each synthesized frame of the live studio interface image into the spatial domain, it is used to:

[0236] Obtain a preset area image of the live broadcast room interface image, transform the preset area image into the frequency domain, synthesize the second layout information into the preset area image through a blind watermark in the frequency domain, and then convert the synthesized preset area image into the spatial domain, and synthesize the synthesized preset area image into each frame of the live broadcast room interface image in the spatial domain.

[0237] In a fifth aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0238] The memory stores computer-executable instructions;

[0239] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the live broadcast processing method described in the first aspect and various possible designs of the first aspect.

[0240] In the sixth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer execution instructions are stored. When the processor executes the computer execution instructions, the live broadcast processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0241] In the seventh aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer execution instructions. When a processor executes the computer execution instructions, the live broadcast processing method as described in the first aspect and various possible designs of the first aspect is implemented.

[0242] The above description is merely an illustration of the technical principles employed in the embodiments of the present disclosure. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by mutually replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0243] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0244] Although the present disclosure has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A live broadcast processing method, applied to a viewer terminal device, the method comprising: When the live streaming data does not include the supplementary enhancement information, obtaining the live streaming room interface image; The supplementary enhancement information includes first layout information of the live broadcast room interface; Acquire second layout information of the live studio interface according to preset coding information included in the live studio interface image; The live broadcast room interface is layout controlled according to the second layout information.

2. The method according to claim 1, wherein: The performing layout control on the live broadcast room interface according to the second layout information includes: If the second interface layout information changes, the layout of the live broadcast room interface is changed according to the changed second interface layout information.

3. The method according to claim 2, wherein: The first layout information is the layout information of the live broadcast room interface of the current frame; The second layout information is the layout information of the live broadcast room interface of the current frame, or the second layout information is the identification information corresponding to the layout information of the live broadcast room interface of the current frame.

4. The method according to claim 3, wherein: If the second layout information is identification information corresponding to the layout information of the live room interface of the current frame, then if the second interface layout information changes, the layout of the live room interface is changed according to the changed second interface layout information, including: If the identification information changes, the layout information corresponding to the changed identification information is requested from the server; wherein the mapping relationship between different identification information and different layout information is stored in the server; The layout of the live broadcast room interface is changed according to the layout information corresponding to the changed identification information.

5. The method according to any one of claims 1 to 4, wherein: The preset coding information included in the live studio interface image is a blind watermark synthesized in the live studio interface image, and the blind watermark includes the second layout information of the live studio interface; The acquiring the second layout information of the live broadcast room interface according to the preset coding information included in the live broadcast room interface image includes: The blind watermark included in the live broadcast room interface image is parsed to obtain the second layout information of the live broadcast room interface.

6. The method according to claim 5, wherein: The step of parsing the blind watermark included in the live studio interface image to obtain second layout information of the live studio interface includes: The live broadcast room interface image is transformed into the frequency domain, and the blind watermark included in the live broadcast room interface image is parsed in the frequency domain to obtain the second layout information of the live broadcast room interface.

7. The method according to claim 5 or 6, wherein: The step of parsing the blind watermark included in the live studio interface image to obtain second layout information of the live studio interface includes: The blind watermark included in the preset area of ​​the live broadcast room interface image is parsed to obtain the second layout information of the live broadcast room interface.

8. A live broadcast processing method, applied to a host terminal device, the method comprising: Adding supplementary enhancement information to the live streaming data of the live broadcast room, wherein the supplementary enhancement information includes first layout information of the live broadcast room interface; Adding preset coding information to each frame of the live studio interface image of the live streaming data, wherein the preset coding information includes second layout information of the live studio interface; The live streaming data is pushed to the server, so that the audience device can obtain the live streaming data from the server for playback.

9. The method according to claim 8, wherein: The first layout information is the layout information of the live broadcast room interface of the current frame; The second layout information is the layout information of the live broadcast room interface of the current frame, or the second layout information is the identification information corresponding to the layout information of the live broadcast room interface of the current frame.

10. The method according to claim 9, wherein: If the second layout information is identification information corresponding to the layout information of the live broadcast room interface of the current frame, the method further includes: When the layout information of the live broadcast room interface of the current frame changes, updating the second layout information; The updated second layout information and the corresponding changed layout information of the live broadcast room interface of the current frame are uploaded to the server.

11. The method according to claim 10, wherein: When the layout information of the live broadcast room interface of the current frame changes, updating the second layout information includes: When the number of connected hosts in the live broadcast room changes, or when the position and size of the video images of each connected host in the live broadcast room change in the live broadcast room interface, it is determined that the layout information of the live broadcast room interface of the current frame has changed, and the second layout information is updated.

12. The method according to any one of claims 8 to 11, wherein: The adding of preset coding information in each frame of the live room interface image of the live streaming data comprises: The second layout information of the live broadcast room interface is synthesized into each frame of the live broadcast room interface image of the live broadcast stream data through a blind watermark.

13. The method according to claim 12, wherein: The step of synthesizing the second layout information of the live studio interface into each frame of the live studio interface image of the live streaming data through a blind watermark includes: The second layout information of the live broadcast room interface is synthesized into a preset area of ​​each frame of the live broadcast room interface image of the live broadcast stream data through a blind watermark.

14. The method according to claim 12 or 13, wherein: The step of synthesizing the second layout information of the live studio interface into each frame of the live studio interface image of the live streaming data through a blind watermark includes: Each frame of the live studio interface image is transformed into the frequency domain, the second layout information is synthesized into each frame of the live studio interface image through a blind watermark in the frequency domain, and then each synthesized frame of the live studio interface image is converted to the spatial domain.

15. The method according to claim 14, wherein: The method of transforming each frame of the live studio interface image into the frequency domain, synthesizing the second layout information into each frame of the live studio interface image through a blind watermark in the frequency domain, and then converting each synthesized frame of the live studio interface image into the spatial domain includes: Obtain a preset area image of the live broadcast room interface image, transform the preset area image into the frequency domain, synthesize the second layout information into the preset area image through a blind watermark in the frequency domain, and then convert the synthesized preset area image to the spatial domain, and synthesize the synthesized preset area image into each frame of the live broadcast room interface image in the spatial domain.

16. A live broadcast processing device, comprising: An image acquisition unit, configured to acquire a live broadcast room interface image when the live broadcast stream data does not include the supplementary enhancement information; The supplementary enhancement information includes first layout information of the live broadcast room interface; A parsing unit, configured to obtain second layout information of the live broadcast room interface according to preset coding information included in the live broadcast room interface image; The layout control unit is configured to perform layout control on the live broadcast room interface according to the second layout information.

17. A live broadcast processing device, comprising: A data processing unit is configured to add supplementary enhancement information to the live streaming data of the live broadcast room, wherein the supplementary enhancement information includes first layout information of the live broadcast room interface; A synthesis unit is configured to add preset coding information to each frame of the live studio interface image of the live streaming data, wherein the preset coding information includes second layout information of the live studio interface; The communication unit is configured to push the live streaming data to the server, so that the viewer device can obtain the live streaming data from the server for playback.

18. An electronic device, comprising: at least one processor and memory, wherein The memory is configured to store computer-executable instructions; The at least one processor is configured to execute the computer-executable instructions stored in the memory, so that the at least one processor executes the live broadcast processing method as described in any one of claims 1-15.

19. A computer-readable storage medium storing computer-executable instructions, wherein: When the processor executes the computer-executable instruction, the live broadcast processing method as described in any one of claims 1 to 15 is implemented.

20. A computer program product, wherein: It includes computer-executable instructions. When a processor executes the computer-executable instructions, it implements the live broadcast processing method as described in any one of claims 1 to 15.

Images