Live broadcast processing method and device, and storage medium

Abstract

The embodiment of the invention provides a live broadcast processing method and device and a storage medium, an anchor end device adds supplemental enhancement information in live broadcast stream data, the supplemental enhancement information comprises first layout information of a live broadcast room interface, and meanwhile, preset coding information is added in a live broadcast room interface image of the live broadcast stream data; the preset coding information can comprise second layout information of the live broadcast room interface, so that the audience end equipment has two ways to obtain the layout information of the live broadcast room interface, and when the supplementary enhancement information is lost, the second layout information of the live broadcast room interface is obtained according to the preset coding information in the live broadcast room interface image; therefore, the audience end equipment can stably obtain the layout information of the live broadcast room interface, timely sense the change of the layout of the live broadcast room interface, and perform corresponding layout control, so that the user experience is ensured.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer and network communication technologies, and in particular, to a live broadcast processing method, device, and storage medium. Background Art

[0002] In the scenario of network live broadcast, the layout of the live broadcast room interface may change. For example, in the scenario of live connection of microphones, the host and the guest connected by microphone in the live broadcast room can have real-time audio and video calls, and the content of the audio and video calls can be pushed to the client device after being merged with multiple streams. However, as the number of hosts and guests connected by microphone changes, the layout of the live broadcast room interface will also change, and the content of the picture on the client device will also change accordingly.

[0003] Since the client device needs to load some interface components according to the layout of the live broadcast room interface, it is necessary to sense the change of the layout of the live broadcast room interface. In the prior art, it is usually adopted to carry Supplemental Enhancement Information (SEI) in the live stream data to transmit the change of the layout of the live broadcast room interface.

[0004] However, the Supplemental Enhancement Information carried in the live stream data is easily lost, resulting in the client device being unable to sense the change of the layout of the live broadcast room interface and affecting the user experience. Summary of the Invention

[0005] Embodiments of the present disclosure provide a live broadcast processing method, device, and storage medium to ensure that the client device can stably obtain the layout information of the live broadcast room interface and timely sense the change of the layout of the live broadcast room interface.

[0006] In a first aspect, embodiments of the present disclosure provide a live broadcast processing method applied to a client device, and the method includes:

[0007] When the live stream data does not include Supplemental Enhancement Information, obtain an image of the live broadcast room interface; wherein the Supplemental Enhancement Information includes first layout information of the live broadcast room interface;

[0008] Obtain second layout information of the live broadcast room interface according to preset coding information included in the image of the live broadcast room interface;

[0009] Perform layout control on the live broadcast room interface according to the second layout information.

[0010] In a second aspect, embodiments of the present disclosure provide a live broadcast processing method applied to a host device, and the method includes:

[0011] Add Supplemental Enhancement Information to the live stream data of the live broadcast room, wherein the Supplemental Enhancement Information includes first layout information of the live broadcast room interface;

[0012] Add preset encoding information to each frame of the live stream data in the live broadcast room interface image, where the preset encoding information includes the second layout information of the live broadcast room interface;

[0013] Push the live stream data to the server for the viewer device to obtain the live stream data from the server for playback.

[0014] In a third aspect, an embodiment of the present disclosure provides a live broadcast processing device, including:

[0015] An image acquisition unit, configured to acquire the live broadcast room interface image when the live stream data does not include supplementary enhancement information; where the supplementary enhancement information includes the first layout information of the live broadcast room interface;

[0016] An analysis unit, configured to obtain the second layout information of the live broadcast room interface according to the preset encoding information included in the live broadcast room interface image;

[0017] A layout control unit, configured to perform layout control on the live broadcast room interface according to the second layout information.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a live broadcast processing device, including:

[0019] A data processing unit, configured to add supplementary enhancement information to the live stream data in the live broadcast room, where the supplementary enhancement information includes the first layout information of the live broadcast room interface;

[0020] A synthesis unit, configured to add preset encoding information to each frame of the live stream data in the live broadcast room interface image, where the preset encoding information includes the second layout information of the live broadcast room interface;

[0021] A communication unit, configured to push the live stream data to the server for the viewer device to obtain the live stream data from the server for playback.

[0022] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0023] The memory stores computer-executable instructions;

[0024] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the live broadcast processing method as described in the first aspect and various possible designs of the first aspect above.

[0025] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the live broadcast processing method as described in the first aspect above and all possible designs of the first aspect.

[0026] In a seventh aspect, an embodiment of the present disclosure provides a computer program product including computer-executable instructions, which, when executed by a processor, implement the live broadcast processing method as described in the first aspect above and all possible designs of the first aspect.

[0027] In the live broadcast processing method, device, and storage medium provided by the embodiments of the present disclosure, the host device adds supplementary enhancement information to the live stream data. The supplementary enhancement information includes the first layout information of the live broadcast room interface. At the same time, preset coding information is added to the live broadcast room interface image in the live stream data. The preset coding information may include the second layout information of the live broadcast room interface. In this way, there are two ways for the viewer device to obtain the layout information of the live broadcast room interface. When the supplementary enhancement information is lost, the second layout information of the live broadcast room interface is obtained according to the preset coding information in the live broadcast room interface image, ensuring that the viewer device can stably obtain the layout information of the live broadcast room interface, promptly perceive the change in the layout of the live broadcast room interface, perform corresponding layout control, and ensuring the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] To more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art. Obviously, the following-described drawings are some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0029] Figure 1 It is an example diagram of the layout of the live broadcast room interface in the prior art;

[0030] Figure 2 It is a schematic flowchart of the live broadcast processing method provided by an embodiment of the present disclosure;

[0031] Figure 3 It is a schematic flowchart of the live broadcast processing method provided by another embodiment of the present disclosure;

[0032] Figure 4 It is a schematic flowchart of the live broadcast processing method provided by another embodiment of the present disclosure;

[0033] Figure 5 It is a block diagram of the structure of the live broadcast processing device provided by an embodiment of the present disclosure;

[0034] Figure 6Block diagram of a live processing device provided in another embodiment of the present disclosure;

[0035] Figure 7 Schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. Detailed implementation manners

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are some but not all of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts shall fall within the protection scope of the present disclosure.

[0037] The blind watermarking function refers to adding a watermark to a picture in an invisible form. Such a watermark does not affect the visual perception of the picture and at the same time ensures the originality of the picture. By extracting the blind watermark, the invisible watermark is extracted to verify the ownership of the picture. One method of adding a blind watermark is the method of adding a digital watermark in the frequency domain, which means that the image is transformed to the frequency domain (wavelet domain) through a certain transformation means (Fourier transform, discrete cosine transform, wavelet transform, etc.), the watermark is added to the image in the frequency domain, and then through the inverse transformation, the image is converted back to the spatial domain. The blind watermark in the frequency domain has stronger invisibility and higher anti-attack ability. The watermark content can still be parsed even after the picture has been compressed, cropped, or scaled.

[0038] Supplemental Enhancement Information (SEI) is text data inserted into an audio-video stream to convey additional information. It can be accurately synchronized with the relevant audio-video content and reach the client through the transmission link.

[0039] In the scenario of network live broadcast, the layout of the live broadcast room interface may change. For example, in the scenario of live connection of microphones, the host and the guest connected by microphone in the live broadcast room can have real-time audio-video calls, and the content of the audio-video call can be pushed to the viewer device after the multi-channel streams are merged. However, as the number of hosts and guests connected by microphone changes, the layout of the live broadcast room interface will also change accordingly, and the picture content of the viewer device will also change accordingly. As Figure 1 shown, different layouts of the live broadcast room interface can be presented as the number of hosts and guests connected by microphone changes.

[0040] Since the viewer-side device needs to load some interface components according to the live broadcast room interface layout, such as the anchor avatar, name, and PK score of each connected anchor in the broadcast room, it is necessary to perceive the changes in the live broadcast room interface layout. In the prior art, the SEI is usually carried in the live broadcast stream data to transmit the changes in the live broadcast room interface layout to the viewer-side device. SEI is aligned with the video frame, and using SEI to perceive the changes in the video picture is the most timely and accurate.

[0041] SEI is supplementary enhancement information in H.264 and H.265 encoding formats. It is easy to be lost in the transmission link of streaming media, especially in the transcoding process. As a data transmission link, it has poor stability, and there will be problems such as complete loss of SEI and SEI truncation causing unresolvable problems. Currently, the live broadcast and microphone connection business strongly relies on SEI to transmit information and status. The stability of the SEI link is poor, resulting in the inability of the audience's end device to perceive the changes in the layout of the live broadcast room interface, affecting the user experience.

[0042] In order to solve the above technical problems, the present disclosure provides a live broadcast processing method, in which the anchor-end device adds supplementary enhancement information to the live broadcast stream data, the supplementary enhancement information includes the first layout information of the live broadcast room interface, and at the same time adds preset coding information to the live broadcast room interface image of the live broadcast stream data, the preset coding information may include the second layout information of the live broadcast room interface, so that the audience-end device has two ways to obtain the layout information of the live broadcast room interface, and it is preferred to obtain the layout information of the live broadcast room interface from the supplementary enhancement information. When the supplementary enhancement information is lost, the layout information of the live broadcast room interface can be obtained through the preset coding information in the live broadcast room interface image. In this way, it is ensured that the audience-end device can stably obtain the layout information of the live broadcast room interface, and timely perceive the changes in the layout of the live broadcast room interface, and then perform corresponding layout control, thereby ensuring user experience.

[0043] The live broadcast processing method disclosed in the present invention is applicable to a live broadcast scenario, which includes an anchor-end device, a server-end device and an audience-end device. The anchor-end device pushes the live broadcast stream data to the server-end, and the audience-end device pulls and plays the live broadcast stream data from the server-end. The specific process will not be repeated here.

[0044] It should be noted that the user information and data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0045] The live broadcast processing method disclosed in the present invention will be described in detail below in conjunction with specific embodiments.

[0046] refer to Figure 2 ,Figure 2 The flowchart of the live broadcast processing method provided by an embodiment of the present disclosure. The method of this embodiment can be applied to the viewer device in the live broadcast scenario. The live broadcast processing method includes:

[0047] S201. When the supplementary enhancement information is not included in the live stream data, obtain the live broadcast room interface image; wherein the supplementary enhancement information includes the first layout information of the live broadcast room interface.

[0048] In this embodiment, the live stream data of the live broadcast room will carry supplementary enhancement information, and the supplementary enhancement information includes the first layout information of the live broadcast room interface. Since the supplementary enhancement information is synchronized with the live stream data, the first layout information of the live broadcast room interface can be the layout information of the live broadcast room interface of the current frame of the live stream data. In the co-hosting scenario of the live broadcast room, the layout information of the live broadcast room interface includes the position information of the video images of each co-host, such as the coordinate information of the video images of each co-host (such as the coordinates of the four corners of the video image). The viewer device can load interface components according to the layout information in the live broadcast room interface, for example, load host avatars, names, co-hosting PK scores and other components on the video images of each co-host according to the coordinate information of the video images of each co-host.

[0049] However, the supplementary enhancement information may be lost during the transcoding process of the live stream data. For example, when the viewer device switches the live broadcast clarity, it will transcode the high-definition live stream data into high-definition or low-definition live stream data, which may cause the supplementary enhancement information to be lost. Or when the viewer device performs other operations such as compression, cropping, and scaling on the live stream data, it may also cause the supplementary enhancement information to be lost, resulting in the viewer device being unable to perceive the layout change of the live broadcast room interface, especially when the number of co-hosts in the live broadcast room changes, or the position and size of the video images of each co-host in the live broadcast room interface change, resulting in an incorrect layout of the live broadcast room interface on the viewer device.

[0050] Therefore, in this embodiment, the viewer device can detect whether the supplementary enhancement information is included in the live stream data of the live broadcast room, and the supplementary enhancement information includes the first layout information of the live broadcast room interface. If the supplementary enhancement information is included in the live stream data, the first layout information of the live broadcast room interface can be obtained from the supplementary enhancement information to perform layout control on the live broadcast room interface, including loading interface components, updating interface components, etc.; if the supplementary enhancement information is not included in the live stream data, S201 can be executed.

[0051] It should be noted that in this embodiment, the viewer device can detect in real time whether the live stream data in the live broadcast room includes supplementary enhancement information; or, considering that the supplementary enhancement information is lost in the live stream data due to transcoding or other reasons, the supplementary enhancement information will always be absent in the live stream data. Therefore, as long as the viewer device detects that the live stream data does not include the supplementary enhancement information, there is no need to detect whether the live stream data includes the supplementary enhancement information subsequently; or, the viewer device can detect once whether the live stream data includes the supplementary enhancement information after entering the live broadcast room, or detect once whether the live stream data includes the supplementary enhancement information after each switching of the video quality.

[0052] In this embodiment, when it is detected that the live stream data does not include the supplementary enhancement information, the live broadcast room interface image is obtained in real time or periodically. Optionally, the live broadcast room interface image can be obtained by taking a screenshot of the player. Of course, it can also be obtained by other means, which is not limited here. In addition, when obtaining the live broadcast room interface image periodically, the acquisition period can be set according to actual needs. For example, the acquisition period can be 1 second. The shorter the acquisition period, the more sensitive the perception of the layout change of the live broadcast room interface, but it occupies more resources of the viewer device.

[0053] S202. Obtain the second layout information of the live broadcast room interface according to the preset coding information included in the live broadcast room interface image.

[0054] In this embodiment, the preset coding information can be pre-encoded into each frame (or every few frames) of the live broadcast room interface image. The preset coding information includes the second layout information of the live broadcast room interface, or the second layout information of the live broadcast room interface can be obtained according to the preset coding information. And the preset coding information is invisible and does not affect the user's viewing of the live broadcast.

[0055] Optionally, pre-encoding the preset coding information into the live broadcast room interface image can be adding a blind watermark to each frame of the live broadcast room interface image, or adding a blind watermark every few frames. The second layout information of the live broadcast room interface included in the blind watermark is added to each frame of the live broadcast room interface image by the host device.

[0056] Of course, in this embodiment, other known methods of adding invisible information to the image can also be used, not limited to the blind watermark method.

[0057] The second layout information may be the layout information of the live streaming room interface of the current frame (i.e., the same as the first layout information); considering that the information that the preset coding information (such as a blind watermark) can carry is limited, the second layout information may also be the identification information corresponding to the layout information of the live streaming room interface of the current frame, and the server maintains the mapping relationship between the identification information and the layout information. The viewer device can query the corresponding layout information from the server according to the identification information. Optionally, when the host device determines that the layout information of the live streaming room interface of the current frame has changed, it can update the layout information of the live streaming room interface of the current frame and the corresponding identification information, and then upload the updated layout information of the live streaming room interface of the current frame and the corresponding identification information to the server for the viewer device to query.

[0058] S203. Perform layout control on the live streaming room interface according to the second layout information.

[0059] In this embodiment, after the viewer device obtains the second layout information of the live streaming room interface included in the preset coding information, it can perform UI layout control on the live streaming room interface according to the second layout information.

[0060] Optionally, if the second interface layout information changes, it means that the layout of the live streaming room interface has changed. For example, the number of co-hosts in the live streaming room changes, or the positions and sizes of the video images of the co-hosts in the live streaming room interface change. Then, the layout of the live streaming room interface can be changed according to the changed second interface layout information. For example, components such as the host avatar, name, and co-host PK score of the newly joined co-host are loaded, components such as the host avatar, name, and co-host PK score of the co-host who exits the connection are deleted, and the positions of components such as the host avatar, name, and co-host PK score are changed.

[0061] Among them, if the second layout information is the identification information corresponding to the layout information of the live streaming room interface of the current frame, when the second interface layout information changes, the viewer device can request the layout information corresponding to the changed identification information from the server, so as to obtain the changed layout information of the live streaming room interface, that is, the latest layout information of the live streaming room interface, and realize timely perception of the change in the layout of the live streaming room interface.

[0062] Optionally, if the second interface layout information does not change, it means that the layout of the live streaming room interface has not changed, and the viewer device can control the layout of the live streaming room interface to remain unchanged.

[0063] The live processing method provided in this embodiment is such that the viewer device detects whether the live stream data in the live broadcast room includes supplementary enhancement information, where the supplementary enhancement information includes the first layout information of the live broadcast room interface; in the case where the live stream data does not include the supplementary enhancement information, the live broadcast room interface image is obtained; the second layout information of the live broadcast room interface is obtained according to the preset coding information included in the live broadcast room interface image; and the layout of the live broadcast room interface is controlled according to the second layout information. In this embodiment, the second layout information of the live broadcast room interface is obtained through the preset coding information included in the live broadcast room interface image as a backup link for the supplementary enhancement information. When the supplementary enhancement information is lost, the preset coding information in the live broadcast room interface image can be parsed to obtain the second layout information of the live broadcast room interface, ensuring that the viewer device can stably obtain the layout information of the live broadcast room interface, promptly perceive the change in the layout of the live broadcast room interface, perform corresponding layout control, and ensure the user experience.

[0064] Based on any of the above embodiments, the preset coding information can be pre-coded into a preset area of the live broadcast room interface image. In particular, if the preset coding information uses a blind watermark, the blind watermark is synthesized into a preset area of the live broadcast room interface image, and the preset area can be the background area of the live broadcast room interface image. For example Figure 1 in the background area above and / or below the live broadcast room interface image (the gray areas above and below), no video images of any anchor are displayed in the background area, but the blind watermark can be synthesized therein.

[0065] Further, the blind watermark included in the live broadcast room interface image is parsed to obtain the second layout information of the live broadcast room interface included in the blind watermark. Specifically, it may include:

[0066] The blind watermark included in the preset area of the live broadcast room interface image is parsed to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0067] In this embodiment, since the blind watermark is synthesized into the preset area of the live broadcast room interface image, during parsing, only the preset area of the live broadcast room interface image needs to be parsed to obtain the second layout information included in the blind watermark. Optionally, the preset area image can be intercepted from the live broadcast room interface image, and the preset area image can be parsed to obtain the second layout information included in the blind watermark; optionally, when obtaining the live broadcast room interface image, only the preset area image in the live broadcast room interface image can be obtained.

[0068] Based on any of the above embodiments, considering that adding a blind watermark in the frequency domain of an image has higher anti-attack performance and better stability, and the content of the blind watermark can still be parsed after compression, cropping, and scaling, in this embodiment, the blind watermark can be synthesized into the live broadcast room interface image in the frequency domain. Therefore, parsing the blind watermark included in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface included in the blind watermark may specifically include:

[0069] Transform the live broadcast room interface image into the frequency domain, and parse the blind watermark included in the live broadcast room interface image in the frequency domain to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0070] In this embodiment, there is no limitation on the transformation means used to transform the live broadcast room interface image into the frequency domain. It can be Fourier transform, discrete cosine transform, wavelet transform, etc., and can be consistent with the transformation means used when adding the blind watermark.

[0071] Optionally, if the blind watermark is synthesized into a preset area of the live broadcast room interface image, only the image of the preset area of the live broadcast room interface needs to be transformed into the frequency domain, and the blind watermark included in the preset area image is parsed in the frequency domain to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0072] Based on any of the above embodiments, the live broadcast processing method provided in this embodiment, as Figure 3 shown, specifically includes the following steps:

[0073] S301. When entering the live broadcast room, obtain the latest layout information of the live broadcast room interface from the server, and load interface components in the live broadcast room interface according to the latest layout information of the live broadcast room interface to perform an initial layout of the live broadcast room interface;

[0074] In this embodiment, when the viewer's device enters the live broadcast room, it can first obtain the latest layout information of the live broadcast room interface from the server. The latest layout information of the live broadcast room interface can be included in the live broadcast room information (Room information), or can be the latest layout information requested separately from the server.

[0075] Loading interface components in the live broadcast room interface according to the latest layout information of the live broadcast room interface includes, but is not limited to, loading components such as the host avatars, names, and co-hosting PK scores of each co-hosting host in the live broadcast room at appropriate positions, so as to complete the initial layout of the live broadcast room interface.

[0076] S302. Detect whether the live stream data of the live broadcast room includes supplementary enhancement information, where the supplementary enhancement information includes the first layout information of the live broadcast room interface.

[0077] Refer to the above embodiments, which will not be elaborated here. If it is detected that the live stream data includes supplementary enhancement information, then execute S303; otherwise, execute S304. Optionally, since the viewer device has just entered the live broadcast room and it takes a certain amount of time for processes such as downloading and decoding the live stream data, it is possible to wait for a preset duration (e.g., N seconds) and then detect whether the live stream data in the live broadcast room includes supplementary enhancement information.

[0078] S303. If it is detected that the live stream data in the live broadcast room includes supplementary enhancement information, then obtain the first layout information of the live broadcast room interface included in the supplementary enhancement information.

[0079] Further, according to the first layout information, execute S306 to perform layout control on the live broadcast room interface.

[0080] S304. If it is detected that the live stream data does not include the supplementary enhancement information, then obtain the live broadcast room interface image in real time or periodically.

[0081] S305. Analyze the blind watermark included in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0082] Further, according to the second layout information, execute S306 to perform layout control on the live broadcast room interface.

[0083] S306. Perform layout control on the live broadcast room interface.

[0084] It should be noted that if other known methods are used to add preset coding information to the live broadcast room interface image, then the process of parsing the blind watermark in the above steps is replaced accordingly.

[0085] For the live processing method provided in this embodiment, there are two ways for the viewer device to obtain the layout information of the live broadcast room interface. It preferentially selects to obtain the first layout information of the live broadcast room interface from the supplementary enhancement information. When the supplementary enhancement information is lost, it can analyze the blind watermark in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface in the blind watermark. This ensures that the viewer device can stably obtain the layout information of the live broadcast room interface, promptly perceive the changes in the layout of the live broadcast room interface, and then perform corresponding layout control, thus ensuring the user experience.

[0086] Reference Figure 4 , Figure 4 is a schematic flowchart of the live processing method provided by an embodiment of the present disclosure. The method of this embodiment can be applied to the host device in the live broadcast scenario. The live processing method includes:

[0087] S401. Add supplementary enhancement information to the live stream data of the live broadcast room, where the supplementary enhancement information includes the first layout information of the live broadcast room interface.

[0088] In this embodiment, the host device can obtain the first layout information of the live broadcast room interface and add the first layout information to the live stream data as supplementary enhancement information. Optionally, since the supplementary enhancement information is synchronized with the live stream data, the first layout information of the live broadcast room interface can be the layout information of the live broadcast room interface of the current frame of the live stream data. Optionally, in the scene of a joint live broadcast in the live broadcast room, the layout information of the live broadcast room interface includes the position information of the video images of each joint host, such as the coordinate information of the video images of each joint host (such as the coordinates of the four corners of the video image).

[0089] Optionally, the host device can add supplementary enhancement information to each frame of the live stream data, or add supplementary enhancement information at intervals of several frames, or add supplementary enhancement information to the live stream data only when the layout of the live broadcast room interface changes and the first layout information is updated.

[0090] S402. Add preset coding information to each frame of the live broadcast room interface image in the live stream data, where the preset coding information includes the second layout information of the live broadcast room interface.

[0091] In this embodiment, the host device can add the second layout information of the live broadcast room interface to each frame of the live broadcast room interface image in the live stream data through the preset coding information.

[0092] Optionally, the host device can synthesize the second layout information of the live broadcast room interface into each frame of the live broadcast room interface image in the live stream data through a blind watermark (or add a blind watermark at intervals of several frames). Of course, other known methods of adding invisible information to an image can also be used, and this is not limited in this embodiment.

[0093] The second layout information can be the layout information of the live broadcast room interface of the current frame (i.e., the same as the first layout information); considering that the information that the preset coding information (such as a blind watermark) can carry is limited, the second layout information can also be the identification information corresponding to the layout information of the live broadcast room interface of the current frame.

[0094] Optionally, in this embodiment, the second layout information of the live broadcast room interface can be synthesized into a preset area of each frame of the live broadcast room interface image in the live stream data through a blind watermark. The preset area can be the background area of the live broadcast room interface image, such as Figure 1 the upper and / or lower background areas in the live broadcast room interface image (the upper and lower gray areas), where no video image of any host is displayed, but a blind watermark can be synthesized therein.

[0095] Specifically, the preset area image of the live broadcast room interface image can be obtained first (if the preset area image has not changed, there is no need to repeatedly obtain the preset area image), and the second layout information is synthesized into the preset area image through a blind watermark (if the second layout information has not changed, there is no need to repeatedly execute the process of synthesizing the blind watermark into the preset area image), and then the synthesized preset area image is synthesized into each frame of the live broadcast room interface image.

[0096] Optionally, considering that adding a blind watermark in the frequency domain of the image has higher anti-attack performance and better stability, in this embodiment, the blind watermark can be synthesized into the live broadcast room interface image in the frequency domain. Specifically, each frame of the live broadcast room interface image can be transformed into the frequency domain, and the second layout information is synthesized into each frame of the live broadcast room interface image through a blind watermark in the frequency domain, and then each frame of the synthesized live broadcast room interface image is transformed back to the spatial domain. The transformation means used for transforming the live broadcast room interface image into the frequency domain is not limited, and can be Fourier transform, discrete cosine transform, wavelet transform, etc.; and the inverse transform of the above transformation means can be used to transform each frame of the synthesized live broadcast room interface image back to the spatial domain.

[0097] Based on the above embodiment, if the blind watermark is synthesized into the preset area of the live broadcast room interface image, the preset area image of the live broadcast room interface image can be obtained first (if the preset area image has not changed, there is no need to repeatedly obtain the preset area image); the preset area image is transformed into the frequency domain, and the second layout information is synthesized into the preset area image through a blind watermark in the frequency domain, and then the synthesized preset area image is transformed back to the spatial domain (if the second layout information has not changed, there is no need to repeatedly execute the process of synthesizing the blind watermark into the preset area image); and the synthesized preset area image is synthesized into each frame of the live broadcast room interface image in the spatial domain.

[0098] Taking the background area above and / or below the live broadcast room interface image as an example, the background area image can be obtained first (if the background area has not changed, there is no need to repeatedly obtain the background area image); the background area image is transformed into the frequency domain, and the second layout information is synthesized into the background area image through a blind watermark in the frequency domain, and then the synthesized background area image is transformed back to the spatial domain (if the second layout information has not changed, there is no need to repeatedly execute the process of synthesizing the blind watermark into the background area image); and the synthesized background area image is synthesized into each frame of the live broadcast room interface image in the spatial domain.

[0099] It should be noted that the execution order of S401 and S402 is not limited. Preferably, S401 and S402 are executed simultaneously.

[0100] S403. Push the live stream data to the server so that the viewer device can obtain the live stream data from the server for playback.

[0101] In this embodiment, after the host device completes the above process, the process of pushing the live stream data is performed, which will not be elaborated here; the viewer device performs the process of pulling the live stream data, and after pulling the stream, the live processing method on the viewer device side in the above embodiment can be executed.

[0102] For the live processing method provided in this embodiment, the host device adds supplementary enhancement information to the live stream data in the live room, where the supplementary enhancement information includes the first layout information of the live room interface; adds preset coding information to each frame of the live room interface image in the live stream data, where the preset coding information includes the second layout information of the live room interface; and pushes the live stream data to the server so that the viewer device can obtain the live stream data from the server for playback. The host device provides two ways for the viewer device to obtain the layout information of the live room interface, ensuring that the viewer device can stably obtain the layout information of the live room interface, timely perceive the change of the live room interface layout, and then perform corresponding layout control, ensuring the user experience.

[0103] Based on any of the above embodiments, when the layout of the live room interface in the current frame changes, for example, when the number of co-hosts in the live room changes, or when the position and size of the video images of each co-host in the live room interface change, etc., the layout information of the live room interface in the current frame changes, and the host device needs to update the above first layout information and second layout information to ensure that the supplementary enhancement information and the preset coding information (such as blind watermark) in the live stream data also change synchronously, so that the viewer device can perceive the change of the live room interface layout.

[0104] Among them, if the second layout information is the identification information corresponding to the layout information of the live room interface in the current frame, when the layout information of the live room interface in the current frame changes, update the second layout information, that is, update the identification information, and then upload the updated second layout information and the corresponding changed layout information of the live room interface in the current frame to the server, and maintain the mapping relationship between the identification information and the layout information on the server, so that the viewer device can query the corresponding layout information from the server according to the identification information after obtaining the identification information from the preset coding information (such as blind watermark).

[0105] Corresponding to the live processing method on the viewer device side in the above embodiment, Figure 5 This is the structural block diagram of the live processing device provided in the embodiments of the present disclosure. For the convenience of description, only the parts related to the embodiments of the present disclosure are shown. Refer to Figure 5, the live processing device 500 includes: an image acquisition unit 501, an analysis unit 502, and a layout control unit 503.

[0106] Among them, the image acquisition unit 501 is used to acquire the live room interface image when the supplementary enhancement information is not included in the live stream data; wherein the supplementary enhancement information includes the first layout information of the live room interface.

[0107] The analysis unit 502 is used to obtain the second layout information of the live room interface according to the preset coding information included in the live room interface image.

[0108] The layout control unit 503 is used to perform layout control on the live room interface according to the second layout information.

[0109] In one or more embodiments of the present disclosure, the device may further include a detection unit for detecting whether the supplementary enhancement information is included in the live stream data of the live room, wherein the supplementary enhancement information includes the first layout information of the live room interface.

[0110] In one or more embodiments of the present disclosure, when the layout control unit 503 performs layout control on the live room interface according to the second layout information, it is used for:

[0111] If the second interface layout information changes, the layout of the live room interface is changed according to the changed second interface layout information.

[0112] In one or more embodiments of the present disclosure, the first layout information is the layout information of the live room interface of the current frame.

[0113] The second layout information is the layout information of the live room interface of the current frame, or the second layout information is the identification information corresponding to the layout information of the live room interface of the current frame.

[0114] In one or more embodiments of the present disclosure, if the second layout information is the identification information corresponding to the layout information of the live room interface of the current frame, then if the second interface layout information changes, when the layout control unit 503 changes the layout of the live room interface according to the changed second interface layout information, it is used for:

[0115] If the identification information changes, request the layout information corresponding to the changed identification information from the server; wherein different identification information and different layout information mapping relationships are stored in the server.

[0116] Change the layout of the live room interface according to the layout information corresponding to the changed identification information.

[0117] In one or more embodiments of the present disclosure, the preset coding information included in the live broadcast room interface image is a blind watermark synthesized in the live broadcast room interface image, and the blind watermark includes the second layout information of the live broadcast room interface.

[0118] When obtaining the second layout information of the live broadcast room interface according to the preset coding information included in the live broadcast room interface image, the parsing unit 502 is configured to:

[0119] Parse the blind watermark included in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface.

[0120] In one or more embodiments of the present disclosure, when parsing the blind watermark included in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface included in the blind watermark, the parsing unit 502 is configured to:

[0121] Transform the live broadcast room interface image into the frequency domain, and parse the blind watermark included in the live broadcast room interface image in the frequency domain to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0122] In one or more embodiments of the present disclosure, when parsing the blind watermark included in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface included in the blind watermark, the parsing unit 502 is configured to:

[0123] Parse the blind watermark included in a preset area of the live broadcast room interface image to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0124] In one or more embodiments of the present disclosure, the layout control unit 503 is further configured to:

[0125] When entering the live broadcast room, obtain the latest layout information of the live broadcast room interface from the server, and load interface components in the live broadcast room interface according to the latest layout information of the live broadcast room interface to perform an initial layout of the live broadcast room interface.

[0126] The live broadcast processing device provided in this embodiment can be used to execute the technical solutions of the method embodiments on the viewer device side above. The implementation principles and technical effects are similar, and will not be elaborated here.

[0127] Corresponding to the live broadcast processing method on the host device side in the above embodiment, Figure 6 is a structural block diagram of the live broadcast processing device provided in the embodiments of the present disclosure. For the sake of simplicity, only parts related to the embodiments of the present disclosure are shown. Refer to Figure 6, the live processing device 600 includes: a data processing unit 601, a synthesis unit 602, and a communication unit 603.

[0128] Among them, the data processing unit 601 is used to add supplementary enhancement information to the live stream data in the live broadcast room, and the supplementary enhancement information includes the first layout information of the live broadcast room interface;

[0129] The synthesis unit 602 is used to add preset coding information to each frame of the live broadcast room interface image in the live stream data, and the preset coding information includes the second layout information of the live broadcast room interface;

[0130] The communication unit 603 is used to push the live stream data to the server for the viewer device to obtain the live stream data from the server for playback.

[0131] In one or more embodiments of the present disclosure, the first layout information is the layout information of the current frame of the live broadcast room interface;

[0132] The second layout information is the layout information of the current frame of the live broadcast room interface, or the second layout information is the identification information corresponding to the layout information of the current frame of the live broadcast room interface.

[0133] In one or more embodiments of the present disclosure, if the second layout information is the identification information corresponding to the layout information of the current frame of the live broadcast room interface;

[0134] The data processing unit 601 is further used to update the second layout information when the layout information of the current frame of the live broadcast room interface changes;

[0135] The communication unit 603 is further used to upload the updated second layout information and the corresponding changed layout information of the current frame of the live broadcast room interface to the server.

[0136] In one or more embodiments of the present disclosure, when the data processing unit 601 updates the second layout information when the layout information of the current frame of the live broadcast room interface changes, it is used for:

[0137] When the number of co-hosts in the live broadcast room changes, or when the position and size of the video images of each co-host in the live broadcast room interface change, it is determined that the layout information of the current frame of the live broadcast room interface has changed, and the second layout information is updated.

[0138] In one or more embodiments of the present disclosure, when the synthesis unit 602 adds preset coding information to each frame of the live broadcast room interface image in the live stream data, it is used for:

[0139] Synthesize the second layout information of the live broadcast room interface into each frame of the live broadcast room interface image of the live stream data through a blind watermark.

[0140] In one or more embodiments of the present disclosure, when the synthesizing unit 602 synthesizes the second layout information of the live broadcast room interface into each frame of the live broadcast room interface image of the live stream data through a blind watermark, it is configured to:

[0141] Synthesize the second layout information of the live broadcast room interface into a preset area of each frame of the live broadcast room interface image of the live stream data through a blind watermark.

[0142] In one or more embodiments of the present disclosure, when the synthesizing unit 602 synthesizes the second layout information of the live broadcast room interface into each frame of the live broadcast room interface image of the live stream data through a blind watermark, it is configured to:

[0143] Transform each frame of the live broadcast room interface image into the frequency domain, synthesize the second layout information into each frame of the live broadcast room interface image in the frequency domain through a blind watermark, and then transform the synthesized each frame of the live broadcast room interface image back to the spatial domain.

[0144] In one or more embodiments of the present disclosure, when the synthesizing unit 602 transforms each frame of the live broadcast room interface image into the frequency domain, synthesizes the second layout information into each frame of the live broadcast room interface image in the frequency domain through a blind watermark, and then transforms the synthesized each frame of the live broadcast room interface image back to the spatial domain, it is configured to:

[0145] Obtain the preset area image of the live broadcast room interface image, transform the preset area image into the frequency domain, synthesize the second layout information into the preset area image in the frequency domain through a blind watermark, then transform the synthesized preset area image back to the spatial domain, and synthesize the synthesized preset area image into each frame of the live broadcast room interface image in the spatial domain.

[0146] The live broadcast processing device provided in this embodiment can be used to execute the technical solutions of the method embodiments on the host device side described above. The implementation principles and technical effects are similar, and will not be elaborated here.

[0147] Refer to Figure 7, which shows a schematic structural diagram of an electronic device 700 suitable for implementing the embodiments of the present disclosure. The electronic device 700 can be a terminal device or a server. Among them, the terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0148] As Figure 7 shown, the electronic device 700 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage device 708 into the random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. The input / output (I / O) interface 705 is also connected to the bus 704.

[0149] Generally, the following devices can be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 can allow the electronic device 700 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 7 shows the electronic device 700 with various devices, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices can be implemented or had.

[0150] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a computer-readable medium, and the computer program includes program code for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by a processing device 701, it executes the live processing method of the above-described viewer device or host device defined in the method of the embodiment of the present disclosure.

[0151] It should be noted that the above-mentioned computer-readable medium in the present disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of a computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, and the computer-readable signal medium can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0152] The above-mentioned computer-readable medium can be included in the above-mentioned electronic device; or it can exist separately and not be assembled into the electronic device.

[0153] The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device is caused to execute the method shown in the above embodiment.

[0154] Computer program code for performing the operations of this disclosure can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computer (e.g., by connecting through the Internet using an Internet service provider).

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0156] The units involved in the embodiments described in this disclosure can be implemented in software or in hardware. Among them, the name of the unit does not constitute a limitation on the unit itself in some cases. For example, the first acquisition unit can also be described as "the unit for acquiring at least two Internet protocol addresses".

[0157] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGA), Application Specific Integrated Circuits (ASIC), Application Specific Standard Products (ASSP), Systems on Chip (SOC), Complex Programmable Logic Devices (CPLD), and so on.

[0158] In the context of the present disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0159] In a first aspect, according to one or more embodiments of the present disclosure, there is provided a live processing method, including:

[0160] When the supplementary enhancement information is not included in the live stream data, obtaining an image of the live room interface; wherein the supplementary enhancement information includes first layout information of the live room interface;

[0161] Obtaining second layout information of the live room interface according to preset coding information included in the image of the live room interface;

[0162] Performing layout control on the live room interface according to the second layout information.

[0163] According to one or more embodiments of the present disclosure, the performing layout control on the live room interface according to the second layout information includes:

[0164] If the second interface layout information changes, performing layout change on the live room interface according to the changed second interface layout information.

[0165] According to one or more embodiments of the present disclosure, the first layout information is the layout information of the live room interface of the current frame;

[0166] The second layout information is the layout information of the live room interface of the current frame, or the second layout information is identification information corresponding to the layout information of the live room interface of the current frame.

[0167] According to one or more embodiments of the present disclosure, if the second layout information is identification information corresponding to the layout information of the live room interface of the current frame, then the if the second interface layout information changes, performing layout change on the live room interface according to the changed second interface layout information includes:

[0168] If the identification information changes, request the layout information corresponding to the changed identification information from the server; wherein different mapping relationships between different identification information and different layout information are stored in the server;

[0169] Change the layout of the live broadcast room interface according to the layout information corresponding to the changed identification information.

[0170] According to one or more embodiments of the present disclosure, the preset coding information included in the live broadcast room interface image is a blind watermark synthesized in the live broadcast room interface image, and the blind watermark includes the second layout information of the live broadcast room interface;

[0171] The obtaining of the second layout information of the live broadcast room interface according to the preset coding information included in the live broadcast room interface image includes:

[0172] Parse the blind watermark included in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface.

[0173] According to one or more embodiments of the present disclosure, the parsing of the blind watermark included in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface included in the blind watermark includes:

[0174] Transform the live broadcast room interface image into the frequency domain, and parse the blind watermark included in the live broadcast room interface image in the frequency domain to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0175] According to one or more embodiments of the present disclosure, the parsing of the blind watermark included in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface included in the blind watermark includes:

[0176] Parse the blind watermark included in a preset area of the live broadcast room interface image to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0177] According to one or more embodiments of the present disclosure, before detecting whether the live stream data of the live broadcast room includes supplementary enhancement information, it further includes:

[0178] When entering the live broadcast room, obtain the latest layout information of the live broadcast room interface from the server, and load interface components in the live broadcast room interface according to the latest layout information of the live broadcast room interface to perform an initial layout of the live broadcast room interface.

[0179] In a second aspect, according to one or more embodiments of the present disclosure, a live broadcast processing method is provided, which is applied to a host device, and the method includes:

[0180] Add supplementary enhancement information to the live stream data of the live broadcast room, where the supplementary enhancement information includes the first layout information of the live broadcast room interface;

[0181] Add preset coding information to each frame of the live broadcast room interface image in the live stream data, where the preset coding information includes the second layout information of the live broadcast room interface;

[0182] Push the live stream data to the server for the viewer device to obtain the live stream data from the server for playback.

[0183] According to one or more embodiments of the present disclosure, the first layout information is the layout information of the live broadcast room interface of the current frame;

[0184] The second layout information is the layout information of the live broadcast room interface of the current frame, or the second layout information is the identification information corresponding to the layout information of the live broadcast room interface of the current frame.

[0185] According to one or more embodiments of the present disclosure, if the second layout information is the identification information corresponding to the layout information of the live broadcast room interface of the current frame, the method further includes:

[0186] When the layout information of the live broadcast room interface of the current frame changes, update the second layout information;

[0187] Upload the updated second layout information and the corresponding changed layout information of the live broadcast room interface of the current frame to the server.

[0188] According to one or more embodiments of the present disclosure, when the layout information of the live broadcast room interface of the current frame changes, updating the second layout information includes:

[0189] When the number of co-hosts in the live broadcast room changes, or the position and size of the video images of each co-host in the live broadcast room change in the live broadcast room interface, it is determined that the layout information of the live broadcast room interface of the current frame has changed, and the second layout information is updated.

[0190] According to one or more embodiments of the present disclosure, adding preset coding information to each frame of the live broadcast room interface image in the live stream data includes:

[0191] Synthesize the second layout information of the live broadcast room interface into each frame of the live broadcast room interface image of the live stream data through blind watermarking.

[0192] According to one or more embodiments of the present disclosure, synthesizing the second layout information of the live broadcast room interface into each frame of the live broadcast room interface image of the live stream data through blind watermarking includes:

[0193] Synthesize the second layout information of the live broadcast room interface into a preset area of each frame of the live broadcast room interface image in the live stream data through a blind watermark.

[0194] According to one or more embodiments of the present disclosure, synthesizing the second layout information of the live broadcast room interface into each frame of the live broadcast room interface image in the live stream data through a blind watermark includes:

[0195] Transform each frame of the live broadcast room interface image into the frequency domain, synthesize the second layout information into each frame of the live broadcast room interface image in the frequency domain through a blind watermark, and then transform the synthesized each frame of the live broadcast room interface image back into the spatial domain.

[0196] According to one or more embodiments of the present disclosure, transforming each frame of the live broadcast room interface image into the frequency domain, synthesizing the second layout information into each frame of the live broadcast room interface image in the frequency domain through a blind watermark, and then transforming the synthesized each frame of the live broadcast room interface image back into the spatial domain includes:

[0197] Obtain an image of a preset area of the live broadcast room interface image, transform the image of the preset area into the frequency domain, synthesize the second layout information into the image of the preset area in the frequency domain through a blind watermark, then transform the synthesized image of the preset area back into the spatial domain, and synthesize the synthesized image of the preset area into each frame of the live broadcast room interface image in the spatial domain.

[0198] In a third aspect, according to one or more embodiments of the present disclosure, there is provided a live broadcast processing device, including:

[0199] An image acquisition unit, configured to acquire a live broadcast room interface image when the supplementary enhancement information is not included in the live stream data; wherein the supplementary enhancement information includes the first layout information of the live broadcast room interface;

[0200] An analysis unit, configured to obtain the second layout information of the live broadcast room interface according to preset coding information included in the live broadcast room interface image;

[0201] A layout control unit, configured to perform layout control on the live broadcast room interface according to the second layout information.

[0202] According to one or more embodiments of the present disclosure, when performing layout control on the live broadcast room interface according to the second layout information, the layout control unit is configured to:

[0203] If the second interface layout information changes, perform layout change on the live broadcast room interface according to the changed second interface layout information.

[0204] According to one or more embodiments of the present disclosure, the first layout information is the layout information of the live broadcast room interface of the current frame;

[0205] The second layout information is the layout information of the live broadcast room interface of the current frame, or the second layout information is the identification information corresponding to the layout information of the live broadcast room interface of the current frame.

[0206] According to one or more embodiments of the present disclosure, if the second layout information is the identification information corresponding to the layout information of the live broadcast room interface of the current frame, then if the second interface layout information changes, when the layout control unit changes the layout of the live broadcast room interface according to the changed second interface layout information, it is used for:

[0207] If the identification information changes, request the layout information corresponding to the changed identification information from the server; wherein different identification information and different layout information mapping relationships are stored in the server;

[0208] Change the layout of the live broadcast room interface according to the layout information corresponding to the changed identification information.

[0209] According to one or more embodiments of the present disclosure, the preset coding information included in the live broadcast room interface image is a blind watermark synthesized in the live broadcast room interface image, and the blind watermark includes the second layout information of the live broadcast room interface.

[0210] Obtaining the second layout information of the live broadcast room interface according to the preset coding information included in the live broadcast room interface image includes:

[0211] Parse the blind watermark included in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface.

[0212] According to one or more embodiments of the present disclosure, when the parsing unit parses the blind watermark included in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface included in the blind watermark, it is used for:

[0213] Transform the live broadcast room interface image into the frequency domain, and parse the blind watermark included in the live broadcast room interface image in the frequency domain to obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0214] According to one or more embodiments of the present disclosure, when the parsing unit parses the blind watermark included in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface included in the blind watermark, it is used for:

[0215] Parse the blind watermark included in the preset area of the live broadcast room interface image, and obtain the second layout information of the live broadcast room interface included in the blind watermark.

[0216] According to one or more embodiments of the present disclosure, the layout control unit is further configured to:

[0217] When entering the live broadcast room, obtain the latest layout information of the live broadcast room interface from the server, and load interface components in the live broadcast room interface according to the latest layout information of the live broadcast room interface to perform an initial layout on the live broadcast room interface.

[0218] Fourthly, according to one or more embodiments of the present disclosure, there is provided a live broadcast processing device, including:

[0219] A data processing unit, configured to add supplementary enhancement information to the live stream data of the live broadcast room, where the supplementary enhancement information includes the first layout information of the live broadcast room interface;

[0220] A synthesis unit, configured to add preset coding information to each frame of the live broadcast room interface image of the live stream data, where the preset coding information includes the second layout information of the live broadcast room interface;

[0221] A communication unit, configured to push the live stream data to the server for the viewer device to obtain the live stream data from the server for playing.

[0222] According to one or more embodiments of the present disclosure, the first layout information is the layout information of the current frame of the live broadcast room interface;

[0223] The second layout information is the layout information of the current frame of the live broadcast room interface, or the second layout information is the identification information corresponding to the layout information of the current frame of the live broadcast room interface.

[0224] According to one or more embodiments of the present disclosure, if the second layout information is the identification information corresponding to the layout information of the current frame of the live broadcast room interface;

[0225] The data processing unit is further configured to update the second layout information when the layout information of the current frame of the live broadcast room interface changes;

[0226] The communication unit is further configured to upload the updated second layout information and the corresponding changed layout information of the current frame of the live broadcast room interface to the server.

[0227] According to one or more embodiments of the present disclosure, when the data processing unit updates the second layout information when the layout information of the current frame of the live broadcast room interface changes, it is configured to:

[0228] When the number of co-hosts in the live broadcast room changes, or when the position and size of the video images of the co-hosts in the live broadcast room change in the live broadcast room interface, it is determined that the layout information of the live broadcast room interface of the current frame has changed, and the second layout information is updated.

[0229] According to one or more embodiments of the present disclosure, when adding preset coding information to each frame of the live broadcast room interface image of the live stream data, the synthesizing unit is configured to:

[0230] Synthesize the second layout information of the live broadcast room interface into each frame of the live broadcast room interface image of the live stream data through blind watermarking.

[0231] According to one or more embodiments of the present disclosure, when synthesizing the second layout information of the live broadcast room interface into each frame of the live broadcast room interface image of the live stream data through blind watermarking, the synthesizing unit is configured to:

[0232] Synthesize the second layout information of the live broadcast room interface into a preset area of each frame of the live broadcast room interface image of the live stream data through blind watermarking.

[0233] According to one or more embodiments of the present disclosure, when the synthesizing unit synthesizes the second layout information of the live broadcast room interface into each frame of the live broadcast room interface image of the live stream data through blind watermarking, the synthesizing unit is configured to:

[0234] Transform each frame of the live broadcast room interface image into the frequency domain, synthesize the second layout information into each frame of the live broadcast room interface image in the frequency domain through blind watermarking, and then transform the synthesized each frame of the live broadcast room interface image back to the spatial domain.

[0235] According to one or more embodiments of the present disclosure, when transforming each frame of the live broadcast room interface image into the frequency domain, synthesizing the second layout information into each frame of the live broadcast room interface image in the frequency domain through blind watermarking, and then transforming the synthesized each frame of the live broadcast room interface image back to the spatial domain, the synthesizing unit is configured to:

[0236] Obtain the preset area image of the live broadcast room interface image, transform the preset area image into the frequency domain, synthesize the second layout information into the preset area image in the frequency domain through blind watermarking, then transform the synthesized preset area image back to the spatial domain, and synthesize the synthesized preset area image into each frame of the live broadcast room interface image in the spatial domain.

[0237] In a fifth aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, including: at least one processor and a memory;

[0238] The memory stores computer-executable instructions;

[0239] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the live processing method as described in the first aspect above and various possible designs of the first aspect.

[0240] In a sixth aspect, according to one or more embodiments of the present disclosure, there is provided a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the live processing method as described in the first aspect above and various possible designs of the first aspect.

[0241] In a seventh aspect, according to one or more embodiments of the present disclosure, there is provided a computer program product including computer-executable instructions, which, when executed by a processor, implement the live processing method as described in the first aspect above and various possible designs of the first aspect.

[0242] The above description is only a preferred embodiment of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) having similar functions disclosed in the present disclosure.

[0243] In addition, although the operations are depicted in a specific order, this should not be construed as requiring the operations to be performed in the specific order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although a number of specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments may also be implemented combinatorially in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0244] Although the subject matter has been described in language specific to structural features and / or methodological act logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms for implementing the claims.

Claims

1. A live broadcast processing method, characterized in that, applied to the viewer device, the method includes: When the supplementary enhancement information is not included in the live stream data, obtaining the live broadcast room interface image; wherein the supplementary enhancement information includes the first layout information of the live broadcast room interface; Obtaining the second layout information of the live broadcast room interface according to the preset coding information included in the live broadcast room interface image; Performing layout control on the live broadcast room interface according to the second layout information.

2. The method according to claim 1, characterized in that, The performing layout control on the live broadcast room interface according to the second layout information includes: If the second interface layout information changes, performing layout change on the live broadcast room interface according to the changed second interface layout information.

3. The method according to claim 2, characterized in that, The first layout information is the layout information of the current frame of the live broadcast room interface; The second layout information is the layout information of the current frame of the live broadcast room interface, or the second layout information is the identification information corresponding to the layout information of the current frame of the live broadcast room interface.

4. The method according to claim 3, characterized in that, If the second layout information is the identification information corresponding to the layout information of the current frame of the live broadcast room interface, then the if the second interface layout information changes, performing layout change on the live broadcast room interface according to the changed second interface layout information includes: If the identification information changes, requesting the layout information corresponding to the changed identification information from the server; wherein different identification information and different layout information mapping relationships are stored in the server; Performing layout change on the live broadcast room interface according to the layout information corresponding to the changed identification information.

5. The method according to claim 1, characterized in that, The preset coding information included in the live broadcast room interface image is a blind watermark synthesized in the live broadcast room interface image, and the blind watermark includes the second layout information of the live broadcast room interface; The obtaining the second layout information of the live broadcast room interface according to the preset coding information included in the live broadcast room interface image includes: Parsing the blind watermark included in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface.

6. The method according to claim 5, characterized in that, The parsing the blind watermark included in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface includes: Transforming the live broadcast room interface image to the frequency domain, and parsing the blind watermark included in the live broadcast room interface image in the frequency domain to obtain the second layout information of the live broadcast room interface.

7. The method according to claim 5 or 6, characterized in that, The parsing the blind watermark included in the live broadcast room interface image to obtain the second layout information of the live broadcast room interface includes: Parsing the blind watermark included in the preset area of the live broadcast room interface image to obtain the second layout information of the live broadcast room interface.

8. A live broadcast processing method, characterized in that, applied to the host device, the method includes: Add supplementary enhancement information to the live stream data in the live broadcast room, where the supplementary enhancement information includes the first layout information of the live broadcast room interface; Add preset coding information to each frame of the live broadcast room interface image in the live stream data, where the preset coding information includes the second layout information of the live broadcast room interface; Push the live stream data to the server for the viewer device to obtain the live stream data from the server for playback.

9. The method according to claim 8, wherein, The first layout information is the layout information of the current frame of the live broadcast room interface; The second layout information is the layout information of the current frame of the live broadcast room interface, or the second layout information is the identification information corresponding to the layout information of the current frame of the live broadcast room interface.

10. The method according to claim 9, wherein, If the second layout information is the identification information corresponding to the layout information of the current frame of the live broadcast room interface, the method further includes: When the layout information of the current frame of the live broadcast room interface changes, update the second layout information; Upload the updated second layout information and the corresponding changed layout information of the current frame of the live broadcast room interface to the server.

11. The method according to claim 9, wherein, The updating the second layout information when the layout information of the current frame of the live broadcast room interface changes includes: When the number of co-hosts in the live broadcast room changes, or the position and size of the video images of each co-host in the live broadcast room change in the live broadcast room interface, it is determined that the layout information of the current frame of the live broadcast room interface has changed, and the second layout information is updated.

12. The method according to claim 8, wherein, The adding preset coding information to each frame of the live broadcast room interface image in the live stream data includes: Synthesize the second layout information of the live broadcast room interface into each frame of the live broadcast room interface image in the live stream data through blind watermarking.

13. The method according to claim 12, wherein, The synthesizing the second layout information of the live broadcast room interface into each frame of the live broadcast room interface image in the live stream data through blind watermarking includes: Synthesize the second layout information of the live broadcast room interface into a preset area of each frame of the live broadcast room interface image in the live stream data through blind watermarking.

14. The method according to claim 12 or 13, wherein, The synthesizing the second layout information of the live broadcast room interface into each frame of the live broadcast room interface image in the live stream data through blind watermarking includes: Transform each frame of the live broadcast room interface image into the frequency domain, synthesize the second layout information into each frame of the live broadcast room interface image through blind watermarking in the frequency domain, and then transform the synthesized each frame of the live broadcast room interface image back to the spatial domain.

15. The method according to claim 14, wherein, The transforming each frame of the live broadcast room interface image into the frequency domain, synthesizing the second layout information into each frame of the live broadcast room interface image through blind watermarking in the frequency domain, and then transforming the synthesized each frame of the live broadcast room interface image back to the spatial domain includes: Obtain the image of the preset area of the live broadcast room interface, transform the image of the preset area into the frequency domain, synthesize the second layout information into the image of the preset area in the frequency domain through blind watermarking, then convert the synthesized image of the preset area back to the spatial domain, and synthesize the synthesized image of the preset area into each frame of the live broadcast room interface image in the spatial domain.

16. A live broadcast processing device, characterized in that, it includes: An image acquisition unit, configured to obtain the image of the live broadcast room interface when the supplementary enhancement information is not included in the live broadcast stream data; wherein the supplementary enhancement information includes the first layout information of the live broadcast room interface; An analysis unit, configured to obtain the second layout information of the live broadcast room interface according to the preset coding information included in the live broadcast room interface image; A layout control unit, configured to perform layout control on the live broadcast room interface according to the second layout information.

17. A live broadcast processing device, characterized in that, it includes: A data processing unit, configured to add supplementary enhancement information to the live broadcast stream data of the live broadcast room, wherein the supplementary enhancement information includes the first layout information of the live broadcast room interface; A synthesis unit, configured to add preset coding information to each frame of the live broadcast room interface image in the live broadcast stream data, wherein the preset coding information includes the second layout information of the live broadcast room interface; A communication unit, configured to push the live broadcast stream data to the server for the client device to obtain the live broadcast stream data from the server for playback.

18. An electronic device, characterized in that, it includes: At least one processor and a memory; The memory stores computer execution instructions; The at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the method according to any one of claims 1-15.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the method according to any one of claims 1-15 is implemented.

20. A computer program product, characterized in that, It includes computer execution instructions, and when the processor executes the computer execution instructions, the method according to any one of claims 1-15 is implemented.

Images