Live broadcast stream processing method and device, computer equipment, storage medium and program product

Abstract

The embodiment of the invention provides a live broadcast stream processing method and device, computer equipment, a storage medium and a program product, and relates to the technical field of video live broadcast, and the method comprises the steps: obtaining gift information given by a user in a live broadcast room; determining a background effect of the live broadcast room according to the gift information, wherein the background effect comprises a to-be-replaced live broadcast room background image and the duration of the live broadcast room background image; determining a video frame to be subjected to background replacement in an original live broadcast stream corresponding to the live broadcast room based on the duration and the gift giving time in the gift information; and performing background replacement processing on the video frame to be subjected to background replacement in the original live broadcast stream according to the live broadcast room background image to obtain a target live broadcast stream. According to the technical scheme of the embodiment of the invention, the background of the live broadcasting room can be dynamically switched along with the gift giving behavior of the user, and the interaction experience is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video live broadcast technology, and in particular, to a live stream processing method, apparatus, computer equipment, storage medium, and program product. Background Art

[0002] In existing live broadcast platforms, the main ways for viewers to interact with anchors are to send bullet comments, give gifts, etc. After viewers give gifts to anchors through the gift system, the corresponding gift screen will generally be displayed in the live broadcast room to enhance the interactivity between viewers and anchors.

[0003] However, the existing gift systems generally only display the number of gifts or simple animation effects when displaying gifts. The visual effects of the gift display lack variation, resulting in the interaction between the audience and the anchor being limited to superficial interaction and lacking in-depth interactive experience, resulting in a poor user experience.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0005] The embodiments of the present application provide a live stream processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0006] An aspect of an embodiment of the present application provides a live stream processing method, the method comprising: Get information about gifts given by users in the live broadcast room; Determine the background effect of the live broadcast room according to the gift information, wherein the background effect includes the live broadcast room background image to be replaced and the duration of the live broadcast room background image; Determine the video frame to be replaced with the background in the original live stream corresponding to the live broadcast room based on the duration and the gift giving time in the gift information; Background replacement processing is performed on the video frame to be replaced with the background in the original live stream according to the live broadcast room background image to obtain a target live stream.

[0007] Optionally, determining the background effect of the live broadcast room according to the gift information includes: Perform gift category identification on the gift information to obtain category information of the gift information; The background effect of the live broadcast room is determined based on the category information.

[0008] Optionally, the gift information includes type information, quantity information and value information of the virtual gift, and the performing gift category identification on the gift information to obtain the category information of the gift information includes: Performing a first identification process on the type information in the gift information to obtain a first category of the gift information; Performing a second identification process on the quantity information in the gift information to obtain a second category of the gift information; Performing a third identification process on the value information in the gift information to obtain a third category of the gift information; Category information of the gift information is determined based on the first category, the second category, and the third category.

[0009] Optionally, performing background replacement processing on the video frame to be replaced with the background in the live stream according to the live broadcast room background image to obtain the target live stream includes: Performing a cutout process on the video frame to be replaced with the background in the original live stream in an asynchronous manner, so as to separate the anchor image from the video frame to be replaced with the background; GPU acceleration technology is used to perform image synthesis processing on the live broadcast room background image and the separated anchor image to obtain the target live broadcast stream.

[0010] Optionally, performing cutout processing on the video frame to be replaced with the background in the original live stream to separate the anchor image from the video frame to be replaced with the background includes: Using a pre-trained face recognition model to perform face recognition on the video frame to be replaced with the background in the original live stream, so as to identify the position of the anchor avatar from the video frame to be replaced with the background; The video frame to be replaced with the background is subjected to cutout processing according to the position of the anchor portrait, so as to separate the anchor image from the video frame to be replaced with the background.

[0011] Optionally, the method further comprises: The target live stream is transcoded to obtain live streams with multiple resolutions.

[0012] Optionally, the method further comprises: The target live broadcast stream is pushed to the audience terminal in the live broadcast room, so that the audience terminal can decode and play the target live broadcast stream.

[0013] Another aspect of an embodiment of the present application provides a live stream processing device, the device comprising: The acquisition module is used to obtain the gift information given by the user in the live broadcast room; A first determination module is used to determine the background effect of the live broadcast room according to the gift information, wherein the background effect includes a live broadcast room background image to be replaced and a duration of the live broadcast room background image; A second determination module is used to determine the video frame to be replaced with the background in the original live stream corresponding to the live broadcast room based on the duration and the gift giving time in the gift information; The replacement module is used to perform background replacement processing on the video frame to be replaced with the background in the original live stream according to the live broadcast room background image to obtain the target live stream.

[0014] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0015] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0016] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0017] The above technical solution adopted in the embodiment of the present application may have the following advantages: The gift information given by the user in the live broadcast room is obtained, and the background effect of the live broadcast room is determined according to the gift information, wherein the background effect includes the live broadcast room background image to be replaced and the duration of the live broadcast room background image, wherein different gift information corresponds to different background effects. Based on the duration and the gift giving time in the gift information, the video frames to be replaced with the background in the original live broadcast stream corresponding to the live broadcast room are determined, thereby determining which video frames need to be processed for background replacement. The background replacement processing is performed on the video frames to be replaced with the background in the original live broadcast stream according to the live broadcast room background image to obtain the target live broadcast stream. The target live broadcast stream obtained in the above manner can enable the audience terminal to display the screen in which the background of the live broadcast room is dynamically switched following the user's gift-giving behavior when decoding and playing the target live broadcast stream. In this way, the audience can intuitively feel the impact of the gift-giving behavior on the background of the live broadcast room and enhance the interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 The following schematically shows an operating environment diagram of a live streaming processing method according to Embodiment 1 of the present application; Figure 2 The flowchart of the live streaming processing method according to the first embodiment of the present application is schematically shown; Figure 3 Schematically shows Figure 2 Flow chart of sub-steps of step S202; Figure 4 Schematically shows Figure 3 Flow chart of sub-steps of step S300; Figure 5 Schematically shows Figure 2 Flow chart of sub-steps of step S206; Figure 6 Schematically shows Figure 5 Flow chart of sub-steps of step S500; Figure 7 A block diagram schematically shows a processing device for a live broadcast room according to Embodiment 2 of the present application; and Figure 8 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0021] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0022] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0023] First, the following terms are explained: Asynchronous: is a term in computer science and programming that describes an operation mode in which the execution of a task does not block the execution flow of the current task, that is, when executing other tasks, there is no need to wait for the current task to complete before executing the other task.

[0024] GPU acceleration technology: refers to the technology that uses graphics processing units (GPUs) to increase the processing speed of computing tasks. Common GPU acceleration technologies include OpenGL (Open Graphics Library) and Vulkan. Among them, OpenGL is a cross-language, cross-platform graphics application programming interface (API) for rendering 2D and 3D vector graphics. Vulkan is a modern cross-platform graphics and computing API that provides more direct control over the GPU to improve performance and efficiency.

[0025] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: In current live broadcast technology, the host's background is often fixed. Even if the virtual background is replaced through green screen technology, these backgrounds are usually pre-set and cannot change in real time based on the audience's gift-giving behavior. As a result, the interaction between the audience and the host is often limited to superficial interaction and lacks an in-depth interactive experience.

[0026] To this end, the embodiment of the present application provides a live streaming processing technology solution. In this technology solution: (1) When the audience sends a gift, the gift type can be instantly identified, and the background of the live broadcast room can be dynamically switched according to the preset rules, so that the audience can immediately see the effect of their gift-giving behavior, which enhances the user's sense of participation and satisfaction. (2) When the audience sends a gift, different background switching effects can be generated according to different gift types. For example, when sending a flower gift, the background becomes a sea of ​​flowers, and when sending a sailing gift, the background becomes the sea, which brings rich visual changes and improves the overall viewing quality of the live broadcast. (3) When the audience sends a gift, the background switching effects of different complexities can be generated according to the different gift values, so that the gift-giving actions of the audience with different values ​​can produce unique visual effects. (4) By realizing the real-time dynamic change of the live broadcast room background based on the user's gift-giving behavior, not only the audience's interactive experience is enhanced, but also a more vivid and rich visual effect is created for the live broadcast room, which improves the entertainment and attractiveness of the live broadcast content, thereby increasing the audience's retention rate and the user stickiness of the platform. See below for details.

[0027] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0028] like Figure 1 As shown, the operating environment diagram includes: service platform 2, anchor terminals (4A, 4B, ..., 4M), and audience terminals (6A, 6B, ..., 6N). In the live broadcast scenario, the anchor terminal (4A, 4B, ..., 4M) logs in to the service platform 2, and the service platform 2 pushes the live broadcast data to the audience terminal (6A, 6B, ..., 6N) in real time.

[0029] Service platform 2 can provide live broadcast service, which can be a single server, a server cluster or a cloud computing service center.

[0030] The anchor terminal (4A, 4B, ..., 4M) is used to generate live broadcast data in real time and perform streaming operations on the live broadcast data. The live broadcast data may include audio data or video data. The anchor terminal may be an electronic device such as a smart phone or a tablet computer. Of course, the anchor terminal may be a virtual computing instance in the service platform 2.

[0031] The audience terminal (6A, 6B, ..., 6N) may be configured to receive the live broadcast data of the anchor terminal in real time. The audience terminal (6A, 6B, ..., 6N) may be any type of computing device, such as a smart phone, a tablet device, a laptop computer, a smart TV, a vehicle-mounted terminal, etc. The audience terminal (6A, 6B, ..., 6N) may have a built-in browser or a special program, and receive the live broadcast data through the browser or the special program to output content to the user. The content may include video, audio, comments, text data and / or the like.

[0032] The audience terminal (6A, 6B, ..., 6N) may include a player. The player outputs (e.g., displays, presents) content to the user. The content may include video, audio, comments, text data and / or the like. The audience terminal (6A, 6B, ..., 6N) may include an interface, which may include an input element (touch screen). For example, the input element may be configured to receive user instructions, and the user instructions may enable the audience terminal (6A, 6B, ..., 6N) to perform various operations, such as sending a bullet screen, entering a comment, giving a gift, etc.

[0033] The anchor terminals (4A, 4B, ..., 4M), the audience terminals (6A, 6B, ..., 6N) and the service platform 2 may be connected via a network. The network may include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls and / or proxy devices, etc. The network may include physical links, such as coaxial cable links, twisted pair cable links, optical fiber links, combinations thereof and / or the like. The network may include wireless links, such as cellular links, satellite links, Wi-Fi links and / or the like.

[0034] It should be noted that the number of the host terminals and the audience terminals in the figure is only for illustration and is not used to limit the scope of patent protection of this application. According to actual conditions, there can be any number of host terminals and audience terminals.

[0035] The following uses a server as the execution subject to introduce the technical solution of the present application through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be interpreted as being limited to the embodiments described here.

[0036] Embodiment 1 Figure 2 The flowchart of the live streaming processing method according to the first embodiment of the present application is schematically shown.

[0037] like Figure 2 As shown, the live stream processing method may include steps S200 to S206, wherein: Step S200, obtaining information about gifts given by users in the live broadcast room.

[0038] Step S202, determining the background effect of the live broadcast room according to the gift information, wherein the background effect includes the live broadcast room background image to be replaced and the duration of the live broadcast room background image.

[0039] Step S204, determining the video frame to be replaced with the background in the original live stream corresponding to the live broadcast room based on the duration and the gift giving time in the gift information.

[0040] Step S206, performing background replacement processing on the video frame to be replaced with the background in the original live stream according to the live broadcast room background image to obtain a target live stream.

[0041] The live stream processing method provided in this embodiment obtains the gift information given by the user in the live broadcast room, and determines the background effect of the live broadcast room according to the gift information, wherein the background effect includes the live broadcast room background image to be replaced and the duration of the live broadcast room background image, wherein different gift information corresponds to different background effects. Based on the duration and the gift giving time in the gift information, the video frames to be replaced with the background in the original live broadcast stream corresponding to the live broadcast room are determined, thereby determining which video frames need to be processed for background replacement. The background replacement processing is performed on the video frames to be replaced with the background in the original live broadcast stream according to the live broadcast room background image to obtain the target live broadcast stream. The target live broadcast stream obtained in the above manner can enable the audience terminal to display the screen in which the background of the live broadcast room is dynamically switched following the gift-giving behavior of the user when decoding and playing the target live broadcast stream. In this way, the audience can intuitively feel the impact of the gift-giving behavior on the background of the live broadcast room and enhance the interactive experience.

[0042] The following combination Figure 2 , each step in steps S200~S206 and other optional steps are described in detail.

[0043] Step S200 , get the gift information given by the user in the live broadcast room.

[0044] The gift information is information about the virtual gift given by the user to the anchor, and may include information about the type, quantity, value, and gift giving time of the virtual gift. The type information is information used to determine what type of virtual gift the virtual gift is, and the type information may be the name of the virtual gift. The quantity information is information used to determine the number of virtual gifts. The value information is information used to determine the value of the virtual gift.

[0045] Exemplarily, when a viewer sends a virtual gift, the viewer terminal sends the gift information to the server through the API interface. In this embodiment, in order to obtain the gift information sent by the viewer terminal, a monitoring log can be added in the server to monitor the gift information sent by the viewer terminal to the server.

[0046] Step S202 , determine the background effect of the live broadcast room according to the gift information, and the background effect includes the live broadcast room background image to be replaced and the duration of the live broadcast room background image.

[0047] Exemplarily, a gift-background effect library may be maintained in advance, through which background effects for each virtual gift, background effects for virtual gifts of different values, and background effects for different quantities of the same virtual gift are defined.

[0048] In an optional embodiment, if Figure 3 As shown, step S202 may include: Step S300, identifying the gift category of the gift information to obtain the category information of the gift information.

[0049] Step S302, determining the background effect of the live broadcast room based on the category information.

[0050] For example, after acquiring the gift information, the gift category of the virtual gift can be identified so as to classify different gift information, for example, distinguishing between ordinary gifts, medium-value gifts, high-value gifts, etc. Another example is distinguishing between flower gifts, sailing gifts, celebration gifts, etc. Another example is distinguishing between single gifts and batch gifts, etc. After identifying the category information, the corresponding background effect can be matched according to the category information.

[0051] In this embodiment, by first identifying the category information of the gift and then matching the corresponding background effect based on the category information, it is possible to quickly and accurately determine the background effect of the live broadcast room corresponding to the current gift information.

[0052] In an optional embodiment, the gift information includes type information, quantity information and value information of the virtual gift, such as Figure 4 As shown, step S300 may include: Step S400: Perform a first identification process on the type information in the gift information to obtain a first category of the gift information.

[0053] Step S402: Perform a second identification process on the quantity information in the gift information to obtain a second category of the gift information.

[0054] Step S404: Perform a third identification process on the value information in the gift information to obtain a third category of the gift information.

[0055] Step S406: determining category information of the gift information based on the first category, the second category, and the third category.

[0056] Exemplarily, the first category of each virtual gift is predefined. For example, rose gifts, chrysanthemum gifts, sunflower gifts, etc. are defined as the flower category; destroyer gifts, aircraft carrier gifts, etc. are defined as the nautical category; fireworks gifts, salute gifts, etc. are defined as the celebration category. When the audience sends a virtual gift, the type information of the gift is identified from the gift information, and then the corresponding first category is matched according to the type information. Each first category gift will trigger a specific background effect. For example, flower gifts trigger a flower sea background, and nautical gifts trigger an ocean background.

[0057] In addition, second categories corresponding to different quantities of the same virtual gift are pre-defined. For example, the category to which one virtual gift belongs is defined as a single gift; the category to which multiple identical virtual gifts belong is defined as a batch gift. When the audience sends a virtual gift, the quantity information of the gift is identified from the gift information, and then the corresponding second category is matched according to the quantity information. Different second category gifts will trigger background effects of different complexity or dynamics. For example, sending one gift at a time will trigger a slight background change effect, while sending 10 gifts in batches will trigger a more significant background change effect.

[0058] In addition, the third category corresponding to the value of each virtual gift will be pre-defined. For example, virtual gifts with a value of 0-10 are defined as ordinary gifts, virtual gifts with a value of 10-50 are defined as medium-value gifts, and virtual gifts with a value of more than 50 are defined as high-value gifts. When the audience sends a virtual gift, the value information of the gift is identified from the gift information, and then the corresponding third category is matched according to the value information. Each third category of gifts will trigger background effects of different complexity or dynamics. For example, high-value gifts will trigger more complex or special background effects, such as backgrounds with dynamic particle effects, light and shadow effects, etc., while ordinary gifts will trigger simpler background effects.

[0059] For example, when the gift given by the audience is a rose, the first category of the gift will be identified as a flower gift, and the corresponding background effect will be a flower sea background, for example, a picture of flying petals will appear in the live broadcast. When the gift given by the audience is a sailboat, the first category of the gift will be identified as a sailing gift, and the corresponding background effect will be a sea background, for example, the live broadcast will show the host on the vast ocean. When the gift given by the audience is fireworks, the first category of the gift will be identified as a celebration gift, and the corresponding background effect will be a fireworks background, for example, the live broadcast will show the night sky with fireworks blooming.

[0060] For example, if the gift given by the audience is 1 "rose", the second category of the gift will be identified as a single gift, and the corresponding background effect is a few petals falling, with a small number of petals. If the gift given by the audience is 10 "roses", the second category of the gift will be identified as a batch gift, and the corresponding background effect is a whole flower sea animation filling the live screen, with a large number of petals.

[0061] For example, when the gift given by the audience is a "little heart" gift with a value of 5, the third category of the gift will be identified as a normal gift, and the corresponding background effect is a slight color change or a small heart animation, which lasts for a short time and will not interfere with the live broadcast content. When the gift given by the audience is a "golden rose" gift with a value of 20, the third category of the gift will be identified as a medium-value gift, and the corresponding background effect is a more conspicuous petal effect, with significantly more color and dynamic elements in the background, and lasts for a slightly longer time.

[0062] In this embodiment, the category information is comprehensively determined based on the type information, quantity information and value information, thereby facilitating the subsequent matching of more background effects and enhancing the user experience.

[0063] Step S204 , based on the duration and the gift giving time in the gift information, determine the video frame to be replaced by the background in the original live stream corresponding to the live room.

[0064] Exemplarily, the duration is used to determine how many video frames in the original video stream need to be processed for background replacement. For example, if the duration is 1 second and the frame rate of the original video stream is 30 frames per second, it can be determined that 30 video frames in the original video stream need to be processed for background replacement. The gift giving time is used to determine at what point in time the original video stream starts to be processed for background replacement. For example, if the gift giving time is 2 minutes and 30 seconds, it can be determined that the video frame corresponding to the 2nd minute and 30 seconds in the original video stream is the first video frame that needs to be processed for background replacement.

[0065] Step S206 , performing background replacement processing on the video frame to be replaced with the background in the original live stream according to the live broadcast room background image to obtain a target live stream.

[0066] Exemplarily, after determining the video frame that needs background replacement, the live broadcast room background image in the corresponding video frame will be replaced by each of the live broadcast room background images in turn to obtain the target live broadcast stream. Afterwards, the target live broadcast stream will be sent to the audience terminals corresponding to all viewers in the live broadcast room so that all viewers can watch the background switching effect of giving gifts.

[0067] In one embodiment, when the live broadcast room background image is replaced with the live broadcast room background image in the video frame, for gifts given by different viewers, the corresponding live broadcast room background image can also be determined according to the gifts given by each viewer, and then, the live broadcast room background image in the corresponding video frame is replaced according to the live broadcast room background image corresponding to different viewers, so that different target live broadcast streams can be obtained, and finally, the different target live broadcast streams are sent to the corresponding viewer terminals. For example, the target live broadcast stream a obtained by viewer A giving gift a is sent to the viewer terminal corresponding to viewer A. The target live broadcast stream b obtained by viewer B giving gift b is sent to the viewer terminal corresponding to viewer B. Through the above processing method, different viewers can watch the background switching effect presented by their gifts, and different viewers do not affect each other's viewing effect.

[0068] In an optional embodiment, if Figure 5 As shown, step S206 may include: Step S500: performing cutout processing on the video frame to be replaced with the background in the original live stream in an asynchronous manner, so as to separate the anchor image from the video frame to be replaced with the background.

[0069] Step S502: GPU acceleration technology is used to perform image synthesis processing on the live broadcast room background image and the separated anchor image to obtain the target live broadcast stream.

[0070] Exemplarily, a character matting model based on a convolutional neural network (CNN) can be used to perform matting processing on video frames to separate the host image from the video frame to be replaced by the background. The character matting model uses a pre-trained deep learning model, such as DeepLab, U-Net or MODNet (Mobile Portrait Matting). These models can achieve high-precision foreground segmentation in character matting tasks while taking into account the computing speed, and are suitable for live broadcast applications with high real-time requirements. In addition, since the live broadcast scene has real-time requirements, the model will also select optimized lightweight model versions, which can reduce computing resource consumption without significantly sacrificing matting accuracy, such as MobileNet V3.

[0071] In this embodiment, when performing image synthesis processing, GPU acceleration technology is used to hand over the image segmentation task to the GPU to achieve faster computing speed to meet high concurrency real-time requirements, and the GPU can quickly process screen rendering through parallel computing, including dynamic background rendering and background switching effect processing, so that the system can complete synthesis within milliseconds.

[0072] In this embodiment, multithreading is used to assign image synthesis and matting tasks to different processing threads, so as to realize parallel asynchronous execution of multiple tasks, thereby reducing the overall processing delay. In addition, GPU acceleration technology (such as OpenGL or Vulkan) is used to speed up the speed of background rendering and image synthesis, thereby ensuring the real-time performance of the subsequently generated video stream.

[0073] In an exemplary embodiment, in order to reduce the overall amount of calculation each time the background is switched, the host image and the background image may be rendered into different layers respectively and superimposed in real time during the final synthesis.

[0074] In an optional embodiment, if Figure 6 As shown, step S500 may include: Step S600: Use a pre-trained face recognition model to perform face recognition on the video frame to be replaced with the background in the original live stream, so as to identify the position of the anchor's head portrait from the video frame to be replaced with the background.

[0075] Step S602: performing a cutout process on the video frame to be replaced with the background according to the position of the anchor portrait, so as to separate the anchor image from the video frame to be replaced with the background.

[0076] Exemplarily, the position of the host's avatar in the video frame is identified through a face recognition model, and then the cutout range can be limited according to the position of the host's avatar. Then, when performing the cutout process, only the area where the host's avatar is located can be cutout.

[0077] In this embodiment, by only performing cutout processing on the anchor's head portrait position, the image area that needs to be processed is reduced, and the cutout accuracy and efficiency are also improved.

[0078] In an optional embodiment, the method further comprises: The target live stream is transcoded to obtain live streams with multiple resolutions.

[0079] In this embodiment, in order to adapt to audience terminals under different network conditions, after obtaining the target live stream, the target live stream will be transcoded, thereby realizing video encoding and compression processing of the target live stream, so that live streams of multiple resolutions can be generated. In this way, the server can push live streams of corresponding resolutions according to the network status of the audience terminal, thereby ensuring that the video stream remains high-definition and smooth when output to the audience end.

[0080] In an optional embodiment, the method further comprises: The target live broadcast stream is pushed to the audience terminal in the live broadcast room, so that the audience terminal can decode and play the target live broadcast stream.

[0081] In this embodiment, after generating the target live stream, the server will actively push the target live stream to the audience terminal in the live broadcast room, so that the audience terminal can decode and play the target live stream. As a result, the audience can promptly and intuitively feel the impact of gift-giving behavior on the background of the live broadcast room, thereby enhancing the interactive experience.

[0082] In another implementation, the viewer terminal may actively obtain the target live stream from the server by pulling the stream, and then decode and play the pulled target live stream.

[0083] Embodiment 2 Figure 7 The block diagram of the live streaming processing device 700 according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 7 As shown, the apparatus 700 may include an acquisition module 701, a first determination module 702, a second determination module 703, and a replacement module 704, wherein: The acquisition module 701 is used to obtain the gift information given by the user in the live broadcast room; A first determination module 702 is used to determine the background effect of the live broadcast room according to the gift information, wherein the background effect includes the live broadcast room background image to be replaced and the duration of the live broadcast room background image; The second determination module 703 is used to determine the video frame to be replaced with the background in the original live stream corresponding to the live broadcast room based on the duration and the gift giving time in the gift information; The replacement module 704 is used to perform background replacement processing on the video frame to be replaced with the background in the original live stream according to the live broadcast room background image to obtain a target live stream.

[0084] As an optional embodiment, determining the background effect of the live broadcast room according to the gift information includes: Perform gift category identification on the gift information to obtain category information of the gift information; The background effect of the live broadcast room is determined based on the category information.

[0085] As an optional embodiment, the gift information includes type information, quantity information and value information of the virtual gift, and the gift category identification of the gift information to obtain the category information of the gift information includes: Performing a first identification process on the type information in the gift information to obtain a first category of the gift information; Performing a second identification process on the quantity information in the gift information to obtain a second category of the gift information; Performing a third identification process on the value information in the gift information to obtain a third category of the gift information; Category information of the gift information is determined based on the first category, the second category, and the third category.

[0086] As an optional embodiment, performing background replacement processing on the video frame to be replaced with the background in the live stream according to the live broadcast room background image to obtain the target live stream includes: Performing a cutout process on the video frame to be replaced with the background in the original live stream in an asynchronous manner, so as to separate the anchor image from the video frame to be replaced with the background; GPU acceleration technology is used to perform image synthesis processing on the live broadcast room background image and the separated anchor image to obtain the target live broadcast stream.

[0087] As an optional embodiment, the performing a cutout process on the video frame to be replaced with the background in the original live stream to separate the anchor image from the video frame to be replaced with the background includes: Using a pre-trained face recognition model to perform face recognition on the video frame to be replaced with the background in the original live stream, so as to identify the position of the anchor avatar from the video frame to be replaced with the background; The video frame to be replaced with the background is subjected to cutout processing according to the position of the anchor portrait, so as to separate the anchor image from the video frame to be replaced with the background.

[0088] As an optional embodiment, the device 700 further includes a transcoding module.

[0089] The transcoding module is used to transcode the target live stream to obtain live streams with multiple resolutions.

[0090] As an optional embodiment, the device 700 further includes a push module.

[0091] The push module is used to push the target live stream to the audience terminal in the live broadcast room so that the audience terminal can decode and play the target live stream.

[0092] Embodiment 3 Figure 8The hardware architecture diagram of a computer device 800 suitable for implementing a live stream processing method according to the third embodiment of the present application is schematically shown. In some embodiments, the computer device 800 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 800 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Figure 8 As shown, the computer device 800 includes but is not limited to: a memory 801, a processor 802, and a network interface 803 that can communicate with each other through a system bus. Among them: The memory 801 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 801 can be an internal storage module of the computer device 800, such as a hard disk or memory of the computer device 800. In other embodiments, the memory 801 can also be an external storage device of the computer device 800, such as a plug-in hard disk equipped on the computer device 800, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 801 can also include both the internal storage module of the computer device 800 and its external storage device. In this embodiment, the memory 801 is generally used to store the operating system and various application software installed on the computer device 800, such as the program code of the live streaming processing method, etc. In addition, the memory 801 can also be used to temporarily store various types of data that have been output or are to be output.

[0093] In some embodiments, the processor 802 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 802 is generally used to control the overall operation of the computer device 800, such as performing control and processing related to data interaction or communication with the computer device 800. In this embodiment, the processor 802 is used to run the program code stored in the memory 801 or process data.

[0094] The network interface 803 may include a wireless network interface or a wired network interface, and the network interface 803 is generally used to establish a communication link between the computer device 800 and other computer devices. For example, the network interface 803 is used to connect the computer device 800 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 800 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0095] It should be pointed out that Figure 8 Only a computer device having components 801 - 803 is shown, but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0096] In this embodiment, the live stream processing method stored in the memory 801 can also be divided into one or more program modules and executed by one or more processors (such as processor 802) to complete the embodiment of the present application.

[0097] Embodiment 4 An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the live stream processing method in the embodiment are implemented.

[0098] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the computer-readable storage medium may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the live stream processing method in the embodiment. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.

[0099] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0100] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0101] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A live stream processing method, characterized in that: The method comprises: Get information about gifts given by users in the live broadcast room; Determine the background effect of the live broadcast room according to the gift information, wherein the background effect includes the live broadcast room background image to be replaced and the duration of the live broadcast room background image; Determine the video frame to be replaced with the background in the original live stream corresponding to the live broadcast room based on the duration and the gift giving time in the gift information; Background replacement processing is performed on the video frame to be replaced with the background in the original live stream according to the live broadcast room background image to obtain a target live stream.

2. The method according to claim 1, characterized in that Determining the background effect of the live broadcast room according to the gift information includes: Perform gift category identification on the gift information to obtain category information of the gift information; The background effect of the live broadcast room is determined based on the category information.

3. The method according to claim 2, characterized in that The gift information includes type information, quantity information and value information of the virtual gift, and the gift category identification of the gift information to obtain the category information of the gift information includes: Performing a first identification process on the type information in the gift information to obtain a first category of the gift information; Performing a second identification process on the quantity information in the gift information to obtain a second category of the gift information; Performing a third identification process on the value information in the gift information to obtain a third category of the gift information; Category information of the gift information is determined based on the first category, the second category, and the third category.

4. The method according to claim 1, characterized in that The performing background replacement processing on the video frame to be replaced with the background in the live stream according to the live broadcast room background image to obtain the target live stream comprises: Performing a cutout process on the video frame to be replaced with the background in the original live stream in an asynchronous manner, so as to separate the anchor image from the video frame to be replaced with the background; GPU acceleration technology is used to perform image synthesis processing on the live broadcast room background image and the separated anchor image to obtain the target live broadcast stream.

5. The method according to claim 4, characterized in that The performing a cutout process on the video frame to be replaced with the background in the original live stream to separate the anchor image from the video frame to be replaced with the background includes: Using a pre-trained face recognition model to perform face recognition on the video frame to be replaced with the background in the original live stream, so as to identify the position of the anchor avatar from the video frame to be replaced with the background; The video frame to be replaced with the background is subjected to cutout processing according to the position of the anchor portrait, so as to separate the anchor image from the video frame to be replaced with the background.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The target live stream is transcoded to obtain live streams with multiple resolutions.

7. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The target live broadcast stream is pushed to the audience terminal in the live broadcast room, so that the audience terminal can decode and play the target live broadcast stream.

8. A live stream processing device, characterized in that: The device comprises: The acquisition module is used to obtain the gift information given by the user in the live broadcast room; A first determination module is used to determine the background effect of the live broadcast room according to the gift information, wherein the background effect includes a live broadcast room background image to be replaced and a duration of the live broadcast room background image; A second determination module is used to determine the video frame to be replaced with the background in the original live stream corresponding to the live broadcast room based on the duration and the gift giving time in the gift information; The replacement module is used to perform background replacement processing on the video frame to be replaced with the background in the original live stream according to the live broadcast room background image to obtain the target live stream.

9. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 7 are implemented.

Images