Virtual gift special effects playback method and its device, equipment, and medium
Through hardware decoding video processing and interpolation sampling technology, the color support and compatibility problems of virtual gift animation special effects in the live broadcast platform are solved, and efficient and smooth gift special effects display is achieved, improving the user experience and live broadcast atmosphere.
Patent Information
- Application Number
- CN202310193850.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-02
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-03-02
AI Technical Summary
In the existing live broadcast platform, virtual gift animation special effects technology has limited color support, poor compatibility and high performance occupancy, resulting in poor display of gift animation special effects and affecting the user experience.
Through hardware decoding video processing, the decoded image frames of the gift special effect video are obtained, the gift color texture images, transparent texture images and custom mask texture images are cropped and merged, and the image resolution and size are adjusted using interpolation sampling rules to generate transparent special effect images. And combine custom images for merge display.
It improves the display effect and fluency of gift effects, saves storage space and device performance, and improves the operation stability and user experience of live broadcast applications.
Smart Images

Figure CN116193151B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of online live broadcasting, and in particular to a method for playing virtual gift special effects, and also to corresponding devices, equipment and non-volatile storage media of the method. Background Art
[0002] In the existing live broadcast room, audience users can interact with the anchor user by giving virtual gifts to the live broadcast, and some virtual gifts will display corresponding gift animation effects in the live broadcast room interface after being given. Rich and gorgeous gift animation effects can effectively mobilize the live broadcast atmosphere of the live broadcast room. Conventional gift animation effects technologies are Gif, Apng or Lottie. Among them, Gif has the advantages of high compatibility and low storage space occupation, but it only supports up to 256 colors. The gift animation effects with a low number of colors affect the user's perception. Correspondingly, Apng supports a wider and clearer color range than Gif, but its compatibility is poor, which limits the development and design of gift animation effects. Although Lottie has a high vector animation compression rate, it does not support complex particle effects, cannot display exquisite gift animation effects, and requires a lot of performance for processing, affecting the running stability of the live broadcast application. When the gift animation effects are displayed, it is easy to cause the live broadcast room interface to freeze, affecting the user's live broadcast experience.
[0003] In view of the problems existing in the virtual gift animation effects in the existing live broadcast platform, the applicant has made corresponding explorations in order to solve the problem. Summary of the invention
[0004] The purpose of this application is to provide a method for playing special effects of virtual gifts to meet user needs. In addition, it also relates to corresponding devices, equipment, non-volatile storage media and computer program products of the method.
[0005] In order to achieve the purpose of this application, the following technical solutions are adopted:
[0006] A method for playing virtual gift special effects proposed for the purpose of this application includes the following steps:
[0007] In response to a display instruction of a target gift special effect, a custom image and a gift special effect video corresponding to the target gift special effect are obtained, and the gift special effect video is decoded to obtain a plurality of decoded image frames, wherein the decoded image frames include a gift color texture image, a gift transparent texture image, and a custom mask texture image;
[0008] Cut out the gift color texture image, gift transparent texture image and custom mask texture image contained in the current decoded image frame;
[0009] According to the resolution ratio between the gift transparent texture image and the gift color texture image, calling the corresponding image interpolation sampling rule, sampling the gift transparent texture image, and modifying the resolution and size of the gift transparent texture image to be the same as those of the gift color texture image;
[0010] The gift transparent texture image and the gift color texture image are merged to generate a gift transparent special effects image, and the custom image and the custom mask texture image are merged to generate a custom composite image, and the custom composite image is merged into the gift transparent special effects image to generate the current gift special effects frame image, and the current gift special effects frame image is output and displayed on the live broadcast room interface.
[0011] In a further embodiment, the step of responding to the display instruction of the target gift special effect, obtaining the custom image and gift special effect video corresponding to the target gift special effect, and decoding the gift special effect video to obtain multiple decoded image frames includes the following steps:
[0012] In response to a gift selection and giving event of a virtual gift control in a live broadcast room interface, a target gift special effect corresponding to the event is determined, and a display instruction of a gift logo including the target gift special effect is triggered;
[0013] Query whether there is a target gift special effect video corresponding to the target gift identifier in the gift special effect library. If so, obtain the target gift video from the gift special effect library. If not, obtain the target gift special effect video from the server.
[0014] Obtain a user avatar image corresponding to the current user terminal, and use the user avatar image as a custom image corresponding to the target gift special effect;
[0015] Call the hardware decoding interface to decode the target gift special effects video, obtain multiple decoded image frames contained in the target gift special effects video, and sort the decoded image frames according to the timeline of the target gift special effects video.
[0016] In a further embodiment, the step of cutting out the gift color texture image, the gift transparent texture image and the custom mask texture image contained in the current decoded image frame comprises the following steps:
[0017] Acquire the texture area coordinate information of the target gift special effect, wherein the texture area coordinate information includes the area coordinates corresponding to the gift color texture image, the gift transparent texture image and the custom mask texture image;
[0018] According to the texture area coordinate information, the image areas of the gift color texture image, the gift transparent texture image and the custom mask texture image contained in the current decoded image frame are determined, and the texture images possessed by each of the image areas are cropped;
[0019] Each of the texture images is corresponded as the gift color texture image, the gift transparent texture image and the custom mask texture image of the current decoded image frame.
[0020] In a further embodiment, the step of calling the corresponding image interpolation sampling rule according to the resolution ratio between the gift transparent texture image and the gift color texture image, sampling the gift transparent texture image, and modifying the resolution and size of the gift transparent texture image to be the same as those of the gift color texture image, includes the following steps:
[0021] Determine a resolution ratio between the gift transparent texture image and the gift color texture image;
[0022] Check whether the resolution ratio exceeds a preset ratio. If not, sample the gift transparent texture image according to a linear interpolation sampling rule, and modify the resolution and size of the gift transparent texture image to be the same as those of the gift color texture image.
[0023] When the resolution ratio exceeds a preset ratio, the gift transparent texture image is sampled according to the bicubic interpolation sampling rule, and the resolution and size of the gift transparent texture image are modified to be the same as those of the gift color texture image.
[0024] In a further embodiment, the step of sampling the gift transparent texture image according to the bicubic interpolation sampling rule and modifying the resolution and size of the gift transparent texture image to be the same as those of the gift color texture image includes the following steps:
[0025] Modify the image size of the gift transparent texture image to the image size of the gift color texture image;
[0026] Obtain four neighboring pixels in the previous row of the current interpolation calculation pixel in the gift transparent texture image, call the bicubic interpolation algorithm to calculate each of the neighboring pixels, and calculate the previous row interpolation;
[0027] Obtain four neighboring pixel points in the next row of the current interpolation calculation pixel point in the gift transparent texture image, call the bicubic interpolation algorithm to calculate each of the neighboring pixel points, and calculate the next row of interpolation;
[0028] A linear interpolation algorithm is called to calculate the previous row interpolation and the next row interpolation to calculate the interpolation corresponding to the current interpolation calculation pixel point.
[0029] In a further embodiment, the step of merging the custom composite image into the gift transparent special effect image to generate a current gift special effect frame image, and outputting and displaying the current gift special effect frame image to the live broadcast room interface includes the following steps:
[0030] Merging the custom composite image into the gift transparent special effect image to generate a current gift special effect frame image;
[0031] Determine the gift display position of the target gift special effect in the current live broadcast room interface;
[0032] The image size of the current gift special effects frame image is adjusted according to the interface size of the current live broadcast room interface, and the current gift special effects frame image is output to the gift display position for display.
[0033] In a further embodiment, the step of responding to the display instruction of the target gift special effect and obtaining the custom image and gift special effect video corresponding to the target gift special effect includes the following steps:
[0034] In response to a gift selection and giving event acting on a virtual gift control in a live studio interface, a target gift special effect corresponding to the event is determined, and a custom image selection control is displayed in the live studio interface;
[0035] In response to an image selection completion event acting on the custom image selection control, obtaining a custom image corresponding to the event;
[0036] Trigger the display instruction of the gift logo containing the target gift special effect.
[0037] A virtual gift special effects playback device proposed for the purpose of this application includes:
[0038] A gift video decoding module is used to respond to a display instruction of a target gift special effect, obtain a custom image and a gift special effect video corresponding to the target gift special effect, and decode the gift special effect video to obtain a plurality of decoded image frames, wherein the decoded image frames include a gift color texture image, a gift transparent texture image, and a custom mask texture image;
[0039] The texture image cropping module is used to crop the gift color texture image, gift transparent texture image and custom mask texture image contained in the current decoded image frame;
[0040] An image interpolation sampling module, used to call a corresponding image interpolation sampling rule according to the resolution ratio between the gift transparent texture image and the gift color texture image, sample the gift transparent texture image, and modify the resolution and size of the gift transparent texture image to be the same as those of the gift color texture image;
[0041] An image merging output module is used to merge the gift transparent texture image and the gift color texture image to generate a gift transparent special effects image, and to merge the custom image and the custom mask texture image to generate a custom composite image, and to merge the custom composite image into the gift transparent special effects image to generate a current gift special effects frame image, and to output the current gift special effects frame image and display it on the live broadcast room interface.
[0042] In a further embodiment, the gift video decoding module includes:
[0043] A display instruction response submodule is used to respond to a gift selection and giving event of a virtual gift control in a live broadcast room interface, determine a target gift special effect corresponding to the event, and trigger a display instruction of a gift logo containing the target gift special effect;
[0044] A gift video acquisition submodule is used to query whether there is a target gift special effect video corresponding to the target gift identifier in the gift special effect library. If so, the target gift video is acquired from the gift special effect library; if not, the target gift special effect video is acquired from the server;
[0045] A custom image acquisition submodule is used to acquire a user avatar image corresponding to the current user terminal, and use the user avatar image as a custom image corresponding to the target gift special effect;
[0046] The gift video decoding submodule is used to call the hardware decoding interface to decode the target gift special effects video, obtain multiple decoded image frames contained in the target gift special effects video, and sort each of the decoded image frames according to the timeline of the target gift special effects video.
[0047] In a further embodiment, the texture image cropping module includes:
[0048] The area coordinate information acquisition submodule is used to obtain the texture area coordinate information of the target gift special effect, wherein the texture area coordinate information includes the area coordinates corresponding to the gift color texture image, the gift transparent texture image and the custom mask texture image;
[0049] A texture image cropping submodule is used to determine the image areas of the gift color texture image, the gift transparent texture image and the custom mask texture image contained in the current decoded image frame according to the texture area coordinate information, and crop the texture images of each of the image areas;
[0050] The texture image allocation submodule is used to correspond each of the texture images as the gift color texture image, the gift transparent texture image and the custom mask texture image of the current decoded image frame.
[0051] In a further embodiment, the image interpolation sampling module includes:
[0052] A resolution ratio determination submodule, used to determine the resolution ratio between the gift transparent texture image and the gift color texture image;
[0053] A linear interpolation sampling submodule, used to check whether the resolution ratio exceeds a preset ratio. If not, the gift transparent texture image is sampled according to a linear interpolation sampling rule, and the resolution and size of the gift transparent texture image are modified to be the same as those of the gift color texture image;
[0054] The bicubic interpolation sampling submodule is used to sample the gift transparent texture image according to the bicubic interpolation sampling rule when the resolution ratio exceeds the preset ratio, and modify the resolution and size of the gift transparent texture image to be the same as those of the gift color texture image.
[0055] In a further embodiment, the image merging and outputting module comprises:
[0056] A special effect frame image generation submodule, used for merging the custom composite image into the gift transparent special effect image to generate a current gift special effect frame image;
[0057] A display position determination submodule is used to determine the gift display position of the target gift special effect in the current live broadcast room interface;
[0058] The special effect frame image display submodule is used to adjust the image size of the current gift special effect frame image according to the interface size of the current live broadcast room interface, and output the current gift special effect frame image to the gift display position for display.
[0059] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the above-mentioned virtual gift special effects playback method.
[0060] In order to solve the above technical problems, an embodiment of the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the above-mentioned virtual gift special effects playback method.
[0061] In order to solve the above technical problems, an embodiment of the present application also provides a computer program product, including a computer program and computer instructions. When the computer program and computer instructions are executed by a processor, the processor executes the steps of the above-mentioned virtual gift special effects playback method.
[0062] Compared with the prior art, the advantages of this application are as follows:
[0063] The present application is based on hardware decoding video processing for the display of gift special effects in the live broadcast room. The gift special effects video of the virtual gift is decoded to obtain the decoded image frame for cropping. The gift color texture image, gift transparent texture image and custom mask texture image contained in the decoded image frame are cropped in sequence according to the timeline of the gift image video. The gift special effects frame image is combined with the custom image associated with the live broadcast business to generate a gift special effects frame image for display. Each gift special effects frame special effect displays the transparent gift special effects of the gift in the live broadcast room interface in the form of transparent special effects, and has the characteristics of custom images to distinguish different gift special effects in the live broadcast room, enriching the transparent gift special effects in the live broadcast. The display effects in the live broadcast interface. For example, when the custom image is the avatar of the user who sends a virtual gift, the user who sends the gift can be informed of the displayed gift special effects to enhance the user's experience of using the virtual gift giving service. Compared with traditional gift animation special effects technologies such as Gif, Apng or Lottie, the gift special effects video based on hardware decoding video processing has a higher compression rate, which can save the storage space of the device, and the gift special effects played by decoding with the help of device hardware have higher processing efficiency, which improves the running stability of the live broadcast application, and quickly plays smooth gift special effects for users in the live broadcast room to watch, effectively enhancing the live broadcast atmosphere of the live broadcast room.
[0064] Secondly, the present application combines the mask texture image acting on the custom image into the gift special effects video to save the storage space occupied by the gift special effects, and uses the interpolation sampling rule to repair the image size and resolution of the gift transparent texture image, so that the gift transparent texture image can match the gift color texture image in size and resolution. The interpolation sampling rule can dynamically adjust the interpolation sampling algorithm acting on the gift transparent texture image, and select the corresponding interpolation sampling algorithm for calculation based on the resolution difference between the gift transparent texture image and the gift color texture image, and adjust between saving device computing power and improving the image display effect after interpolation sampling. At the same time, the existing bicubic interpolation algorithm is optimized. Compared with the traditional bicubic interpolation calculation based on the pixel values of 16 original pixels adjacent to the pixel point, the present application is based on 8 adjacent original pixels to improve the interpolation calculation efficiency of the bicubic interpolation calculation and save occupancy performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0066] Figure 1 A schematic diagram of a typical network deployment architecture related to implementing the technical solution of this application;
[0067] Figure 2 This is a flowchart of a typical embodiment of the method for playing special effects of virtual gifts of the present application;
[0068] Figure 3 It is a schematic diagram of a decoded image frame having a gift color texture image, a gift transparent texture image and a custom mask texture image in this application;
[0069] Figure 4 A schematic diagram of merging a gift color texture image and a gift transparent texture image to generate a gift transparent special effect image in this application;
[0070] Figure 5 A schematic diagram of merging a custom image with a custom mask texture image to generate a custom composite image in this application;
[0071] Figure 6 A schematic diagram of synthesizing a custom composite image into a gift image frame to generate a gift special effect image in this application;
[0072] Figure 7 A flowchart of a specific implementation method for triggering display instructions and obtaining gift special effects videos and custom images in this application;
[0073] Figure 8 A schematic diagram of the live broadcast room interface for outputting virtual gift controls displayed in this application;
[0074] Fig. 9 A flowchart diagram of a specific implementation method for cutting out a gift element texture image, a gift transparent texture image and a custom mask texture image in an interface image frame in the present application;
[0075] Fig.10 A flowchart of a specific implementation method of sampling a gift transparent texture image according to an image interpolation sampling rule in this application;
[0076] Fig.11 A schematic diagram of the positional relationship between an interpolation-calculated pixel point and its neighboring pixel points when a bicubic interpolation algorithm is used in this application;
[0077] Fig.12 A flowchart of a specific implementation method of performing interpolation sampling on a gift transparent texture image according to a bicubic interpolation sampling rule in this application;
[0078] Fig.13 This is a schematic diagram of a live broadcast room interface for outputting and displaying gift special effect images in this application;
[0079] Fig.14 A flowchart diagram of a specific implementation method of synthesizing a custom composite image into a gift transparent special effect image to generate a gift special effect frame image for output display in this application;
[0080] Fig.15 A schematic diagram of a live studio interface for outputting and displaying a custom image selection control in this application;
[0081] Fig.16 A flowchart diagram of a specific implementation method for obtaining a custom image in response to an image selection completion event in this application;
[0082] Fig.17 This is a principle block diagram of a typical embodiment of a virtual gift special effects playback device of the present application;
[0083] Fig.18 This is a basic structural block diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION
[0084] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.
[0085] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0086] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.
[0087] It will be understood by those skilled in the art that the "user end", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; traditional laptop and / or palmtop computers or other devices, which have and / or include traditional laptop and / or palmtop computers or other devices with and / or including radio frequency receivers. The "user end", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to operate locally, and / or in a distributed form, at any other location on the earth and / or in space. The "user end", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.
[0088] The hardware referred to by the names such as "server", "user terminal", "work node" etc. in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.
[0089] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.
[0090] See also Figure 1 , the hardware foundation required for the implementation of the relevant technical solutions of this application can be deployed according to the architecture shown in the figure. The server 80 referred to in this application is deployed in the cloud. As an online server, it can be responsible for further connecting relevant data servers and other servers that provide relevant support, etc., so as to form a logically related service cluster to provide services for relevant terminal devices such as the smartphone 81 and personal computer 82 shown in the figure or a third-party server (not shown). The smartphone and personal computer can access the Internet through a well-known network access method and establish a data communication link with the server 80 in the cloud to run terminal applications related to the services provided by the server.
[0091] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions suitable for running on the server in this application can be implemented in the server in this way.
[0092] The application described herein refers to an application running on a server or terminal device. This application implements the relevant technical solutions of the present application in a programming manner. Its program code can be stored in a non-volatile storage medium recognizable by a computer in the form of computer executable instructions, and can be loaded into the memory by the central processing unit for execution. The relevant device of the present application is constructed by the operation of the application on the computer.
[0093] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions suitable for running on the server in this application can be implemented in the server in this way.
[0094] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.
[0095] See also Figure 2 , a virtual gift special effects playback method of the present application, in its typical embodiment, comprises the following steps:
[0096] Step S11, responding to the display instruction of the target gift special effect, obtaining the custom image and gift special effect video corresponding to the target gift special effect, decoding the gift special effect video to obtain multiple decoded image frames, the decoded image frames include a gift color texture image, a gift transparent texture image and a custom mask texture image:
[0097] The display instruction is triggered by the audience user in the live broadcast room or pushed by the business server of the live broadcast platform. If the user currently at the user end is a viewer user who gives a virtual gift to the anchor user in the live broadcast room, the user end determines the target gift special effect corresponding to the event by responding to the gift selection and giving event of the virtual gift control in the live broadcast room interface, so as to generate a display instruction including the gift logo of the target gift special effect. If the user currently at the user end is the anchor user or audience user in the live broadcast room, the display instruction is broadcast and pushed by the business server of the live broadcast platform to the user ends of each of the audience users and the anchor user.
[0098] The display instruction includes a gift identifier corresponding to the target gift special effect, so that the user terminal can query the gift special effect video corresponding to the gift identifier of the target gift special effect from the gift special effects library. If the gift special effects video exists in the gift special effects library, the target gift video is obtained from the gift special effects library. If it does not exist, the user terminal will push a video acquisition instruction containing the gift identifier to the server to obtain the gift special effects video pushed by the server.
[0099] In addition to obtaining the gift special effect video corresponding to the target gift special effect, the custom image acting on the target gift special effect will also be obtained. The custom image can generally be an avatar image of the audience user who gives the virtual gift to which the target gift special effect belongs, a nickname image representing the user nickname of the audience user, or a user level image representing the user level of the audience user, etc. Of course, the custom image can also be a live activity image corresponding to the live activity opened in the live broadcast room where the audience user is located. For example, when the live broadcast room opens a theme live broadcast activity related to festivals such as National Day, Mid-Autumn Festival or Spring Festival, the custom image can be a live activity image representing one of these festivals. It can be understood that the custom image is generally an image associated with the live online service of the platform. Those skilled in the art can flexibly design the specific image content of the custom image, which will not be elaborated here.
[0100] The gift special effects video includes a plurality of decoded image frames arranged in a timeline, and the gift special effects video will be decoded through a hardware decoding interface to obtain the decoded image frames contained in the gift special effects video. Specifically, the gift special effects video will include image frames encoded in RGBA color coding or YUV color coding when it is not decoded, wherein RGB in the RGBA color coding refers to the three optical primary colors, which are the three basic colors in color that cannot be decomposed any further, and the three optical primary colors refer to red (R), green (G), and blue (B), and RGB can form any color. In image display, a picture can be decomposed into countless pixels, and this pixel can be represented by RGB, and A in RGBA represents the transparency Alpha; Y in the YUV color coding refers to the grayscale component or brightness, and UV refers to the chromaticity component, wherein the chromaticity defines the hue and saturation of the color, and the YUV color coding can be encoded in 4:4:4, 4:2:2 or 4:2:0 sampling formats.
[0101] The decoded image frames are generally encoded in RGBA color coding, and each of the decoded image frames contains a gift color texture image, a gift transparent texture image and a custom mask texture image. If the gift special effects video encodes each image frame in YUV color coding, each of the image frames is converted from YUV color coding to an encoding format in RGBA color coding, so that each of the decoded image frames decoded from the gift special effects video contains the gift color texture image, the gift transparent texture image, and the custom mask texture image, so as to facilitate subsequent cropping and merging. Specifically, when the decoded image frame is a luminance and chrominance image frame encoded in YUV color, based on the original image size of the luminance and chrominance image frame, the luminance component and chrominance component contained therein are extracted from the luminance and chrominance image frame, so as to convert the luminance and chrominance image frame into the decoded image frame containing the gift color texture image, the gift transparent texture image and the custom mask texture image based on the luminance component and the chrominance component according to the image conversion algorithm.
[0102] The hardware decoding interface refers to the interface provided by the method for decoding the gift special effects video based on the device hardware (CPU or GPU). Correspondingly, the gift special effects video is generally an MP4 format video. It can be understood that the hardware decoding generally refers to MP4 hardware decoding. Compared with gift animation special effects technologies such as Gif, Apng or Lottie, the gift special effects video corresponding to MP4 hardware decoding has a higher compression rate and can save the storage space of the device, and decoding and playing gift special effects with the help of device hardware has a higher processing efficiency, but MP4 hardware decoding usually discards the Alpha transparent channel in the gift special effects video. Therefore, each decoded image frame in the gift special effects video has the gift color texture image, the gift transparent texture image and the custom mask texture image, wherein the gift color texture image is an image showing the RGB channels in the RGBA encoding information, and the gift transparent texture image and the custom mask texture image are images showing the Alpha channel in the RGBA encoding information. For details, please refer to Figure 3 , Figure 3 The image frame shown in is the video image frame, wherein the image frame includes the gift color texture image (RGB) 301, the gift transparent texture image (Alpha) 302 and the custom mask texture image (Mask) 303. It can be seen that the image areas of the gift transparent texture image (Alpha) 302 and the custom mask texture image (Mask) 303 are generally the same, so as to facilitate subsequent image cropping.
[0103] Step S12, cutting out the gift color texture image, gift transparent texture image and custom mask texture image contained in the current decoded image frame:
[0104] After decoding any of the decoded image frames from the gift special effects video, the gift color texture image, the gift transparent texture image and the custom mask texture image contained in the decoded image frame will be cropped out.
[0105] The current decoded image frame refers to the decoded image frame that is being cropped and merged by the current thread.
[0106] Generally, by obtaining the texture area coordinate information corresponding to the gift special effects video, the image areas where the gift color texture image, gift transparent texture image and custom mask texture image contained in each decoded image frame in the gift special effects video are located are determined for texture image cropping. Specifically, the texture area coordinate information includes the area coordinates corresponding to the gift color texture image, gift transparent texture image and custom mask texture image in each decoded image frame of the corresponding gift special effects video, and then the image areas of the gift color texture image, gift transparent texture image and custom mask texture image in the decoded image frame are determined, and the texture images in each of these image areas are cropped out respectively, so that each texture image corresponds to the gift color texture image, the gift transparent texture image and the custom mask texture image of the current decoded image frame.
[0107] In one embodiment, a gift color texture image area, a gift transparent texture image area and a custom mask texture image area in the decoded image frame can be identified by using an image recognition model based on a neural network and trained to convergence, and then the texture images of the identified gift color texture image area, gift transparent texture image area and custom mask texture image area can be cropped out respectively, so that each of the texture images can correspond to the gift color texture image, the gift transparent texture image and the custom mask texture image of the current decoded image frame.
[0108] Step S13, according to the resolution ratio between the gift transparent texture image and the gift color texture image, call the corresponding image interpolation sampling rule, sample the gift transparent texture image, and modify the resolution and size of the gift transparent texture image to be the same as those of the gift color texture image:
[0109] The gift color texture image, gift transparent texture image and custom mask texture image contained in the current decoded image frame are cropped out, and the gift transparent texture image is interpolated and sampled according to the image interpolation sampling rule to modify the resolution and size of the gift transparent texture image so that the gift transparent texture image has the same resolution and size as the gift color texture image, so that when the gift transparent texture image and the gift color texture image are subsequently merged to generate a gift transparent special effects image, the gift transparent texture image and the gift color texture image overlap each other to ensure the display effect of the gift transparent special effects image generated by the merger of the two.
[0110] The image interpolation sampling rules include linear interpolation sampling rules and bicubic interpolation sampling rules, wherein the linear interpolation sampling rules generally refer to the use of the nearest neighbor interpolation algorithm or the bilinear interpolation algorithm to perform interpolation sampling on the gift transparent texture image, and the bicubic interpolation sampling rules generally refer to the use of the bicubic interpolation algorithm to perform interpolation sampling on the gift transparent texture image. Compared with the bicubic interpolation algorithm, the nearest neighbor interpolation algorithm or the bilinear interpolation algorithm does not need to occupy too much performance of the user-side device for interpolation sampling calculations, but the image effect of the gift transparent texture image after interpolation sampling by the bicubic interpolation algorithm is better than that of the nearest neighbor interpolation algorithm or the bilinear interpolation algorithm. Therefore, the image interpolation sampling rules will preset the corresponding resolution ratio verification rules, and select the corresponding interpolation sampling rules to perform interpolation sampling on the gift transparent texture image to ensure the image effect of the gift transparent texture image after interpolation sampling while saving the device performance of the user-side.
[0111] Specifically, the gift color texture image and the gift transparent texture image are cropped from the decoded image frame, and the image resolutions of the gift color texture image and the gift transparent texture image are compared to determine the resolution ratio between the two. When the resolution ratio does not exceed the preset ratio, the nearest neighbor interpolation algorithm or the bilinear interpolation algorithm corresponding to the linear interpolation sampling rule is used to sample the gift transparent texture image, and the resolution and size of the gift transparent texture image are modified to be the same as those of the gift color texture image; if the resolution ratio exceeds the preset ratio, the bilinear interpolation algorithm in the bicubic interpolation sampling rule is used to sample the gift transparent texture image, and the resolution and size of the gift transparent texture image are modified to be the same as those of the gift color texture image.
[0112] In one embodiment, since the computing resources occupied by the existing bicubic interpolation algorithm are too high, in order to reduce the performance occupied by the user end, the present method optimizes the existing bicubic interpolation algorithm. When the existing bicubic interpolation algorithm calculates pixel interpolation, bicubic interpolation is usually performed based on the pixel values of 16 original pixels adjacent to the pixel point, while the present method only performs bicubic interpolation based on the pixel values of 8 original pixels adjacent to the pixel point, so as to improve the interpolation efficiency of bicubic interpolation and save the performance occupied. Specifically, four neighboring pixels in the previous row of the current interpolation calculation pixel point in the gift transparent texture image are obtained, and the bicubic interpolation algorithm is called to calculate each of the neighboring pixels to calculate the previous row interpolation, and four neighboring pixels in the next row of the current interpolation calculation pixel point in the gift transparent texture image are obtained, and the bicubic interpolation algorithm is called to calculate each of the neighboring pixels to calculate the next row interpolation, and then the linear interpolation algorithm is called to calculate the previous row interpolation and the next row interpolation to calculate the interpolation corresponding to the current interpolation calculation pixel point.
[0113] Step S14, merging the gift transparent texture image and the gift color texture image to generate a gift transparent special effect image, and merging the custom image and the custom mask texture image to generate a custom composite image, and merging the custom composite image into the gift transparent special effect image to generate the current gift special effect frame image, and outputting the current gift special effect frame image to display on the live studio interface:
[0114] After the gift transparent texture image is interpolated and adopted through the image interpolation sampling rule, the resolution and size of the gift transparent texture image are modified to be the same as those of the gift color texture image, and then the gift transparent texture image and the gift color texture image are merged to generate the gift transparent special effects image.
[0115] Please refer to Figure 4 ,like Figure 4 As shown, the gift color texture image is as follows Figure 4 The texture image 401 shown in FIG. 4 is a transparent texture image of a gift. Figure 4 The texture image 402 shown in FIG. 4 is combined with the texture image 401 shown in FIG. 402 to generate the texture image 402 shown in FIG. Figure 4 The gift transparent special effect image 403 is shown.
[0116] In addition to generating the gift transparent special effect image, it is also necessary to merge the custom image with the custom mask texture image to generate the custom composite image. For details, please refer to Figure 5 , Figure 5The custom image 501 shown in the figure is the custom image, and the texture image 502 shown in the figure is the custom mask texture image. The custom composite image 503 is generated by merging the custom image 501 and the texture image 502.
[0117] After synthesizing the gift transparent special effect image and the custom synthesized image, synthesize the gift transparent special effect image into the position corresponding to the custom synthesized image to generate the current gift special effect frame image, wherein the synthesized position corresponding to the gift transparent special effect image in the custom synthesized image is generally a preset position, please refer to Figure 6 , Figure 6 The custom composite image 601 shown in the figure and the gift transparent special effect image 602 shown in the figure are synthesized into the corresponding synthesis position in the gift transparent special effect image 602 to generate a gift special effect frame image 603 shown in the figure.
[0118] The user end generates the gift special effects frame images corresponding to the decoded image frames contained in the gift special effects video in sequence according to the timeline of the gift special effects video, and at the same time outputs the gift special effects frame effects in sequence to the live broadcast room interface of the user end according to the timeline for display, so that the live broadcast room interface can fully display the target gift effects of the virtual gifts given by the audience users to the anchor user in the live broadcast room.
[0119] It can be seen from the typical implementation of the method that the method is based on the display of gift special effects in the live broadcast room based on hardware decoding video processing, and the gift special effects video of the virtual gift is used to obtain the decoded image frame for cropping, and the gift color texture image, gift transparent texture image and custom mask texture image contained in the decoded image frame are cropped in sequence according to the timeline of the gift image video, and combined with the custom image associated with the live broadcast business, they are merged to generate a gift special effects frame image for display, and each gift special effects frame special effect displays the transparent gift special effect of the gift in the live broadcast room interface in the form of a transparent special effect display, and has the characteristics of custom images to distinguish different gift special effects in the live broadcast room, enriching the transparent gift The display effect of virtual gift effects in the live broadcast room interface. For example, when the custom image is the avatar of the user who gives a virtual gift, the user who gives the gift effect can be informed of the displayed gift effect to enhance the user's experience of using the virtual gift giving service. Compared with traditional gift animation special effects technologies such as Gif, Apng or Lottie, the gift special effects video based on hardware decoding video processing has a higher compression rate, which can save the storage space of the device, and the gift special effects played by decoding with the help of device hardware have higher processing efficiency, which improves the running stability of the live broadcast application, and quickly plays smooth gift special effects for users in the live broadcast room to watch, effectively enhancing the live broadcast atmosphere of the live broadcast room.
[0120] Secondly, this method combines the mask texture image acting on the custom image into the gift special effects video to save the storage space occupied by the gift special effects, and uses the interpolation sampling rule to repair the image size and resolution of the gift transparent texture image, so that the gift transparent texture image can match the gift color texture image in size and resolution. The interpolation sampling rule can dynamically adjust the interpolation sampling algorithm acting on the gift transparent texture image, and select the corresponding interpolation sampling algorithm based on the resolution difference between the gift transparent texture image and the gift color texture image for calculation, and adjust between saving device computing power and improving the image display effect after interpolation sampling. At the same time, the existing bicubic interpolation algorithm is optimized. Compared with the traditional bicubic interpolation calculation based on the pixel values of 16 original pixels adjacent to the pixel point, this method can be based on 8 adjacent original pixels, so as to improve the interpolation calculation efficiency of the bicubic interpolation calculation and save occupancy performance.
[0121] The above typical embodiments and their variations fully disclose the implementation scheme of the virtual gift special effects playback method of the present application. However, various variations of the method can still be derived by changing and amplifying some technical means. Other embodiments are briefly described as follows:
[0122] In one embodiment, please refer to Figure 7 and Figure 8 The step of responding to the display instruction of the target gift special effect, obtaining the custom image and gift special effect video corresponding to the target gift special effect, and decoding the gift special effect video to obtain multiple decoded image frames includes the following steps:
[0123] Step S111, responding to the gift selection and giving event of the virtual gift control in the live broadcast room interface, determining the target gift special effect corresponding to the event, and triggering a display instruction of the gift logo containing the target gift special effect:
[0124] Please refer to Figure 8 , Figure 8 A plurality of virtual gifts are displayed in the virtual gift control 801 shown. The virtual gift control 801 is displayed in the live broadcast room interface. Users in the live broadcast room can select virtual gifts through the virtual gift control 801 and give them to the anchor user in the live broadcast room. For example, when the user selects gift B through the virtual gift control 801, the gift selection and giving event acting on gift B will be triggered. The user end will respond to the virtual gift selection and giving event and generate the display instruction of the gift identifier including the target gift special effect of gift B.
[0125] Step S112, query whether there is a target gift special effect video corresponding to the target gift identifier in the gift special effect library, if yes, obtain the target gift video from the gift special effect library, if no, obtain the target gift special effect video from the server:
[0126] After the user terminal responds to the display instruction, it will query the gift special effects library whether there is a target gift special effects video corresponding to the target gift identifier contained in the display instruction. The gift special effects library stores multiple mapping relationship data consisting of gift identifiers and gift special effects videos.
[0127] If the target gift special effects video corresponding to the target gift identifier can be queried from the gift special effects library, the target gift special effects video will be obtained; if the target gift special effects video does not exist in the gift special effects library, the user terminal will push a gift special effects acquisition instruction containing the target gift identifier to the server, so that the server will respond to the gift special effects acquisition instruction, query the target gift special effects video corresponding to the target gift identifier from the remote gift special effects library that is kept updated at all times, and push it to the user terminal.
[0128] Step S113, obtaining a user avatar image corresponding to the current user terminal, and using the user avatar image as a custom image corresponding to the target gift special effect:
[0129] In addition to obtaining the gift special effects video, the user terminal will also obtain the user avatar image corresponding to the user account currently logged in to the live broadcast platform, and use the user avatar image as the custom image corresponding to the target gift special effects. The user terminal can obtain the user avatar image through the server or the user terminal's local storage space.
[0130] Step S114, calling the hardware decoding interface to decode the target gift special effects video, obtaining multiple decoded image frames contained in the target gift special effects video, and sorting the decoded image frames according to the time axis of the target gift special effects video:
[0131] After obtaining the target gift special effects video, the user end will call the hardware interface to decode the target gift special effects video to obtain multiple decoded image frames contained in the target gift special effects video, and sort each of the decoded image frames according to the timeline of the target gift special effects video, so as to subsequently crop and merge these decoded image frames in sequence to generate transparent special effects image frames corresponding to each of these decoded image frames for output.
[0132] In this embodiment, users can interact with the anchor in a live broadcast through the virtual gift giving service. In addition to obtaining the gift special effect video corresponding to the given virtual gift and decoding it to generate a gift special effect frame image, the user end will also obtain the user's user avatar image as a custom image and synthesize it into the gift special effect frame image to represent the user who gave the virtual gift of the gift special effect displayed in the live broadcast room.
[0133] In one embodiment, please refer to Fig. 9 The step of cutting out the gift color texture image, the gift transparent texture image and the custom mask texture image contained in the current decoded image frame includes the following steps:
[0134] Step S121, obtaining the texture area coordinate information of the target gift special effect, wherein the texture area coordinate information includes the area coordinates corresponding to the gift color texture image, the gift transparent texture image and the custom mask texture image:
[0135] The texture area coordinate information is generally stored in correspondence with the gift special effects video of the gift special effects to which it belongs. After the user end obtains the gift special effects video, it will generally also obtain the texture area coordinate information corresponding to the gift special effects video, so as to crop the gift color texture image, gift transparent texture image and custom mask texture image contained in each decoded image frame contained in the gift special effects video.
[0136] The area coordinates corresponding to the gift color texture image, gift transparent texture image and custom mask texture image contained in the texture area coordinate information are generally the four plane coordinates of the corresponding image areas where these texture images are located in the decoded image frame.
[0137] Step S122, according to the texture area coordinate information, determine the image areas of the gift color texture image, the gift transparent texture image and the custom mask texture image contained in the current decoded image frame, and crop the texture images of each of the image areas:
[0138] The user end determines the texture areas corresponding to the gift color texture image, the gift transparent texture image and the custom mask texture image from the current decoded image frame decoded from the gift special effects video based on the area coordinates corresponding to the gift color texture image, the gift transparent texture image and the custom mask texture image contained in the texture area coordinate information, so as to crop the texture images in the texture areas from the current decoded image frame.
[0139] Step S123, a texture image allocation submodule is used to correspond each of the texture images as the gift color texture image, the gift transparent texture image and the custom mask texture image of the current decoded image frame:
[0140] After cutting out the texture images at the area coordinates corresponding to the gift color texture image, the gift transparent texture image and the custom mask texture image from the current decoded image frame, each of the texture images will be classified to respectively determine the gift color texture image, the gift transparent texture image and the custom mask texture image corresponding to the current decoded image frame.
[0141] In this embodiment, according to the area coordinates corresponding to the gift color texture image, gift transparent texture image and custom mask texture image contained in the decoded image frame in the gift special effects video, the gift color texture image, gift transparent texture image and custom mask texture image are cropped out from the decoded image frame, and the texture images contained in the decoded image frame can be efficiently cropped without complicated algorithms.
[0142] In one embodiment, please refer to Fig.10 The step of calling the corresponding image interpolation sampling rule according to the resolution ratio between the gift transparent texture image and the gift color texture image, sampling the gift transparent texture image, and modifying the resolution and size of the gift transparent texture image to be the same as those of the gift color texture image, includes the following steps:
[0143] Step S131, determining the resolution ratio between the gift transparent texture image and the gift color texture image:
[0144] The gift transparent texture image and the gift color texture image are cropped out from the decoded image frame, and the resolutions of the gift transparent texture image and the gift color texture image are compared to determine the resolution ratio between the two.
[0145] Step S132, checking whether the resolution ratio exceeds the preset ratio, if not, sampling the gift transparent texture image according to the linear interpolation sampling rule, and modifying the resolution and size of the gift transparent texture image to be the same as the gift color texture image:
[0146] The preset ratio is generally 1:1.5, where 1 refers to the gift transparent texture image and 1.5 refers to the gift preset texture image. When the resolution ratio between the gift transparent texture image and the gift color texture image does not exceed the preset ratio, the gift transparent texture image will be interpolated and sampled according to the linear interpolation sampling rule using the nearest neighbor interpolation algorithm or the bilinear interpolation algorithm, thereby modifying the resolution and size of the gift transparent texture image to be the same as those of the gift color texture image.
[0147] Regarding the preset ratio, those skilled in the art can design it flexibly, and this step will not be described in detail.
[0148] Step S133, when the resolution ratio exceeds the preset ratio, the gift transparent texture image is sampled according to the bicubic interpolation sampling rule, and the resolution and size of the gift transparent texture image are modified to be the same as those of the gift color texture image:
[0149] When the resolution ratio between the gift transparent texture image and the gift color texture image exceeds the preset ratio, the user terminal will use the bicubic interpolation algorithm to interpolate and sample the gift transparent texture image according to the bicubic interpolation sampling rules to modify the resolution and size of the gift transparent texture image to be the same as those of the gift color texture image.
[0150] In this embodiment, a resolution ratio check is set between the gift color texture image and the gift transparent texture image to determine the interpolation sampling rule used, and the interpolation sampling algorithm applied to the gift transparent texture image is dynamically adjusted. When the resolution ratio difference between the two sides is not large, a conventional interpolation sampling algorithm is used to save device computing power. When the resolution ratio difference between the two sides is large, a bicubic interpolation algorithm is used to improve the display effect of the image after interpolation sampling.
[0151] In one embodiment, please refer to Fig.11 and Fig.12 The step of sampling the gift transparent texture image according to the bicubic interpolation sampling rule and modifying the resolution and size of the gift transparent texture image to be the same as those of the gift color texture image includes the following steps:
[0152] Step S1331, modifying the image size of the gift transparent texture image to the image size of the gift color texture image:
[0153] Before using the bicubic interpolation sampling algorithm, the image size of the gift color texture image will be determined, so that the gift transparent texture image can be modified to the image size before interpolation sampling is performed.
[0154] Step S1332, obtain four neighboring pixels in the previous row of the current interpolation calculation pixel in the gift transparent texture image, call the bicubic interpolation algorithm to calculate each of the neighboring pixels, and calculate the previous row interpolation:
[0155] The interpolation calculated pixel points generally refer to new pixel points generated after modifying the gift transparent texture image. The display accuracy of the gift transparent texture image is improved by calculating the interpolation values corresponding to these new pixel points.
[0156] Please refer to Fig.11 , Fig.11 The G pixel point 1101 shown is the current interpolation calculation pixel point in the gift image texture image shown. Fig.11 The pixel points F1, F2, F3 and F4 are four neighboring pixel points in the upper row of the pixel point G 1101. The bicubic interpolation algorithm is called to calculate each of the neighboring pixel points to calculate Fig.11 The g1 pixel shown is used as the upper row interpolation of the G pixel shown.
[0157] The bicubic interpolation algorithm generally refers to the BiCubic function, the mathematical equation of which is as follows:
[0158]
[0159] Among them, "a" is generally taken as -0.5, and the corresponding x is the distance between each of the neighboring pixels and the interpolation pixel, including the distance between the horizontal coordinate and the vertical coordinate between each of the neighboring pixels and the interpolation pixel, that is, in the mathematical equation shown, x is the distance between the horizontal coordinate and the vertical coordinate between the neighboring pixel and the interpolation pixel, and then the horizontal coordinate weight and the vertical coordinate weight between each of the neighboring pixels and the interpolation pixel are calculated, and then the value is obtained. Fig.11 The interpolation of the g1 pixel is shown.
[0160] Step S1333, obtain four neighboring pixels in the next row of the current interpolation calculation pixel in the gift transparent texture image, call the bicubic interpolation algorithm to calculate each of the neighboring pixels, and calculate the next row of interpolation:
[0161] Please refer to Fig.11 , Fig.11 The F5 pixel point, F6 pixel point, F7 pixel point and F8 pixel point shown in the figure are four neighboring pixel points in the next row of the G pixel point 1101 shown in the figure. The bicubic interpolation algorithm is called to calculate each of the neighboring pixel points to calculate Fig.11 The g2 pixel point shown is used as the next row interpolation of the G pixel point shown.
[0162] Step S1334, calling a linear interpolation algorithm to calculate the previous row interpolation and the next row interpolation, and calculating the interpolation corresponding to the current interpolation calculation pixel point:
[0163] After calculating the previous line interpolation value and the next line interpolation value, the linear interpolation algorithm is called to calculate the previous line interpolation value and the next line interpolation value to calculate the interpolation value corresponding to the current interpolation calculation pixel point.
[0164] By analogy, the interpolation of new pixels generated after modifying the image size of the gift transparent texture image is calculated, and then the bicubic interpolation sampling of the gift transparent texture image is completed, so that the display effect of the gift transparent texture image with the same image size and resolution as the gift color texture image is improved.
[0165] In this embodiment, the existing bicubic interpolation algorithm is optimized, and bicubic interpolation calculation is performed based on the pixel values of 8 original pixels adjacent to the pixel point, so as to improve the interpolation calculation efficiency of the bicubic interpolation calculation and save occupancy performance.
[0166] In one embodiment, please refer to Figure 6 , Fig.13 and Fig.14 The step of merging the custom composite image into the gift transparent special effect image to generate the current gift special effect frame image, and outputting the current gift special effect frame image to display on the live broadcast room interface includes the following steps:
[0167] Step S141, merging the custom composite image into the gift transparent special effect image to generate the current gift special effect frame image:
[0168] Please refer to Figure 6 ,Will Figure 6 The custom composite image 601 shown is merged into the gift transparent special effect image 602 shown to generate the current gift special effect frame image 603 shown.
[0169] Step S142, determining the gift display position of the target gift special effect in the current live broadcast room interface:
[0170] The gift display position of the gift special effects in the live broadcast room interface is the display position of the gift special effects frame image in the live broadcast room interface.
[0171] Step S143, adjusting the image size of the current gift special effect frame image according to the interface size of the current live studio interface, and outputting the current gift special effect frame image to the gift display position for display:
[0172] Before the transparent special effects image frame is output to the gift display position for display, the image size of the gift special effects frame image will be proportionally adjusted according to the interface size of the current live broadcast room interface to prevent the gift special effects frame image frame from blocking too large an interface area and affecting the user's interface operation.
[0173] Please refer to Figure 6 and Fig.13 , Fig.13 The current gift special effect frame image 1301 displayed in the live broadcast room interface is Figure 6The current gift special effects frame image 603 is shown.
[0174] In this embodiment, a gift special effect frame image of a gift special effect is generated by merging a custom composite image with a gift transparent special effect image, and the size of the gift special effect frame image is adjusted based on the size of the live broadcast room interface to accommodate user operations, and then output to the gift display position for display of the gift special effect.
[0175] In one embodiment, please refer to Figure 8 , Fig.15 and Fig.16 , the step of responding to the display instruction of the target gift special effect and obtaining the custom image and gift special effect video corresponding to the target gift special effect includes the following steps:
[0176] Step S111', responding to the gift selection and giving event of the virtual gift control in the live studio interface, determining the target gift special effect corresponding to the event, and displaying the custom image selection control in the live studio interface:
[0177] Please refer to Figure 8 When the user selects gift B through the virtual gift control 801, the gift selection and giving event acting on gift B will be triggered, and the user terminal will respond to the virtual gift selection and giving event and Figure 8 The custom image selection control is displayed in the live broadcast room interface shown in the figure. At this time, the live broadcast room interface on the user side will change from Figure 8 Changes to Fig.15 , displayed in the live broadcast room interface Fig.15 The custom image selection control 1501 shown in .
[0178] Step S112', responding to the image selection completion event acting on the custom image selection control, obtaining the custom image corresponding to the event:
[0179] Please refer to Fig.15 , the user at the user end passes Fig.15 The custom image selection control 1501 shown, after selecting the corresponding picture as the custom image, will trigger the image selection completion event acting on the custom image selection control 1501 shown. The user end responds to the event and obtains the custom image selected by the user through the custom image selection control 1501 shown.
[0180] Step S113', triggering a display instruction of a gift logo including the target gift special effect:
[0181] When the custom image is obtained, the display instruction acting on the gift logo corresponding to the target gift image will be triggered.
[0182] In this embodiment, the custom image synthesized into the gift special effect can be selected by the user. The user can use his or her own user avatar image or a picture in the terminal as the custom image to enrich the display effect of the virtual gift, thereby enhancing the user's enthusiasm for giving virtual gifts.
[0183] Furthermore, by functionalizing each step in the method disclosed in the above embodiments, a virtual gift special effect playback device of the present application can be constructed. According to this idea, please refer to Fig.17 In a typical embodiment, the device includes: a gift video decoding module 11, which is used to respond to the display instruction of the target gift special effect, obtain the custom image and gift special effect video corresponding to the target gift special effect, decode the gift special effect video to obtain multiple decoded image frames, and the decoded image frames include a gift color texture image, a gift transparent texture image and a custom mask texture image; a texture image cropping module 12, which is used to crop the gift color texture image, the gift transparent texture image and the custom mask texture image contained in the current decoded image frame; an image interpolation sampling module 13, which is used to sample the gift color texture image, the gift transparent texture image and the custom mask texture image according to the gift transparent texture image and the gift color texture image. The resolution ratio between the color texture images is called, the corresponding image interpolation sampling rule is called, the gift transparent texture image is sampled, and the resolution and size of the gift transparent texture image are modified to be the same as the gift color texture image; the image merging output module 14 is used to merge the gift transparent texture image and the gift color texture image to generate a gift transparent special effect image, and merge the custom image and the custom mask texture image to generate a custom composite image, and merge the custom composite image into the gift transparent special effect image to generate the current gift special effect frame image, and output the current gift special effect frame image to the live broadcast room interface for display.
[0184] In one embodiment, the gift video decoding module 11 includes: a display instruction response submodule, which is used to respond to the gift selection and giving event of the virtual gift control in the live broadcast room interface, determine the target gift special effect corresponding to the event, and trigger the display instruction of the gift identifier containing the target gift special effect; a gift video acquisition submodule, which is used to query whether there is a target gift special effect video corresponding to the target gift identifier in the gift special effect library. If so, the target gift video is obtained from the gift special effect library; if not, the target gift special effect video is obtained from the server; a custom image acquisition submodule, which is used to obtain the user avatar image corresponding to the current user terminal, and use the user avatar image as the custom image corresponding to the target gift special effect; a gift video decoding submodule, which is used to call the hardware decoding interface to decode the target gift special effect video, obtain multiple decoded image frames contained in the target gift special effect video, and sort each of the decoded image frames according to the timeline of the target gift special effect video.
[0185] In one embodiment, the texture image cropping module 12 includes an area coordinate information acquisition submodule, which is used to obtain the texture area coordinate information of the target gift special effect, wherein the texture area coordinate information includes the area coordinates corresponding to the gift color texture image, the gift transparent texture image and the custom mask texture image; a texture image cropping submodule, which is used to determine the image areas of the gift color texture image, the gift transparent texture image and the custom mask texture image contained in the current decoded image frame according to the texture area coordinate information, and crop the texture images possessed by each of the image areas; a texture image allocation submodule, which is used to correspond each of the texture images as the gift color texture image, the gift transparent texture image and the custom mask texture image of the current decoded image frame.
[0186] In one embodiment, the image interpolation sampling module 13 includes: a resolution ratio determination submodule, used to determine the resolution ratio between the gift transparent texture image and the gift color texture image; a linear interpolation sampling submodule, used to verify whether the resolution ratio exceeds a preset ratio. If not, the gift transparent texture image is sampled according to a linear interpolation sampling rule, and the resolution and size of the gift transparent texture image are modified to be the same as those of the gift color texture image; a bicubic interpolation sampling submodule, used to sample the gift transparent texture image according to a bicubic interpolation sampling rule when the resolution ratio exceeds a preset ratio, and the resolution and size of the gift transparent texture image are modified to be the same as those of the gift color texture image.
[0187] In one embodiment, the image merging and outputting module 14 includes: a special effects frame image generating submodule, used to merge the custom composite image into the gift transparent special effects image to generate the current gift special effects frame image; a display position determining submodule, used to determine the gift display position of the target gift special effects in the current live broadcast room interface; a special effects frame image display submodule, used to adjust the image size of the current gift special effects frame image according to the interface size of the current live broadcast room interface, and output the current gift special effects frame image to the gift display position for display.
[0188] In order to solve the above technical problems, the embodiment of the present application also provides a computer device for running a computer program implemented according to the virtual gift special effect playback method. Fig.18 , Fig.18 This is a basic structural block diagram of the computer device in this embodiment.
[0189] like Fig.18As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected via a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a method for playing virtual gift special effects. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute a method for playing virtual gift special effects. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0190] In this embodiment, the processor is used to execute the specific functions of each module / submodule in the virtual gift special effects playback device of this application, and the memory stores the program code and various data required to execute the above modules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the virtual gift special effects playback device, and the server can call the server's program code and data to execute the functions of all submodules.
[0191] The present application also provides a non-volatile storage medium, and the virtual gift special effects playback method is written into a computer program and stored in the storage medium in the form of computer-readable instructions. When the computer-readable instructions are executed by one or more processors, it means that the program is running in the computer, thereby enabling one or more processors to execute the steps of the virtual gift special effects playback method in any of the above embodiments.
[0192] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0193] To sum up, this application effectively ensures the display effect of transparent gift special effects in the interface and improves the live broadcast viewing experience.
[0194] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0195] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.
[0196] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for playing special effects of virtual gifts, characterized in that: The steps include: In response to a display instruction of a target gift special effect, a custom image and a gift special effect video corresponding to the target gift special effect are obtained, and the gift special effect video is decoded to obtain a plurality of decoded image frames, wherein the decoded image frames include a gift color texture image, a gift transparent texture image, and a custom mask texture image; Cut out the gift color texture image, gift transparent texture image and custom mask texture image contained in the current decoded image frame; According to the resolution ratio between the gift transparent texture image and the gift color texture image, the corresponding image interpolation sampling rule is called to sample the gift transparent texture image, and the resolution and size of the gift transparent texture image are modified to be the same as those of the gift color texture image. When the resolution ratio exceeds the preset ratio, the gift transparent texture image is sampled according to the bicubic interpolation sampling rule, and the resolution and size of the gift transparent texture image are modified to be the same as those of the gift color texture image, including the following steps: modifying the image size of the gift transparent texture image to the image size of the gift color texture image; obtaining four neighboring pixels in the previous row of the current interpolation calculation pixel point in the gift transparent texture image, calling the bicubic interpolation algorithm to calculate each of the neighboring pixels, and calculating the previous row interpolation; obtaining four neighboring pixels in the next row of the current interpolation calculation pixel point in the gift transparent texture image, calling the bicubic interpolation algorithm to calculate each of the neighboring pixels, and calculating the next row interpolation; calling the linear interpolation algorithm to calculate the previous row interpolation and the next row interpolation, and calculating the interpolation corresponding to the current interpolation calculation pixel point; The gift transparent texture image and the gift color texture image are merged to generate a gift transparent special effects image, and the custom image and the custom mask texture image are merged to generate a custom composite image, and the custom composite image is merged into the gift transparent special effects image to generate the current gift special effects frame image, and the current gift special effects frame image is output and displayed on the live broadcast room interface.
2. The method according to claim 1, characterized in that The step of responding to the display instruction of the target gift special effect, obtaining the custom image and gift special effect video corresponding to the target gift special effect, and decoding the gift special effect video to obtain multiple decoded image frames includes the following steps: In response to a gift selection and giving event of a virtual gift control in a live broadcast room interface, a target gift special effect corresponding to the event is determined, and a display instruction of a gift logo including the target gift special effect is triggered; Query whether there is a target gift special effect video corresponding to the gift identifier of the target gift special effect in the gift special effect library. If so, obtain the target gift special effect video from the gift special effect library. If not, obtain the target gift special effect video from the server. Obtain a user avatar image corresponding to the current user terminal, and use the user avatar image as a custom image corresponding to the target gift special effect; Call the hardware decoding interface to decode the target gift special effects video, obtain multiple decoded image frames contained in the target gift special effects video, and sort the decoded image frames according to the timeline of the target gift special effects video.
3. The method according to claim 1, characterized in that The step of cutting out the gift color texture image, the gift transparent texture image and the custom mask texture image contained in the current decoded image frame includes the following steps: Acquire the texture area coordinate information of the target gift special effect, wherein the texture area coordinate information includes the area coordinates corresponding to the gift color texture image, the gift transparent texture image and the custom mask texture image; According to the texture area coordinate information, the image areas of the gift color texture image, the gift transparent texture image and the custom mask texture image contained in the current decoded image frame are determined, and the texture images possessed by each of the image areas are cropped; Each of the texture images is corresponded as the gift color texture image, the gift transparent texture image and the custom mask texture image of the current decoded image frame.
4. The method according to claim 1, characterized in that: The step of calling the corresponding image interpolation sampling rule according to the resolution ratio between the gift transparent texture image and the gift color texture image, sampling the gift transparent texture image, and modifying the resolution and size of the gift transparent texture image to be the same as those of the gift color texture image, includes the following steps: Determine a resolution ratio between the gift transparent texture image and the gift color texture image; Check whether the resolution ratio exceeds a preset ratio. If not, sample the gift transparent texture image according to a linear interpolation sampling rule, and modify the resolution and size of the gift transparent texture image to be the same as those of the gift color texture image. When the resolution ratio exceeds a preset ratio, the gift transparent texture image is sampled according to the bicubic interpolation sampling rule, and the resolution and size of the gift transparent texture image are modified to be the same as those of the gift color texture image.
5. The method according to claim 1, characterized in that The step of merging the custom composite image into the gift transparent special effect image to generate a current gift special effect frame image, and outputting and displaying the current gift special effect frame image on the live broadcast room interface includes the following steps: Merging the custom composite image into the gift transparent special effect image to generate a current gift special effect frame image; Determine the gift display position of the target gift special effect in the current live broadcast room interface; The image size of the current gift special effects frame image is adjusted according to the interface size of the current live broadcast room interface, and the current gift special effects frame image is output to the gift display position for display.
6. The method according to claim 1, characterized in that The step of responding to the display instruction of the target gift special effect and obtaining the custom image and gift special effect video corresponding to the target gift special effect includes the following steps: In response to a gift selection and giving event acting on a virtual gift control in a live studio interface, a target gift special effect corresponding to the event is determined, and a custom image selection control is displayed in the live studio interface; In response to an image selection completion event acting on the custom image selection control, obtaining a custom image corresponding to the event; Trigger the display instruction of the gift logo containing the target gift special effect.
7. A virtual gift special effects playback device, characterized in that: include: A gift video decoding module is used to respond to a display instruction of a target gift special effect, obtain a custom image and a gift special effect video corresponding to the target gift special effect, and decode the gift special effect video to obtain a plurality of decoded image frames, wherein the decoded image frames include a gift color texture image, a gift transparent texture image, and a custom mask texture image; The texture image cropping module is used to crop the gift color texture image, gift transparent texture image and custom mask texture image contained in the current decoded image frame; An image interpolation sampling module is used to call the corresponding image interpolation sampling rule according to the resolution ratio between the gift transparent texture image and the gift color texture image, sample the gift transparent texture image, and modify the resolution and size of the gift transparent texture image to be the same as the gift color texture image; wherein, the image interpolation sampling module includes a bicubic interpolation sampling submodule, which is used to sample the gift transparent texture image according to the bicubic interpolation sampling rule when the resolution ratio exceeds a preset ratio, and modify the resolution and size of the gift transparent texture image to be the same as the gift color texture image, including the following steps: The image size of the gift transparent texture image is modified to the image size of the gift color texture image; four neighboring pixels in the previous row of the current interpolation calculation pixel in the gift transparent texture image are obtained, and the bicubic interpolation algorithm is called to calculate each of the neighboring pixels to calculate the previous row interpolation; four neighboring pixels in the next row of the current interpolation calculation pixel in the gift transparent texture image are obtained, and the bicubic interpolation algorithm is called to calculate each of the neighboring pixels to calculate the next row interpolation; the linear interpolation algorithm is called to calculate the previous row interpolation and the next row interpolation, and the interpolation corresponding to the current interpolation calculation pixel is calculated; An image merging output module is used to merge the gift transparent texture image and the gift color texture image to generate a gift transparent special effects image, and to merge the custom image and the custom mask texture image to generate a custom composite image, and to merge the custom composite image into the gift transparent special effects image to generate a current gift special effects frame image, and to output the current gift special effects frame image and display it on the live broadcast room interface.
8. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 6.
9. A non-volatile storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 6 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.
Citation Information
Patent Citations
Scale transformation method based on image edge direction
CN101790069A
Efficient interpolation
CN113205455A
Quadratic element virtual gift generation method and device, electronic equipment and storage medium
CN113992927A
Animation video generation method and device and animation video playing method and device
CN114598937A
Image magnification / reduction device and method, and program
JP2005063198A
