Bullet screen rendering method and device and electronic equipment

Abstract

The embodiment of the invention provides a bullet screen rendering method and device and electronic equipment, and relates to the technical field of computers. The bullet screen rendering method comprises the following steps: in a target video playing process, acquiring bullet screen data and target audio rhythm data corresponding to a target video; determining a target stage lighting special effect parameter corresponding to the target audio rhythm data according to a corresponding relationship between the audio rhythm data and the stage lighting special effect parameter; performing rendering processing on a blank texture image according to the target stage lighting special effect parameter to obtain a stage lighting texture; performing synthesis processing on the bullet screen texture included in the bullet screen data and the stage light texture to form a light irradiation bullet screen special effect; and displaying the bullet screen data with the light irradiation bullet screen special effect in a playing interface. The invention provides a bullet screen dynamic effect changing along with the audio rhythm of the video frame, and effectively enriches bullet screen display forms.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a bullet screen rendering method, device and electronic equipment. Background Art

[0002] With the development of streaming media, users have higher and higher requirements for the viewing atmosphere. Adding the barrage function during the playback of audio and video has become an important way to improve the audio and video viewing experience.

[0003] The current barrage function mainly supports: during the process of playing audio and video on an electronic device, the user can edit the text information of the barrage so that the electronic device can dynamically display the barrage in the interface playback area of ​​the current audio and video. For example, the barrage function supports the dynamic display of the barrage in the interface playback area by moving along a first direction. However, as an important interactive method under audio and video, the current display form of barrage is relatively single and the user interactivity is limited. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a bullet screen rendering method, device and electronic device, which to a certain extent solves the problem of the single display form of the current bullet screen. The specific technical solution is as follows:

[0005] In a first aspect of the present invention, a bullet screen rendering method is provided, comprising:

[0006] During the target video playback, obtaining barrage data and target audio rhythm data corresponding to the target video;

[0007] Determining target stage lighting special effect parameters corresponding to the target audio rhythm data according to the corresponding relationship between the audio rhythm data and the stage lighting special effect parameters;

[0008] Rendering the blank texture image according to the target stage light special effect parameters to obtain a stage light texture;

[0009] The bullet screen texture and the stage light texture included in the bullet screen data are synthesized to form a light illuminating bullet screen special effect;

[0010] The barrage data with the light illuminating barrage special effect is displayed in the playback interface.

[0011] In a second aspect of the present invention, a bullet screen rendering device is provided, including:

[0012] An acquisition module, used to acquire barrage data and target audio rhythm data corresponding to the target video during the playback of the target video;

[0013] A determination module, used to determine the target stage lighting special effect parameters corresponding to the target audio rhythm data according to the correspondence between the audio rhythm data and the stage lighting special effect parameters;

[0014] A rendering module, used for rendering the blank texture image according to the target stage light special effect parameters to obtain a stage light texture;

[0015] A synthesis module, used for synthesizing the bullet screen texture and the stage light texture included in the bullet screen data to form a light illuminating bullet screen special effect;

[0016] A display module is used to display the barrage data with the light illuminating barrage special effect in the playback interface.

[0017] In a third aspect of the present invention, there is also provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0018] Memory, used to store computer programs;

[0019] The processor is used to implement any method step described in the first aspect when executing the program stored in the memory.

[0020] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, wherein instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer executes any of the bullet screen rendering methods described in the first aspect above.

[0021] In a fifth aspect of the implementation of the present invention, a computer program product is also provided, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, any of the methods described in the first aspect above is implemented.

[0022] The bullet screen rendering method, device and electronic device provided in the embodiment of the present application obtain the bullet screen data and the target audio rhythm data corresponding to the target video during the playback of the target video, so that the target stage lighting special effect parameters corresponding to the target audio rhythm data can be determined according to the correspondence between the audio rhythm data and the stage lighting special effect parameters, and then the blank texture image is rendered according to the target stage lighting special effect parameters to obtain the stage lighting texture, so as to synthesize the bullet screen texture and the stage lighting texture included in the bullet screen data to form a lighting illumination bullet screen special effect, so that the bullet screen data with the lighting illumination bullet screen special effect can be displayed in the playback interface. Because the target stage lighting special effect parameters corresponding to the video frame change with the audio rhythm of the video frame. Therefore, in the lighting illumination bullet screen special effect, the bullet screen can present a dynamic effect under the dynamic lighting illumination that changes with the audio rhythm corresponding to the video frame, enriching the bullet screen display form, enhancing user interactive interest, and increasing the video viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0024] Figure 1 A flowchart of a bullet screen rendering method provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a stage light setting provided in an embodiment of the present application;

[0026] Figure 3 A flowchart of another bullet screen rendering method provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of a bullet screen special effect provided in an embodiment of the present application;

[0028] Figure 5 A function diagram of a transparency function provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of another bullet screen special effect provided in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of another bullet screen special effect provided in an embodiment of the present application;

[0031] Figure 8 A block diagram of a bullet screen processing system provided in an embodiment of the present application;

[0032] Fig. 9 A flowchart of another bullet screen rendering method provided in an embodiment of the present application;

[0033] Fig.10 A block diagram of a bullet screen rendering device provided in an embodiment of the present application;

[0034] Fig.11 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0036] Please refer to Figure 1 , which shows a flow chart of a bullet screen rendering method provided by an embodiment of the present application. The bullet screen rendering method can be applied to electronic devices. Optionally, the electronic device may include a personal computer, a tablet computer, a mobile phone, a wearable device, and the like. It should be noted that the refresh frequency of the bullet screen may be equal to or greater than the refresh frequency of the video frame. When the refresh frequency of the bullet screen is equal to the refresh frequency of the video frame, during the playback of a single video frame, the electronic device can obtain a bullet screen data, and then form a lighting bullet screen special effect based on the bullet screen data, and the lighting bullet screen special effect is displayed synchronously with the video frame.

[0037] In the case where the refresh frequency of the bullet screen is greater than the refresh frequency of the video frame, during the playback of a single video frame, the electronic device can sequentially obtain multiple bullet screen data, and each time a bullet screen data is obtained, a light illuminating bullet screen special effect is formed according to the bullet screen data, and the light illuminating bullet screen special effect is displayed in the video frame. It is understandable that in the case where the refresh frequency of the bullet screen is greater than the refresh frequency of the video frame, a plurality of light illuminating bullet screen special effects corresponding to the bullet screen data will be sequentially formed during the playback of a single video frame.

[0038] For example, suppose that the refresh frequency of the barrage is 60 frames per second and the refresh frequency of the video frame is 30 frames per second. The electronic device can obtain two barrage data in sequence during the process of playing a single video frame. After each barrage data is obtained, a lighting barrage special effect is formed according to the barrage data. Obviously, in the process of playing a single video frame, two lighting barrage special effects will be refreshed. Among them, the process of forming the lighting barrage special effect according to the barrage data is the same. For ease of understanding, the following embodiment of the present application takes a barrage data obtained in the process of playing a single video frame as an example to illustrate the barrage rendering method provided in the embodiment of the present application.

[0039] Of course, when the refresh frequency of the bullet screen is greater than the refresh frequency of the video frame, the electronic device can repeatedly execute the scheme multiple times during the playback of each video frame: obtain the bullet screen data and the target audio rhythm data; determine the target stage lighting special effect parameters corresponding to the target audio rhythm data according to the correspondence between the audio rhythm data and the stage lighting special effect parameters; render the blank texture image according to the target stage lighting special effect parameters to obtain the stage lighting texture; synthesize the bullet screen texture and the stage lighting texture included in the bullet screen data to form the lighting irradiation bullet screen special effect. Figure 1 As shown, the bullet screen rendering method includes:

[0040] Step 101: During the playback of a target video, obtain barrage data and target audio rhythm data corresponding to the target video.

[0041] In an embodiment of the present application, the video stream of the target video may be a barrage video stream. A barrage video stream refers to a video stream containing barrage data. The barrage data includes content data such as text, symbols, and pictures input by the user when watching the video, and also includes the time when the barrage is sent, the sender's location, the display position of the barrage in the video frame, and the barrage texture. Among them, the barrage texture includes the texture data of each barrage pixel in the video frame. Barrage pixels refer to the pixel points in the barrage. It should be noted that a single video frame may have one or more barrages. Accordingly, the barrage data corresponding to a single video frame may include content data of one or more barrages. The barrage texture is a barrage layer interface image with one or more barrages. The barrage layer interface image refers to the interface image of the barrage layer used to display the barrage during video playback.

[0042] The target audio rhythm data corresponding to the target video refers to the rhythm data of the audio corresponding to the target video. During the playback of each video frame of the target video, the electronic device can obtain the barrage data and the target audio rhythm data corresponding to the video frame, and then perform subsequent steps according to the barrage data and the target audio rhythm data to form the light illumination barrage special effect corresponding to the video frame, thereby displaying the video frame in the playback interface and displaying the barrage data with the light illumination barrage special effect. The electronic device can play the target video by displaying each video frame of the target video in the playback interface, and displaying the barrage data with the light illumination barrage special effect corresponding to each video frame.

[0043] The target audio rhythm data of the video frame refers to the rhythm data of the audio corresponding to the video frame. The rhythm data may refer to the pitch data of the audio corresponding to the video frame. Alternatively, the rhythm data may refer to the timbre data or loudness data of the audio corresponding to the video frame.

[0044] In an optional implementation, the electronic device can directly obtain the bullet screen data and the target audio rhythm data of the video frame during the process of playing each video frame of the target video, so as to add the light irradiation bullet screen special effect by using the target audio rhythm data later. Under the light irradiation bullet screen special effect, the bullet screen has the special effect of stage lights, and the number of stage lights, flashing frequency, irradiation direction, etc. can be limited according to the rhythm of the audio corresponding to the video frame, so as to present the effect of dynamic light irradiation to the bullet screen that changes with the rhythm of the audio corresponding to the video frame.

[0045] In another optional implementation, the electronic device may also obtain the bullet screen data and the target audio rhythm data corresponding to the target video when the special effect triggering condition is met, so as to use the target audio rhythm data to display the bullet screen with the added light irradiation bullet screen special effect in each video frame. Among them, satisfying the special effect triggering condition includes at least one of the following: the target video is a target video of the target class video. Responding to the triggering operation of the light irradiation bullet screen control. Responding to the bullet screen publishing operation.

[0046] The target video refers to a video whose audio rhythm meets a preset rhythm rule. The preset rhythm rule may be a music rhythm, a reading rhythm, etc. Optionally, the target video may be a music program video, a reading program video, or a film and television drama segment video with background sound, etc. For example, the electronic device may automatically trigger the light illumination barrage special effect of the barrage during the process of playing the target video of the music program video.

[0047] The light illumination barrage control is used to trigger the light illumination barrage control of the video frame. Optionally, the triggering operation of the light illumination barrage control can be an operation in the form of clicking, long pressing, sliding or voice for the light illumination barrage control. For example, the electronic device displays a light illumination barrage control. The user can trigger the light illumination barrage special effect of the barrage by clicking the light illumination barrage control.

[0048] The bullet comment posting operation refers to the operation of posting a bullet comment when the user is watching a video. For example, after the user sends the bullet comment, the light of the bullet comment is triggered to illuminate the bullet comment special effect, forming an easter egg.

[0049] Step 102: Determine target stage lighting special effect parameters corresponding to the target audio rhythm data according to the corresponding relationship between the audio rhythm data and the stage lighting special effect parameters.

[0050] In an embodiment of the present application, the electronic device can quantize the diverse audio rhythms to obtain multiple audio rhythm levels, also known as audio rhythm ranges.

[0051] A corresponding stage lighting special effect parameter may be set for each audio rhythm gear, and the stage lighting special effect parameters corresponding to multiple audio rhythm gears may be at least partially different. The stage lighting special effect parameter indicates the lighting effect of the stage light. Optionally, the stage lighting special effect parameter indicates the color of light at each position on the stage when the stage light is irradiated to the playback area of ​​the playback interface of the target video as the stage. Specifically, the stage lighting special effect parameter may include the color value of the light at each pixel position on the video frame when the stage light is irradiated to the playback area of ​​the playback interface of the target video as the stage.

[0052] In an optional implementation, the electronic device directly stores audio rhythm data of a plurality of audio rhythm gears, and a corresponding relationship between each audio rhythm data and a stage lighting special effect parameter.

[0053] The electronic device can search for the stage lighting special effect parameters corresponding to the target audio rhythm data from the corresponding relationship between the audio rhythm data and the stage lighting special effect parameters according to the target audio rhythm data.

[0054] In another optional implementation, the electronic device stores audio data of multiple audio rhythm gears, and a corresponding relationship between each audio rhythm data and a stage light parameter. The stage light parameter indicates a virtual stage light setting with the playback area of ​​the playback interface of the target video as the stage. Optionally, the stage light parameter may include parameters such as the number of stage lights, the setting position of the stage lights relative to the stage, the color of the light source, the flashing frequency of the light source, and the irradiation direction of the stage lights.

[0055] It should be noted that, in some embodiments, the specific content of each parameter in the stage light parameter may be a change in time. Figure 2 As shown in the figure, the stage light parameter corresponding to a certain audio rhythm level indicates: four light sources symmetrical to the left and right of the stage are used to illuminate the playback area, and the four light sources are used to swing rhythmically with a set frequency. As the video playback time increases, the swing frequency and direction of the four light sources can change.

[0056] Based on this, the process of the electronic device determining the target stage lighting special effect parameters corresponding to the target audio rhythm data according to the correspondence between the audio rhythm data and the stage lighting special effect parameters may include the following steps 1021 to 1022.

[0057] In step 1021, target stage light parameters corresponding to the target audio rhythm data are determined according to the corresponding relationship between the audio rhythm data and the stage light parameters.

[0058] Optionally, the electronic device may store a correspondence between audio rhythm data and stage light parameters. According to the target audio rhythm data, the stage light parameters corresponding to the target audio rhythm data are searched from the correspondence between the audio rhythm data and the stage light parameters to obtain the target stage light parameters. The target stage light parameters may include at least one of the following: the number of stage lights, the flashing frequency of the light source, and the irradiation direction of the stage lights.

[0059] It should be noted that, in some embodiments, the stage light effect reflected by the stage light parameters corresponding to each audio rhythm gear can be artificially designed. In other embodiments, the stage light effect reflected by the stage light parameters corresponding to each audio rhythm gear can also be calculated and determined according to a specific algorithm. The specific algorithm is used to indicate the preset correlation between the stage light effect and the audio rhythm. For example, the stage light effect can have an emotional correlation with the audio rhythm. The stage light effect can be designed according to the emotion reflected by the music of each audio rhythm gear, and then the stage light parameters corresponding to the stage light effect are determined. For example, the higher the pitch of the music, the more stage lights there are. Or, the faster the pitch of the music increases, the higher the flashing frequency of the light source of the stage light, etc.

[0060] In one example, the correspondence between the audio rhythm data and the stage light parameter includes: the first audio rhythm level corresponds to the first stage light parameter, the second audio rhythm level corresponds to the second stage light parameter, and the third audio rhythm level corresponds to the third stage light parameter. When the electronic device determines that the acquired target audio rhythm data belongs to the second audio rhythm level, it determines that the target stage light parameter is the second stage light parameter.

[0061] In another example, taking the audio rhythm data as pitch data, the corresponding relationship between the audio rhythm data and the stage light parameters includes: bass (frequency range is 20Hz-200Hz) corresponds to the first stage light parameter, midrange (frequency range is 200Hz-2000Hz) corresponds to the second stage light parameter, and treble (frequency range is 2000Hz-20000Hz) corresponds to the third stage light parameter.

[0062] Among them, the first stage light parameters include quantity 1, light source flashing frequency F11, stage center setting, and stage light irradiation direction α11, etc. The second stage light parameters include quantity 2, light source flashing frequencies F21, F22, stage left and right upper corner settings, and stage light irradiation directions α21, α22, etc. The third stage light parameters include quantity 4, light source flashing frequencies F31, F32, F33, left and right symmetric settings of the four corners of the stage, and stage light irradiation directions α31, α32, α33, etc. When the electronic device determines that the target audio rhythm data belongs to the middle tone, it determines that the target stage light parameters include quantity 2, light source flashing frequencies F21, F22, stage left and right upper corner settings, and stage light irradiation directions α21, α22, etc.

[0063] In step 1021, target stage lighting special effect parameters are calculated according to the target stage lighting parameters.

[0064] In the embodiment of the present application, the target stage lighting special effect parameter indicates the color of light at each position on the stage when the virtual stage light illuminates the playback area of ​​the playback interface of the target video as the stage under the stage lighting setting indicated by the target stage lighting parameter. Specifically, the target stage lighting special effect parameter may include the color value of light at each pixel position on the video frame when the virtual stage light illuminates the playback area of ​​the playback interface as the stage under the virtual stage lighting setting indicated by the target stage lighting parameter.

[0065] In an optional implementation, the electronic device calculates target stage light special effect parameters based on the target stage light parameters. The target stage light special effect parameters indicate the implementation process of the lighting effect of the stage light, which may include: calling an image rendering engine and using a lighting algorithm to calculate the target stage light special effect parameters based on the target stage light parameters.

[0066] Optionally, the electronic device can call an image rendering engine through a graphics processing unit (GPU) to adopt a lighting algorithm to calculate the target stage light special effect parameters according to the target stage light parameters. The image rendering engine can be Metal or OpenGL, etc. The lighting algorithm can be a Lambert model algorithm, a Phong model algorithm, a Blinn-Phong model algorithm, a ray tracing algorithm, a path tracing algorithm, etc.

[0067] In another optional implementation, the electronic device may also obtain the correspondence between the stage light parameters and the stage light special effect parameters. According to the target stage light parameters, the stage light special effect parameters corresponding to the target stage light parameters are determined from the correspondence between the acquired stage light parameters and the stage light special effect parameters, and the target stage light special effect parameters are obtained. Among them, the electronic device may locally store the correspondence between the stage light parameters and the stage light special effect parameters so as to directly read the correspondence and improve the rendering speed of the barrage special effect. Alternatively, the correspondence between the stage light parameters and the stage light special effect parameters may be stored in a third-party device. The electronic device may read the correspondence from the third-party device in real time when necessary to reduce the amount of data stored locally in the electronic device.

[0068] In some embodiments of the present application, under the lighting barrage special effect, the color of the stage light (i.e., the light source color in the stage light parameter) can be determined according to the background color of the current video frame. Optionally, step 102 determines the target stage light special effect parameters corresponding to the target audio rhythm data according to the corresponding relationship between the audio rhythm data and the stage light special effect parameters, and also includes:

[0069] Obtain the background color of the video frame of the target video, and determine the light source color as the background color. Add the light source color to the target stage light parameter. Optionally, the electronic device may directly determine the background color of the video frame as the light source color. Alternatively, the electronic device may also determine the color close to the background color of the video frame as the light source color. The color close to the background color refers to a color whose color value differs from the background color by less than a color difference threshold.

[0070] In this way, by setting the special effect color of the barrage special effect under the light, that is, the light source color of the stage light is set as the background color of the video frame, the barrage under the barrage special effect under the light can be better integrated with the video frame, achieving a more natural barrage special effect and improving the barrage effect.

[0071] Further optionally, the process of the electronic device acquiring the background color of the video frame of the target video may include: determining the proportion of various types of colors in the video frame; and selecting the color with the highest proportion among the various types of colors contained in the video frame as the background color.

[0072] In some embodiments, the proportion of various colors in a video frame can be understood as the proportion of the number of pixels of various colors in the total number of pixels in the video frame. In addition, since the total number of pixels of each video frame of the same pixel is usually the same, the color with the highest proportion is the color with the largest number of pixels. Therefore, the number of pixels of various colors in a video frame can also be directly used as the proportion of various types of colors in the video frame. Furthermore, the color with the largest number of pixels is used as the background color.

[0073] For example, the electronic device may obtain the color of each pixel in the video frame, and count the number of pixels of each color, and then use the color with the largest number of pixels as the background color.

[0074] Step 103: Rendering the blank texture image according to the target stage lighting special effect parameters to obtain a stage lighting texture.

[0075] In some embodiments of the present application, a blank texture image may refer to a white texture image, or a blank texture image may refer to a solid color image with a single texture.

[0076] Optionally, the electronic device can call the target rendering pipeline in the image rendering engine through the GPU, and render the blank texture image according to the target stage light special effect parameters to obtain the stage light texture. The rendering process here can be understood as modifying, replacing, and other processing of the color value of each pixel in the blank texture image according to the target stage light special effect parameters, so that the blank texture image presents the form after being illuminated by the stage light indicated by the target stage light special effect parameters. Accordingly, the stage light texture can be understood as a new pixel image texture obtained by adjusting the pixel content of the blank texture image through the rendering pipeline.

[0077] In an optional implementation, the image size of the blank texture image can be equal to the image size of the video frame. In this way, the image size of the rendered stage light texture is also equal to the image size of the video frame, which facilitates the subsequent use of the stage light texture to process the bullet screen, so that the bullet screen under the light-illuminating bullet screen special effect can effectively present the state of being illuminated by the stage light when the bullet screen floats on the stage under the entire area of ​​the playback area of ​​the playback interface illuminated by the stage light.

[0078] Step 104: synthesize the bullet screen texture and the stage light texture included in the bullet screen data to form a light illuminating bullet screen special effect.

[0079] In the embodiment of the present application, the synthesis process can be understood as a mixing technology. The electronic device can merge the bullet screen texture and the stage light texture through a mixing formula to form a light-illuminating bullet screen special effect. Optionally, the synthesis process includes multitexturing processing or texture blending processing. Among them, multitexturing processing is used to draw multiple layers of textures on the same surface by adjusting and mixing different parameters to achieve complex visual effects. Texture blending processing is used to draw multiple textures on the same surface, and to smoothly transition each texture through a blending technique to avoid obvious texture boundaries.

[0080] Step 105: Display the bullet screen data with the special effects of light illumination on the bullet screen in the playback interface.

[0081] In an embodiment of the present application, the electronic device can display each video frame and the barrage data with light illumination barrage special effects corresponding to the video frame in the playback interface to play the target video with barrage.

[0082] In an embodiment of the present application, by obtaining the barrage data and the target audio rhythm data corresponding to the target video during the playback of the target video, the target stage lighting special effect parameters corresponding to the target audio rhythm data can be determined according to the correspondence between the audio rhythm data and the stage lighting special effect parameters, and then the blank texture image is rendered according to the target stage lighting special effect parameters to obtain the stage lighting texture, so that the barrage texture and the stage lighting texture included in the barrage data are synthesized to form a lighting barrage special effect, so that the barrage data with the lighting barrage special effect can be displayed in the playback interface. Since the target stage lighting special effect parameters corresponding to the video frame change with the audio rhythm of the video frame. Therefore, in the lighting barrage special effect, the barrage can present a dynamic effect under the dynamic lighting that changes with the audio rhythm corresponding to the video frame, enriching the barrage display form, enhancing user interactivity and interest, and increasing the video viewing experience.

[0083] In some embodiments of the present application, the bullet screen can also achieve its own color flashing effect to further enrich the bullet screen display form and enhance user interaction and interest. Optionally, the bullet screen texture includes the bullet screen color of each bullet screen pixel, and the bullet screen color is the background color of the video frame. Figure 3 As shown, before the bullet screen texture and the stage light texture included in the bullet screen data are synthesized in step 104 to form the light illuminating bullet screen special effect, the bullet screen rendering method further includes:

[0084] Step 301: Determine target transparency data corresponding to target audio rhythm data according to the corresponding relationship between audio rhythm data and transparency data.

[0085] In the embodiment of the present application, each audio rhythm data corresponds to different transparency data, wherein the transparency data includes the transparency value of each pixel position in the video frame.

[0086] In an optional case, the transparency data may directly include the transparency value of each pixel position in the video frame. For example, the transparency data corresponding to a certain audio rhythm data includes: the transparency value of the pixel position (1,1) in the video frame is 10%, the transparency value of the pixel position (1,2) is 10.5%, ..., the transparency value of the pixel position (i,j) is 50%, ..., the transparency value of the pixel position (n,m) is 100%. The pixel position (i,j) represents the position of the pixel in the i-th row and j-th column in the video frame. n is the total number of pixel rows in the video frame, and m is the total number of pixel columns in the video frame, that is, the number of pixels in the video frame is n×m. Accordingly, 1≤i≤n, 1≤j≤m, and i and j are integers.

[0087] In another optional case, the video frame can be divided into multiple image areas. The transparency values ​​of each pixel position in a single image area are the same under an audio rhythm, and the transparency value is a static value. Among the multiple image areas corresponding to the same audio rhythm data, at least some of the image areas have different transparency values. The transparency values ​​of two adjacent image areas can be the same or different.

[0088] Based on this, the transparency data may include the transparency value of each image area in the video frame. Figure 4 As shown, the video frame is divided into 9 image areas from left to right, and the 9 image areas include image area 1, image area 2, ..., image area 9. The transparency data corresponding to a certain audio rhythm data include: the transparency value of the color of each pixel position in image area 1, image area 5, and image area 7 is 50%; the transparency value of the color of each pixel position in image area 2, image area 3, image area 4, image area 6, image area 8, and image area 9 is 100%. The transparency data corresponding to another audio rhythm data include: the transparency value of the color of each pixel position in image area 1, image area 3, image area 5, image area 7, and image area 9 is 50%; the transparency value of the color of each pixel position in image area 2, image area 4, image area 6, and image area 8 is 100%.

[0089] In another optional case, the video frame can be divided into multiple image areas. Among the multiple image areas corresponding to the same audio rhythm data, there are at least some image areas with different transparency values. The transparency values ​​of two adjacent image areas can be the same or different. The transparency values ​​of each pixel position in a single image area are the same under an audio rhythm, but the transparency value can be a dynamic value that changes with time. Of course, the transparency can also be a dynamic value that changes with other factors. The other factors can be background color, video type, etc.

[0090] In some embodiments, each image area in a video frame may refer to multiple image areas obtained by dividing the video frame according to the moving direction of the barrage in the playback interface, and the transparency functions corresponding to adjacent image areas among the multiple image areas are different, but there are at least two non-adjacent image areas that have the same transparency function.

[0091] Among them, the transparency function uses time as an independent variable to indicate the transparency value of each position in the image area. The time can be the relative playback time of the video frame in the video. Or, the time can be the actual playback time of the video frame. Optionally, the transparency function can be a sine function, a cosine function, etc. In addition, the moving direction of the bullet screen in the playback interface can be the direction from left to right in the playback interface. Alternatively, the moving direction of the bullet screen in the playback interface can also be the direction from top to bottom in the playback interface, etc., which is not limited in the embodiments of the present application.

[0092] As an example, Figure 4 As shown, the video frame can be divided into 9 image areas, namely, image area 1, image area 2, image area 3, image area 4, image area 5, image area 6, image area 7, image area 8 and image area 9, from left to right (i.e., the X-axis direction of the video frame).

[0093] Among them, the transparency data corresponding to an audio rhythm data includes: the transparency value of the color of each pixel position in image area 1, image area 2, image area 8, and image area 9 satisfies a first transparency function with time as the independent variable; the transparency value of the color of each pixel position in image area 3, image area 4, image area 5, image area 6, and image area 7 satisfies a second transparency function with time as the independent variable.

[0094] Transparency data corresponding to another audio rhythm data includes: the transparency value of the color of each pixel position in image area 1, image area 2, image area 4, image area 5, image area 7, and image area 8 satisfies the third transparency function with time as the independent variable; the transparency value of the color of each pixel position in image area 3, image area 6, and image area 9 satisfies the fourth transparency function with time as the independent variable. Among them, the first transparency function, the second transparency function, the third transparency function, and the fourth transparency function are all different.

[0095] As another example, continue with Figure 4 Take the division shown as an example. The transparency data corresponding to a certain audio rhythm data includes: the transparency value of the color of each pixel position in image area 1, image area 2, image area 8, and image area 9 satisfies the first transparency function with time as the independent variable; the transparency value of the color of each pixel position in image area 3, image area 4, image area 5, image area 6, and image area 7 satisfies the second transparency function with time as the independent variable, and the second transparency function is different from the first transparency function, such as Figure 5As shown, the first transparency function may be f1(x)=sin(x-π / 2)×0.5+1. The second transparency function may be f2(x)=sin(x+π / 2)×0.5+1. In the first transparency function and the second transparency function, f(x) represents the transparency value, and x represents the time. The time may be the relative playback time of the video frame in the video. Alternatively, the time may be the actual playback time of the video frame.

[0096] Optionally, the electronic device can obtain the corresponding relationship between the audio rhythm data and the transparency data, and search for the transparency data corresponding to the target audio rhythm data from the corresponding relationship to obtain the target transparency data. The electronic device can store the corresponding relationship between the audio rhythm data and the transparency data so as to directly read the corresponding relationship stored in itself. Alternatively, the electronic device can also read the corresponding relationship between the audio rhythm data and the transparency data stored in a third-party device to obtain the corresponding relationship.

[0097] For example, it is assumed that the transparency data corresponding to the first audio rhythm gear includes: the transparency value of the color of each pixel position in image area 1, image area 2, image area 8, and image area 9 is 100%; the transparency value of the color of each pixel position in image area 3, image area 4, image area 5, image area 6, and image area 7 is 50%.

[0098] The transparency data corresponding to the second audio rhythm gear includes: the transparency value of the color of each pixel position in image area 1, image area 2, image area 8, and image area 9 is 50%; the transparency value of the color of each pixel position in image area 3, image area 4, image area 5, image area 6, and image area 7 is 50%.

[0099] The transparency data corresponding to the third audio rhythm gear include: the transparency value of the color of each pixel position in image area 1, image area 2, image area 4, image area 5, image area 7, and image area 8 is 80%; the transparency value of the color of each pixel position in image area 3, image area 6, and image area 9 is 20%.

[0100] If the target audio rhythm data is the third audio rhythm gear, the electronic device determines the target transparency data based on the correspondence between the audio rhythm data and the transparency data, including: the transparency value of the color of each pixel position in image area 1, image area 2, image area 4, image area 5, image area 7, and image area 8 is 80%; the transparency value of the color of each pixel position in image area 3, image area 6, and image area 9 is 20%.

[0101] Step 302: Determine the transparency value of each barrage pixel according to the pixel position corresponding to each barrage pixel in the video frame and the target transparency data.

[0102] In some embodiments of the present application, the electronic device can parse the pixel position corresponding to each barrage pixel in the video frame from the barrage data. The pixel position of the barrage pixel is related to the barrage sending time, the barrage flight speed, the current actual time, etc. For example, take a video frame as an image of n×m pixels. Figure 6 The barrage 601 "I can't get out" was sent at t1, and the starting flying position of barrage pixel 602 in barrage 601 in the video frame is the i-th row and the m-th column. If the barrage flying speed is 10 pixels / second, then 10 seconds after the barrage is sent, the corresponding pixel position of barrage pixel 402 in barrage 401 in the video frame is the i-th row and the m-100th column. It should be noted that Figure 6 The middle barrage pixel 602 is only an exemplary identifier and does not represent the size of the barrage pixel in actual application.

[0103] It is not difficult to understand that the specific way in which the electronic device determines the transparency value of each barrage pixel according to the pixel position corresponding to each barrage pixel in the video frame and the target transparency data is related to the specific content of the aforementioned transparency data. The following is an exemplary description of the confirmation process of the transparency value of each barrage pixel in accordance with the corresponding implementation method of the aforementioned three types of transparency data, but does not limit the confirmation process of the transparency value of each barrage pixel that may be included in the technical solution of the present application.

[0104] In the implementation corresponding to the first optional situation in step 301, the transparency data directly includes the transparency value of each pixel position in the video frame.

[0105] The electronic device can determine the transparency value of each barrage pixel directly from the target transparency data according to the pixel position corresponding to each barrage pixel in the video frame.

[0106] For example, assuming that the target transparent data includes: the transparency value of the pixel position (1,1) in the video frame is 10%, the transparency value of the pixel position (1,2) is 10.5%, ..., the transparency value of the pixel position (i,j) is 50%, ..., the transparency value of the pixel position (n,m) is 100%. If the pixel position corresponding to a certain bullet screen pixel in the video frame is the pixel position (1,2), the electronic device determines that the transparency value of the bullet screen pixel is 10.5% according to the target transparent data.

[0107] In the implementation manner corresponding to the second optional situation in step 301, the transparency value of each pixel position in a single image area under one audio rhythm is the same, and the transparency value is a static value.

[0108] Optionally, the transparency data includes a transparency function of each image area in the video frame. The transparency function is a constant function. Accordingly, the target transparency data includes the transparency function of each image area. Based on this, step 302 determines the transparency value of each barrage pixel according to the pixel position corresponding to each barrage pixel in the video frame and the target transparency data, and the process may include steps 3021A to 3022A.

[0109] In step 3021A, the image area corresponding to each barrage pixel in the video frame is determined according to the pixel position corresponding to each barrage pixel in the video frame.

[0110] Optionally, the electronic device may store pixel position information of each image area. The pixel position information indicates the position of the image area in the video frame. The electronic device may determine, for each bullet screen pixel, the image area where the pixel position corresponding to the bullet screen pixel in the video frame is located according to the pixel position information of each image area.

[0111] For example, assume that the video frame is divided into three image areas from left to right, namely, image area 1, image area 2, and image area 3. The pixel position information of the three image areas indicates that the area of ​​each image area occupies 1 / 3 of the total area of ​​the video frame. When the electronic device determines that the video frame is an image of n×m pixels, it can determine that the area range of image area 1 is: the area composed of pixel position (1,1)-pixel position (1,1 / 3×m)-pixel position (n,1 / 3×m)-pixel position (n,1); the area range of image area 2 is: the area composed of pixel position (1,1 / 3×m+1)-pixel position (1,2 / 3×m)-pixel position (n,2 / 3×m)-pixel position (n,1 / 3×m+1); the area range of image area 3 is: the area composed of pixel position (1,2 / 3×m+1)-pixel position (1,m)-pixel position (n,m)-pixel position (n,2 / 3×m+1). If the pixel position corresponding to a certain bullet-chat pixel A in the video frame is the pixel position (1, 2), and 2<m, the electronic device determines that the image area where the bullet-chat pixel A is located is the image area 1.

[0112] In step 3022A, the transparency value of each barrage pixel is determined based on the image area corresponding to each barrage pixel and the target transparency function of each image area corresponding to the target audio rhythm data.

[0113] Optionally, the electronic device may determine the transparency value indicated by the target transparency function of the image area corresponding to each barrage pixel in the target transparency data as the transparency value of each barrage pixel. For example, continuing with the example in 3021 above, the electronic device determines the transparency value indicated by the target transparency function of image area 1 in the target transparency data as the transparency value of barrage pixel A.

[0114] Alternatively, as in the above example, the transparency data includes the transparency value of each image area in the video frame. Accordingly, the electronic device can determine the image area corresponding to the barrage pixel in the video frame according to the pixel position corresponding to the barrage pixel in the video frame for each barrage pixel, and then determine the transparency value of the image area in the target transparent data as the transparency value of the barrage pixel, thereby obtaining the transparency value of each barrage pixel.

[0115] In the implementation manner corresponding to the third optional situation in step 301, the transparency value of each pixel position in a single image area at one audio rhythm is the same, but the transparency value may be a dynamic value that changes with time.

[0116] Optionally, the transparency data includes a transparency function of each image area in the video frame. The transparency function in the transparency data takes time as an independent variable. Accordingly, the target transparency data includes a target transparency function of each image area. Based on this, step 302 determines the transparency value of each barrage pixel according to the image area corresponding to each barrage pixel and the target transparency function of each image area corresponding to the target audio rhythm data, and the process may include steps 3021B to 3023B.

[0117] In step 3021B, the image area corresponding to each barrage pixel in the video frame is determined according to the pixel position corresponding to each barrage pixel in the video frame.

[0118] The explanation and implementation of this step can refer to the explanation and implementation of the aforementioned step 3021A, and this embodiment of the present application will not be elaborated on this.

[0119] In step 3022B, the playback time of the video frame is input into the target transparency function of each image area to obtain the transparency value of each image area.

[0120] Optionally, the electronic device can input the playback time of the video frame into the target transparency data and the target transparency function of each image area in sequence to obtain the transparency value corresponding to the playback time of each image area and the video frame, and the audio rhythm of the video frame.

[0121] For example, assume that the target transparency data corresponding to the target audio rhythm data includes: the transparency value of the color of each pixel position in image area 1, image area 2, image area 8, and image area 9 satisfies the first transparency function: f1(x)=sin(x-π / 2)×0.5+1; the transparency value of the color of each pixel position in image area 3, image area 4, image area 5, image area 6, and image area 7 satisfies the second transparency function: f2(x)=sin(x+π / 2)×0.5+1.

[0122] The electronic device inputs the playback time x1 of the video frame into the first transparency function and the second transparency function respectively, obtains the calculation result f1(x1) of the first transparency function and the calculation result f2(x1) of the second transparency function, and determines that the transparency values ​​of image area 1, image area 2, image area 8, and image area 9 are f1(x1), and the transparency values ​​of image area 3, image area 4, image area 5, image area 6, and image area 7 are f2(x1).

[0123] In step 3023B, the transparency value of each barrage pixel is determined according to the image area corresponding to each barrage pixel and the transparency value of each image area.

[0124] The explanation and implementation of this step can refer to the explanation and implementation of the aforementioned step 3022A, and the embodiments of the present application are not limited to this.

[0125] In some embodiments of the present application, after the target transparency data corresponding to the target audio rhythm data of the video frame, for example, in step 3022B, the playback time of the video frame is input into the target transparency function of each image area to obtain the transparency value of each image area, the electronic device also performs edge smoothing transition processing on each image area for the transparency value, updates the transparency value of each image area, and determines the transparency value of each barrage pixel according to the image area corresponding to each barrage pixel and the transparency value of each image area after the update.

[0126] It should be noted that in step 3022A, the transparency value of each barrage pixel is determined according to the image area corresponding to each barrage pixel and the target transparency function of each image area corresponding to the target audio rhythm data. The electronic device may also first perform edge smoothing transition processing on each image area for the transparency value after determining the transparency value of each image area according to the target transparency function of each image area corresponding to the target audio rhythm data, update the transparency value of each image area, and then determine the transparency value of each barrage pixel according to the image area corresponding to each barrage pixel and the transparency value of each image area after the update.

[0127] Optionally, the smooth transition processing of the transparency value of the image area may refer to balancing the transparency value of the edge area of ​​the image area according to the transparency value of the image area and the transparency value of the image area adjacent to the image area, so as to eliminate the difference in transparency value in relatively close areas between two adjacent image areas, and avoid abrupt color changes of the barrage located between two adjacent image areas and poor display effect of the barrage due to excessive difference in transparency value between the two adjacent image areas.

[0128] In an optional implementation manner, the electronic device performs edge smoothing transition processing on each image area with respect to a transparency value, and the process of updating the transparency value of each image area may include:

[0129] For each image region, the transparency value of the image region is interpolated and calculated according to the transparency value of the image region and the transparency values ​​of the image regions adjacent to the image region to obtain a transition transparency value. The transparency value of the edge region of the image region close to its adjacent image region is updated to the transition transparency value. Correspondingly, the transparency value of the middle region of the image region far from its adjacent image region is retained. It should be noted that

[0130] The interpolation calculation process may be an average calculation process, a linear calculation process, etc. The transition transparency value may be a value, that is, the transparency of each pixel position in the edge region of the image region close to its adjacent image region is the same. Optionally, the transition transparency value may be the transparency value of the image region and the average of the transparency values ​​of the image regions adjacent to the image region. Alternatively, the transition transparency value may be the transparency value of the image region and a random value between the transparency value of the adjacent image region.

[0131] Alternatively, the transition transparency value may also be a set of values, that is, the transparency of each pixel position in the edge area of ​​the image area close to its adjacent image area may be at least partially different. Optionally, the closer the transparency value of the image area is to the adjacent image area, the closer it is to the transparency value of the adjacent image area. It should be noted that the size of the edge area of ​​each image area can be set according to actual needs, for example, the area of ​​the edge area is less than 10% of the area of ​​the image area, etc. The embodiments of the present application are not limited to this.

[0132] For example, Figure 7As shown, the video frame is divided into three image areas from left to right, image area 1, image area 2, and image area 3. The electronic device updates the average value of the transparency values ​​of image area 1 and image area 2 as a transition transparency value to the transparency value of edge area 1-2 in image area 1 close to image area 2. The electronic device updates the average value of the transparency values ​​of image area 1 and image area 2 as a transition transparency value to the transparency value of edge area 2-1 in image area 2 close to image area 1. The electronic device updates the average value of the transparency values ​​of image area 2 and image area 3 as a transition transparency value to the transparency value of edge area 2-3 in image area 2 close to image area 3. The electronic device updates the average value of the transparency values ​​of image area 2 and image area 3 as a transition transparency value to the transparency value of edge area 3-2 in image area 3 close to image area 2.

[0133] Step 303: Update the bullet chat texture. The updated bullet chat texture also includes the transparency value of each bullet chat pixel.

[0134] In an embodiment of the present application, the updated barrage texture includes: the barrage color (i.e., the background color of the video frame) and the transparency value of each barrage pixel.

[0135] Step 304: perform illumination processing on the bullet screen texture to obtain a processed bullet screen texture.

[0136] In some embodiments of the present application, the electronic device performs illumination processing on the bullet screen texture, and the process of obtaining the processed bullet screen texture may include:

[0137] The electronic device can perform basic lighting calculation processing, bullet screen material setting processing, shadow design processing, and normal mapping processing on the bullet screen texture in sequence according to the stage light parameters to obtain the processed bullet screen texture. Among them, the basic lighting calculation processing can be used to calculate the reflectivity of each bullet screen pixel in the bullet screen texture under the lighting conditions indicated by the stage light parameters according to the lighting model, so as to adjust the lighting effect of the bullet screen texture under the illumination of the stage light. The bullet screen material setting processing is used to adjust the reflectivity of each bullet screen pixel in the bullet screen texture according to the preset bullet screen material and the material properties of the bullet screen material such as diffuse reflection, specular reflection, and whether it is self-luminous, so as to further adjust the illumination effect of the bullet screen texture. The shadow design processing is used to use shadow mapping processing to render the bullet screen texture with the position of the stage light as the viewing angle in the scene where the video frame is located, so as to enhance the illumination authenticity of the bullet screen texture. Normal mapping is used to add a normal map (Normal Map) on the surface of the bullet screen texture to replace the real normal of the object surface, so that more complex surface features can be simulated during lighting calculation, the visual details of the texture can be increased, and the lighting effect can be made more realistic.

[0138] In this way, by performing lighting processing on the barrage texture, the appearance of the barrage texture is adjusted according to the lighting conditions indicated by the stage light parameters, the naturalness of the barrage texture is enhanced, it is more coordinated with the lighting environment, and the effect of the barrage texture is improved.

[0139] In the embodiment of the present application, the electronic device performs step 304, and the implementation process of step 104 is also to synthesize the processed bullet screen texture and the stage light texture to form a light irradiation bullet screen special effect. The implementation method can refer to the implementation method of the aforementioned step 104, and the embodiment of the present application will not be repeated.

[0140] In this way, by determining the target transparent data corresponding to the target audio rhythm data according to the corresponding relationship between the audio rhythm data and the transparency data, the transparency value of each barrage pixel is determined according to the pixel position corresponding to each barrage pixel in the video frame and the target transparent data, and then the barrage texture is updated, and the barrage texture is illuminated to obtain the processed barrage texture, and then according to the barrage texture, after subsequent synthesis processing with the stage light texture, the color flickering special effect of the barrage itself can be formed. In addition, since the transparency value of the barrage in the video frame changes with the audio rhythm of the video frame. Therefore, in the light-irradiated barrage special effect, the color of the barrage itself can also present a flickering effect that changes with the audio rhythm corresponding to the video frame, further enriching the barrage display form, enhancing user interactivity and interest, and increasing the video viewing experience.

[0141] The barrage rendering method provided in the embodiment of the present application can generate a light-illuminating barrage special effect during the playback of the target video. And because the target stage lighting special effect parameters corresponding to the video frame change with the audio rhythm of the video frame. Therefore, in the light-illuminating barrage special effect, the barrage can present a dynamic effect under dynamic lighting that changes with the audio rhythm corresponding to the video frame. And, because the transparency value of the barrage in the video frame changes with the audio rhythm of the video frame. Therefore, in the light-illuminating barrage special effect, the color of the barrage itself can also present a flashing effect that changes with the audio rhythm corresponding to the video frame, enriching the barrage display form, enhancing user interactivity and interest, and increasing the video viewing experience.

[0142] In order to facilitate understanding of the technical solution of this application, the following is further explained with examples. For example, please refer to Figure 8 , which shows a block diagram of a bullet screen processing system to which the bullet screen rendering method provided in the embodiment of the present application can be applied. Figure 8 As shown, the barrage processing system can execute the barrage rendering method provided in the embodiment of the present application. The barrage processing system includes: a business layer, a rendering layer and a presentation screen.

[0143] The business layer includes an operation configuration module and a user trigger configuration module. The operation configuration module is used to store the configured special effect trigger conditions. The user trigger configuration module is used to determine whether the special effect trigger conditions are currently met based on the special effect trigger conditions provided by the operation configuration module, so as to determine whether the user triggers the light illumination bullet screen special effect. The rendering layer includes: a stage light generation module, a stage light rendering module, a bullet screen texture rendering module, and a rendering pipeline synthesis module.

[0144] The generation module of the stage light can be used to call the processor (Central Processing Unit, CPU) to obtain audio rhythm information and light source color value (i.e., light source color). Among them, audio rhythm information, also known as audio rhythm data, includes the corresponding relationship between audio rhythm data and stage light special effect parameters. The light source color value is the background color of the video frame. The generation module of the stage light is also used to call the GPU to generate target stage light special effect parameters, which describe stage lights with specific stage light quantity, light source flashing frequency, stage light irradiation direction, light source color and other characteristics.

[0145] The stage light rendering module can be used to call the GPU, and render the blank texture image according to the target stage light special effect parameters through the vertex shader and the fragment shader to obtain the stage light texture.

[0146] The bullet screen texture rendering module can be used to call the CPU to determine the transparency value of each bullet screen pixel in the video frame bullet screen, call the GPU, and perform lighting processing through the vertex shader and fragment shader according to the transparency value and color value of each bullet screen pixel to obtain the bullet screen texture.

[0147] The rendering pipeline synthesis module can be used to synthesize the bullet screen texture and the stage light texture to form a light-illuminating bullet screen special effect.

[0148] The presentation screen is used to display the light illumination barrage special effects when the special effects triggering conditions are met during the playback of each video frame of the target video.

[0149] By way of further example, Fig. 9 As shown, combined Figure 8 The barrage processing system shown further illustrates the barrage processing method of the present application. When the user executes the barrage publishing operation to send the barrage and triggers the easter egg (i.e., the light irradiating the barrage special effect), the business layer of the electronic device determines that the special effect triggering conditions are met according to the special effect triggering conditions of the operation configuration module, and triggers the field control special effect barrage. The field control special effect barrage refers to a barrage with a light irradiating barrage special effect.

[0150] The rendering layer of the electronic device begins to draw the scene control special effects bullet screen, including:

[0151] In the process of playing each video frame of the target video, the CPU calls the first application programming interface (API) to obtain the audio rhythm information of the video frame. According to the audio rhythm information, the target stage lighting special effect parameters are determined, and the target stage lighting special effect parameters include: the number of stage lights, the flashing frequency of the light source, the irradiation direction of the stage lights, etc.

[0152] The CPU calls the second API to obtain the background color (i.e., the background color) of the video frame, determine the light source color as the background color, and update the target stage lighting special effect parameters. The updated target stage lighting special effect parameters include: the number of stage lights, the flashing frequency of the light source, the irradiation direction of the stage lights, the color of the light source, etc.

[0153] The GPU calls the image rendering engine to adopt a specific lighting algorithm, and uses the vertex shader and fragment shader of the GPU to calculate the target stage lighting special effect parameters to generate the desired stage lights.

[0154] Through the vertex shader and fragment shader of the GPU, the blank texture image is rendered according to the target stage light special effect parameters to obtain the stage light texture to make the background texture of the stage light.

[0155] The CPU determines the target transparency function of each image area according to the current audio rhythm level of the video frame (i.e., the target audio rhythm data) and the correspondence between the audio rhythm data and the transparency data. The target transparency function takes the playback time of the video frame as an independent variable.

[0156] Through the vertex shader and fragment shader of the GPU, for each barrage pixel, the block number of the image area where the barrage pixel is located is determined according to the pixel position corresponding to the barrage pixel in the video frame. The target transparency function of the image area indicated by the block number is obtained, and the transparency value of the barrage pixel is calculated to obtain the transparency value of each barrage pixel, and a new barrage texture is determined, which includes the transparency value and color value of each barrage pixel. Afterwards, the barrage texture is illuminated by the fragment shader of the GPU to obtain the processed barrage texture to generate an illuminated barrage texture. Among them, the processed barrage texture has a specified color, a flickering effect, and a lighting effect.

[0157] The bullet screen texture and the stage light texture are synthesized by the GPU to be mixed and rendered into a video frame with a bullet screen special effect of light illumination. The video frame is displayed on the screen of the electronic device to display the video frame and the bullet screen special effect of light illumination corresponding to the video frame in the playback interface.

[0158] In summary, the barrage rendering method provided by the embodiment of the present application obtains the barrage data and the target audio rhythm data corresponding to the target video during the playback of the target video, so that the target stage lighting special effect parameters corresponding to the target audio rhythm data can be determined according to the correspondence between the audio rhythm data and the stage lighting special effect parameters, and then the blank texture image is rendered according to the target stage lighting special effect parameters to obtain the stage lighting texture, so as to synthesize the barrage texture and the stage lighting texture included in the barrage data to form a lighting barrage special effect, so that the barrage data with the lighting barrage special effect can be displayed in the playback interface. Because the target stage lighting special effect parameters corresponding to the video frame change with the audio rhythm of the video frame. Therefore, in the lighting barrage special effect, the barrage can present a dynamic effect under the dynamic lighting that changes with the audio rhythm corresponding to the video frame, enriching the barrage display form, enhancing user interactive interest, and increasing the video viewing experience.

[0159] Please refer to Fig.10 , which shows a block diagram of a bullet screen rendering device provided by an embodiment of the present application. Fig.10 As shown, the bullet screen rendering device 1001 includes: an acquisition module 1001, a determination module 1002, a rendering module 1003 and a synthesis module 1004.

[0160] The acquisition module 1001 is used to acquire the bullet screen data and the target audio rhythm data corresponding to the target video during the target video playback process;

[0161] A determination module 1002 is used to determine target stage lighting special effect parameters corresponding to target audio rhythm data according to the correspondence between the audio rhythm data and the stage lighting special effect parameters;

[0162] The rendering module 1003 is used to render the blank texture image according to the target stage light special effect parameters to obtain the stage light texture;

[0163] The synthesis module 1004 is used to synthesize the bullet screen texture and the stage light texture included in the bullet screen data to form a light illuminating bullet screen special effect;

[0164] The display module 1005 is used to display the bullet screen data with the light illumination bullet screen special effect in the playback interface.

[0165] Optionally, the determination module 1002 is further configured to:

[0166] According to the correspondence between the audio rhythm data and the stage light parameters, determine the target stage light parameters corresponding to the target audio rhythm data, the stage light parameters indicate the virtual stage light settings with the play area of ​​the play interface as the stage, and the target stage light parameters include at least one of the following: the number of stage lights, the flashing frequency of the light source, and the irradiation direction of the stage lights;

[0167] According to the target stage light parameters, the target stage light special effect parameters are calculated, and the target stage light special effect parameters indicate the light color of each position on the stage when the virtual stage light is irradiated to the video display area of ​​the playback interface as the stage under the virtual stage light setting indicated by the target stage light parameters.

[0168] Optionally, the determination module 1002 is further configured to:

[0169] Get the background color of the video frame of the target video, and determine the light source color as the background color;

[0170] Adds the light source color to the target stage light parameters.

[0171] Optionally, the determination module 1002 is further configured to:

[0172] Determine the proportion of each color class in the video frame;

[0173] Among the various types of colors contained in the video frame, select the color with the highest proportion as the background color.

[0174] Optionally, the determination module 1002 is further configured to call an image rendering engine and calculate target stage light special effect parameters according to target stage light parameters using a lighting algorithm.

[0175] Optionally, the bullet screen texture includes the bullet screen color of each bullet screen pixel, and the bullet screen color is the background color of the video frame; the determination module 1002 is further used to determine the target transparent data corresponding to the target audio rhythm data according to the corresponding relationship between the audio rhythm data and the transparency data, and the transparency data includes the transparency value of each pixel position in the video frame,

[0176] And also used to determine the transparency value of each barrage pixel according to the pixel position corresponding to each barrage pixel in the video frame and the target transparency data;

[0177] The bullet-screen rendering device 1001 further includes: an updating module, which is used to update the bullet-screen texture, and the updated bullet-screen texture also includes a transparency value of each bullet-screen pixel;

[0178] The special effect processing module is used to perform lighting processing on the bullet screen texture to obtain the processed bullet screen texture.

[0179] Optionally, the transparency data includes a transparency function of each image area in the video frame; the determination module 1002 is further used to:

[0180] Determine the image area corresponding to each barrage pixel in the video frame according to the pixel position corresponding to each barrage pixel in the video frame;

[0181] The transparency value of each barrage pixel is determined according to the image area corresponding to each barrage pixel and the target transparency function of each image area corresponding to the target audio rhythm data.

[0182] Optionally, the transparency function takes time as an independent variable; the determination module 1002 is further used for:

[0183] Input the playback time of the video frame into the target transparency function of each image area to obtain the transparency value of each image area;

[0184] The transparency value of each barrage pixel is determined according to the image area corresponding to each barrage pixel and the transparency value of each image area.

[0185] Optionally, the determination module 1002 is further configured to perform edge smoothing transition processing on the transparency value of each image region, and update the transparency value of each image region.

[0186] Optionally, each image area in the video frame refers to multiple image areas obtained by dividing the video frame according to the moving direction of the barrage in the playback interface, and the transparency functions corresponding to adjacent image areas among the multiple image areas are different, but there are at least two non-adjacent image areas whose corresponding transparency functions are the same.

[0187] Optionally, the acquisition module 1001 is further configured to acquire the bullet screen data and the target audio rhythm data corresponding to the target video when a special effect triggering condition is met, and the special effect triggering condition includes at least one of the following:

[0188] The target video is a target-type video, and the audio rhythm of the target-type video meets the preset rhythm rule;

[0189] Responding to the triggering operation of the bullet screen control irradiated by light;

[0190] Respond to the bullet comment posting operation.

[0191] In summary, the barrage rendering device provided in the embodiment of the present application obtains the barrage data and the target audio rhythm data corresponding to the target video during the playback of the target video, so that the target stage lighting special effects parameters corresponding to the target audio rhythm data can be determined according to the correspondence between the audio rhythm data and the stage lighting special effects parameters, and then the blank texture image is rendered according to the target stage lighting special effects parameters to obtain the stage lighting texture, so as to synthesize the barrage texture and the stage lighting texture included in the barrage data to form a lighting barrage special effect, so that the barrage data with the lighting barrage special effect can be displayed in the playback interface. Because the target stage lighting special effects parameters corresponding to the video frame change with the audio rhythm of the video frame. Therefore, in the lighting barrage special effect, the barrage can present a dynamic effect under the dynamic lighting that changes with the audio rhythm corresponding to the video frame, enriching the barrage display form, enhancing user interactivity and interest, and increasing the video viewing experience. The embodiment of the present application also provides an electronic device, which can be the above-mentioned terminal or server. Fig.11 As shown, it includes a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104.

[0192] Memory 1103, used for storing computer programs;

[0193] The processor 1101 is used to implement any bullet screen rendering method provided in the embodiments of the present application when executing the program stored in the memory 1103.

[0194] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0195] The communication interface 1104 is used for communication between the above electronic device and other devices.

[0196] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0197] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0198] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor of a terminal, the steps of the bullet screen rendering method described in any of the above embodiments are performed.

[0199] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of the bullet screen rendering method described in any one of the above embodiments.

[0200] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0201] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "and / or" means and and or. For example, "A and / or B" means: A, B and A and B three situations. The term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0202] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0203] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A bullet screen rendering method, characterized in that: The method comprises: During the target video playback, obtaining barrage data and target audio rhythm data corresponding to the target video; Determining target stage lighting special effect parameters corresponding to the target audio rhythm data according to the corresponding relationship between the audio rhythm data and the stage lighting special effect parameters; Rendering the blank texture image according to the target stage light special effect parameters to obtain a stage light texture; The bullet screen texture and the stage light texture included in the bullet screen data are synthesized to form a light illuminating bullet screen special effect; The barrage data with the light illuminating barrage special effect is displayed in the playback interface.

2. The method according to claim 1, characterized in that The step of determining the target stage lighting special effect parameters corresponding to the target audio rhythm data according to the correspondence between the audio rhythm data and the stage lighting special effect parameters comprises: Determine, according to the correspondence between the audio rhythm data and the stage light parameters, the target stage light parameters corresponding to the target audio rhythm data, wherein the stage light parameters indicate virtual stage light settings with the playback area of ​​the playback interface as the stage, and the target stage light parameters include at least one of the following: the number of stage lights, the flashing frequency of the light source, and the irradiation direction of the stage lights; The target stage light special effect parameters are calculated according to the target stage light parameters, wherein the target stage light special effect parameters indicate the light color of each position on the stage when the virtual stage light illuminates the video display area of ​​the playback interface as the stage under the virtual stage light settings indicated by the target stage light parameters.

3. The method according to claim 2, characterized in that The step of determining the target stage lighting special effect parameters corresponding to the target audio rhythm data according to the correspondence between the audio rhythm data and the stage lighting special effect parameters also includes: Obtaining the background color of the video frame of the target video, and determining the light source color to be the background color; The light source color is added to the target stage light parameters.

4. The method according to claim 2, characterized in that: The obtaining the background color of the video frame of the target video includes: Determine the proportion of each type of color in the video frame; Among the various types of colors contained in the video frame, the color with the highest proportion is selected as the background color.

5. The method according to claim 2 or 3, characterized in that: The step of calculating the target stage light special effect parameter according to the target stage light parameter, wherein the target stage light special effect parameter indicates the light effect of the stage light, comprises: An image rendering engine is called, and a lighting algorithm is used to calculate the target stage light special effect parameters according to the target stage light parameters.

6. The method according to claim 1, characterized in that The bullet screen texture includes the bullet screen color of each bullet screen pixel, and the bullet screen color is the background color of the video frame of the target video; the method further includes: Determining target transparency data corresponding to the target audio rhythm data according to the corresponding relationship between the audio rhythm data and the transparency data, wherein the transparency data includes a transparency value of each pixel position in the video frame; Determine the transparency value of each barrage pixel according to the pixel position corresponding to each barrage pixel in the video frame and the target transparency data; Updating the bullet chat texture, wherein the updated bullet chat texture further includes a transparency value of each bullet chat pixel; Performing illumination processing on the bullet screen texture to obtain the processed bullet screen texture.

7. The method according to claim 6, characterized in that The transparency data includes a transparency function of each image area in the video frame; and determining the transparency value of each barrage pixel according to a pixel position corresponding to each barrage pixel in the video frame and the target transparency data, including: Determine, according to a pixel position corresponding to each barrage pixel in the video frame, an image area corresponding to each barrage pixel in the video frame; The transparency value of each barrage pixel is determined according to the image area corresponding to each barrage pixel and the target transparency function of each image area corresponding to the target audio rhythm data.

8. The method according to claim 7, characterized in that The transparency function takes time as an independent variable; the transparency value of each barrage pixel is determined according to the image area corresponding to each barrage pixel and the target transparency function of each image area corresponding to the target audio rhythm data, including: Inputting the playback time of the video frame into the target transparency function of each image area to obtain the transparency value of each image area; The transparency value of each barrage pixel is determined according to the image area corresponding to each barrage pixel and the transparency value of each image area.

9. The method according to claim 8, characterized in that The determining the transparency value of each barrage pixel according to the image area corresponding to each barrage pixel and the target transparency function of each image area corresponding to the target audio rhythm data further includes: An edge smoothing transition process is performed on each of the image regions with respect to the transparency value, and the transparency value of each of the image regions is updated.

10. The method according to any one of claims 7 to 9, characterized in that: Each image area in the video frame refers to a plurality of image areas obtained by dividing the video frame according to the moving direction of the barrage in the playback interface, and the transparency functions corresponding to adjacent image areas among the plurality of image areas are different, but there are at least two non-adjacent image areas whose corresponding transparency functions are the same.

11. The method according to claim 1, characterized in that: The acquiring of the bullet screen data and the target audio rhythm data corresponding to the target video includes: When a special effect triggering condition is met, the bullet screen data and the target audio rhythm data corresponding to the target video are obtained, and the special effect triggering condition includes at least one of the following: The target video is a target-type video, and the audio rhythm of the target-type video satisfies a preset rhythm rule; Responding to the triggering operation of the bullet screen control irradiated by light; Respond to the bullet comment posting operation.

12. A bullet screen rendering device, characterized in that: The device comprises: An acquisition module, used to acquire barrage data and target audio rhythm data corresponding to the target video during the playback of the target video; A determination module, used to determine the target stage lighting special effect parameters corresponding to the target audio rhythm data according to the correspondence between the audio rhythm data and the stage lighting special effect parameters; A rendering module, used for rendering the blank texture image according to the target stage light special effect parameters to obtain a stage light texture; A synthesis module, used for synthesizing the bullet screen texture and the stage light texture included in the bullet screen data to form a light illuminating bullet screen special effect; A display module is used to display the barrage data with the light illuminating barrage special effect in the playback interface.

13. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, configured to implement any one of the methods described in claims 1 to 11 when executing a program stored in a memory.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images