A special effects processing method, device, server, storage medium and program product

Abstract

The present application discloses a special effects processing method, device, server, storage medium and program product, wherein the method comprises: in response to a control signal issued by a user, determining a target special effects identifier corresponding to the control signal; obtaining a target special effects material set corresponding to the target special effects identifier, wherein the target special effects material set includes a plurality of target special effects materials with timestamp information; obtaining a target special effects material with a synchronized timestamp with a current video frame from the target special effects material set; generating a special effects video frame with a current timestamp based on the target special effects material and the current video frame, thereby decoupling special effects processing from a user end, thereby reducing the computing power requirements on the user end while achieving better quality and more efficient special effects, and is particularly suitable for processing complex special effects with high computing power requirements.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a special effect processing method, a special effect processing device, a server, a computer-readable storage medium, and a computer program product. Background Art

[0002] In the process of real-time shooting, live video, on-demand, etc., in order to enrich the content of the video, special effects can be added to the video. In the related art, the method of adding special effects to the video is: the user terminal performs post-processing on the video to add special effects. When the implementation of the special effects to be added is more complex, the computing power of the user terminal is required to be higher. Summary of the invention

[0003] The present application provides a special effects processing method, device, server, storage medium and program product to solve the problem in the prior art that when a user terminal is used to perform relatively complex special effects processing on a video, high computing power is required of the user terminal.

[0004] According to a first aspect of the present application, a special effect processing method is provided, the method being applied in a server, the method comprising:

[0005] In response to a control signal sent by a user, determining a target special effect identifier corresponding to the control signal;

[0006] Acquire a target special effect material set corresponding to the target special effect identifier, wherein the target special effect material set includes a plurality of target special effect materials having timestamp information;

[0007] Acquire a target special effect material having a synchronized timestamp with the current video frame from the target special effect material set;

[0008] A special effects video frame with a current timestamp is generated based on the target special effects material and the current video frame.

[0009] According to a second aspect of the present application, a special effect processing device is provided, the device being arranged in a server, the device comprising:

[0010] A target special effect identifier determination module, configured to determine a target special effect identifier corresponding to a control signal sent by a user;

[0011] A target special effect material acquisition module, used to acquire a target special effect material set corresponding to the target special effect identifier, wherein the target special effect material set includes a plurality of target special effect materials with timestamp information;

[0012] A synchronous special effect material determination module, used to obtain a target special effect material having a synchronous time stamp with the current video frame from the target special effect material set;

[0013] The special effect processing module is used to generate a special effect video frame with a current timestamp based on the target special effect material and the current video frame.

[0014] According to a third aspect of the present application, a server is provided, the server comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method described in the first aspect above.

[0018] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method described in the first aspect when executed.

[0019] According to a fifth aspect of the present application, a computer program product is provided, the computer program product comprising computer executable instructions, the computer executable instructions being used to implement the method described in the first aspect when executed.

[0020] This embodiment performs special effects processing on video frames on the server side, thereby decoupling special effects processing from the user side, thereby achieving better quality and more efficient special effects while reducing the computing power requirements on the user side. It is particularly suitable for processing complex special effects that require high computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a special effects processing method provided in Example 1 of the present application;

[0023] Figure 2 is a flow chart of a special effects processing method provided in Embodiment 2 of the present application;

[0024] Figure 3 is an exemplary schematic diagram of a video frame currently to be processed with virtual scene special effects provided in Embodiment 2 of the present application;

[0025] Figure 4 This is a method provided by Example 2 of the present application. Figure 3 An exemplary schematic diagram of a special effect video frame obtained by performing virtual scene special effect processing on a video frame;

[0026] Figure 5 is a flow chart of a special effects processing method provided in Embodiment 3 of the present application;

[0027] Figure 6 is a flow chart of a special effects processing method provided in Embodiment 4 of the present application;

[0028] Figure 7 is a flow chart of a special effects processing method provided in Embodiment 5 of the present application;

[0029] Figure 8 It is a structural schematic diagram of a special effect processing device provided in Example 6 of the present application;

[0030] Fig. 9 This is a structural diagram of a server provided in Example 7 of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1This is a flowchart of a special effect processing method provided in Embodiment 1 of the present application. This method embodiment can be applied to a server, wherein the server may include a cloud server or a server cluster, which is not limited in this embodiment. Figure 1 As shown, this embodiment may include the following steps:

[0035] Step 101, in response to a control signal sent by a user, determining a target special effect identifier corresponding to the control signal.

[0036] Among them, special effect identifiers can be used to distinguish different special effects. The special effect identifier can be represented by, for example, a special effect name, a special effect number, a special effect identification number, etc., or a combination thereof.

[0037] In this embodiment, the target special effect identifier may be determined by a control signal sent by the user via the client.

[0038] In one example, the control signal may carry a target special effect identifier. The example scenario may be, for example: the client displays a special effect list to the user, and the user may select one or more special effects from the special effect list. The client may then send the special effect identifiers of the one or more special effects selected by the user as the target special effect identifier to the server.

[0039] In one example, the control signal may carry a required scene identifier, and the example scenario may be, for example: the user may input an applicable scene via the client, and after the server obtains the applicable scene input by the user, it obtains the scene feature of the scene, and then selects the special effect corresponding to the scene feature as the target special effect, and determines the target special effect identifier accordingly. For example, if the scene input by the user is a stage scene, the corresponding target special effect may be matched for the user as a stage scene special effect, and the special effect identifier of the special effect may be obtained as the target special effect identifier.

[0040] It should be noted that one target special effect corresponds to one target special effect identifier. The target special effect referred to in this embodiment can preferably be a more complex special effect generated by the modeling of the design team. Such special effects generally contain a large number of special effect elements. For example, the target special effect can be a professional virtual stage special effect driven by a virtual digital human dancing scene. Such special effects can include dozens of stage lights with different functions and camera scripts for dance points, etc. Such special effects require relatively high computing power. In general, it is difficult for the user end (such as mobile devices such as mobile phones) to meet such computing power requirements, resulting in the inability to achieve real-time effects. Therefore, this embodiment performs special effect rendering processing on the server side based on such special effects to meet the computing power requirements.

[0041] Step 102: Acquire a target special effect material set corresponding to the target special effect identifier, wherein the target special effect material set includes a plurality of target special effect materials having timestamp information.

[0042] In one implementation, the special effects material set can be represented in the form of a special effects material video, and each frame in the special effects material video can be called a special effects material frame, which is the special effects material with timestamp information mentioned in this step.

[0043] A special effects material video can be generated by the combined effect of a variety of different material elements. For example, in a professional virtual stage special effects video, the material elements may include but are not limited to: virtual digital human images, lighting effects, camera scripts, stage scene elements, other stage animations, etc.

[0044] In implementation, if the current server has a material storage area, the corresponding target special effect material set can be searched and read from the material storage area according to the target special effect identifier. If the current server does not have a material storage area, the target special effect material set corresponding to the target special effect identifier can be requested from other servers (such as a material server).

[0045] In one embodiment, step 102 may further include the following steps:

[0046] Step 102-1, determining a target material server corresponding to a target special effect identifier.

[0047] In one implementation, if there is only one material server, the material server may be used as the target material server.

[0048] If there are two or more material servers, that is, the material servers are embodied as a server cluster, then a material server storing a target special effect material set corresponding to the target special effect identifier can be determined from the material server cluster as the target material server. For example, an association relationship between the special effect identifier and the material server identifier can be generated in advance, and then the corresponding target material server can be determined by searching for the target special effect identifier from all the association relationships.

[0049] Alternatively, if the material server cluster is a server cluster of a master-slave architecture, the master material server may be used as the target material server.

[0050] Step 102-2: Generate a material request based on the target special effect identifier, and send the material request to the target material server.

[0051] Step 102 - 3 , receiving a target special effect material set returned by the target material server based on the material request.

[0052] After the target material server is determined, the material request generated based on the target special effect identifier can be sent to the target material server, and the target material server will search the target special effect identifier from the material library according to the target special effect identifier to find and read the target special effect material set, and then return the target special effect material set to the server responsible for implementing this embodiment.

[0053] Step 103: Acquire a target special effect material having a synchronized timestamp with the current video frame from the target special effect material set.

[0054] Among them, if the target special effect material set is embodied in the form of a special effect material video, the frame rate of the special effect material video is the same as that of the current video frame, so that the two have the same timestamp information.

[0055] During implementation, you can first obtain the timestamp information of the current video frame as the target timestamp information, and then search the special effects material frame corresponding to the target timestamp information from the special effects material video as the target special effects material, so that the target special effects material and the current video frame have a synchronized timestamp.

[0056] Step 104: Generate a special effect video frame with a current timestamp based on the target special effect material and the current video frame.

[0057] After the target special effect material at the current moment is determined, the target special effect material can be merged with the current video frame to obtain the special effect video frame at the current moment.

[0058] This embodiment performs special effects processing on video frames on the server side, thereby decoupling special effects processing from the user side, thereby achieving better quality and more efficient special effects while reducing the computing power requirements on the user side. It is particularly suitable for processing complex special effects that require high computing power.

[0059] Embodiment 2

[0060] Figure 2 This is a flow chart of a special effect processing method provided in the second embodiment of the present application. This method embodiment describes a special effect processing scenario driven by a virtual digital human based on the first embodiment. Figure 2 As shown, this embodiment may include the following steps:

[0061] Step 201, in response to a control signal sent by a user, determining a target special effect identifier corresponding to the control signal.

[0062] Step 202: Acquire a target special effect material set corresponding to the target special effect identifier, wherein the target special effect material set includes a plurality of target special effect materials having timestamp information.

[0063] Step 203: Acquire a target special effect material having a synchronized timestamp with the current video frame from the target special effect material set.

[0064] Step 204, obtaining the pose information of the current video frame.

[0065] The posture information can be determined according to the driven parts of the virtual digital human, for example, the posture information can be whole body posture information or local posture information. Whole body posture information can include, for example, the position information and posture information of the head, face, neck, limbs, torso and other parts; local posture information is the position information and posture information of one or more local body parts.

[0066] In one embodiment, step 204 may further include the following steps:

[0067] The current video frame is input into a pre-generated posture recognition model, and the posture information output by the posture recognition model after posture recognition of the current video frame is obtained.

[0068] The posture recognition model may be a pre-trained deep learning model, and the posture recognition model may be used to perform posture detection on the video frame to obtain key point information of the current video frame as posture information. The key point information may exemplarily include coordinate positions of key points, posture angle information, etc.

[0069] Step 205: Determine the virtual image corresponding to the user.

[0070] The virtual image is a virtual digital person pre-set by the user. The virtual image is bound to the user identifier and can be driven by the user's position information.

[0071] Step 206: Generate a special effects video frame of the current timestamp based on the posture information, the virtual image and the target special effects material.

[0072] For example, suppose the current video frame is Figure 3 As shown, obtain Figure 3 After obtaining the position information of the key points of the human body such as the head, face, neck, limbs, and torso, the virtual image can be driven based on the position information, and the target special effect material (assuming that it is the material corresponding to the virtual stage scene) can be added to the virtual image. The generated special effect video frame corresponding to the video frame can be as follows Figure 4 shown.

[0073] It should be noted that Figure 3 and Figure 4 It is to illustrate the effect of driving the virtual image and adding special effects to the captured video frame, and is not used to limit the image size. In fact, the image size of the current video frame and the special effect video frame is the same. Figure 3 and Figure 4 It is the effect of graying the image. In practice, Figure 4 It has stage lighting, colors and other effects.

[0074] In one embodiment, step 206 may further include the following steps:

[0075] The posture information, virtual image and target special effect material are input into a pre-generated rendering model, which determines the material parameters according to the target special effect material, and uses the material parameters to add corresponding special effects to the virtual image driven by the posture information, and outputs special effect video frames.

[0076] The rendering model works on driving and rendering. Driving refers to driving the virtual image using posture information; rendering refers to using target special effects materials to render special effects on the scene where the virtual image is located. It should be noted that since most servers do not have display hardware, the rendering mentioned in this embodiment also has a very important off-screen (headless) feature.

[0077] This embodiment does not limit the driving and rendering methods of the rendering model. For example, the rendering model analyzes the target special effect material to determine the material elements contained in the target special effect material and the material parameters of each material element. Then, the virtual image is driven based on the posture information, and the analyzed material elements and material parameters are applied to the driven virtual image to generate a special effect video frame and output the special effect video frame.

[0078] This embodiment does not limit the training process of the rendering model. For example, the rendering parameters of the rendering model may include lighting parameters, material parameters of the virtual image, geometric shape parameters, etc. These rendering parameters are trained as explicit parameters that can be learned through training. After rendering through the preliminary rendering model, a special effect image is obtained, and the special effect image is compared with the real sample image to calculate the loss, and the material, geometric shape, lighting and other parameters are updated through back propagation. The end is an epoch of the training process, and the model training ends when the iteration reaches the specified epoch size.

[0079] In this embodiment, in the special effects processing scenario of driving a virtual digital human, the server performs posture detection and special effects analysis and processing, and then drives the virtual digital human and renders the special effects, thereby generating a special effects video frame of the virtual digital human in the special effects scenario. This embodiment can achieve the following effects:

[0080] 1) Low user-side computing power requirements to improve coverage and penetration, and effectively cover different user groups across platforms and terminals.

[0081] 2) Taking advantage of the huge computing power and scalable hardware of cloud machines, it is possible to produce relatively more complex or realistic stage scenes, lighting effects and other special effects scenes. However, such special effects scenes have higher requirements on algorithms or hardware on the user side, which avoids the user side from making a balance between "power consumption and effect" and avoids generating additional computing or rendering loads on the user side.

[0082] 3) Strong scalability. The entire driving and rendering process is implemented in the server, which is easy to adapt to multi-person interaction or competition. According to different application scenarios, it can be divided into multi-person games, virtual people interacting with the same scene, etc. It is easier to enhance interactivity than the rendering method of traditional applications.

[0083] Embodiment 3

[0084] Figure 5 This is a flowchart of a special effects processing method provided in Embodiment 3 of the present application. This method embodiment describes a scenario of performing special effects processing on an imported video file based on Embodiment 1 or Embodiment 2. Figure 5 As shown, this embodiment may include the following steps:

[0085] Step 301, in response to a control signal sent by a user, determining a target special effect identifier corresponding to the control signal.

[0086] Step 302: Acquire a target special effect material set corresponding to the target special effect identifier, wherein the target special effect material set includes a plurality of target special effect materials having timestamp information.

[0087] Step 303: Acquire a target special effect material having a synchronized timestamp with the current video frame from the target special effect material set, wherein the current video frame is a video frame in the video file.

[0088] The video file may be a video file triggered by a user and imported from outside.

[0089] Step 304: Generate a special effect video frame with a current timestamp based on the target special effect material and the current video frame.

[0090] Step 305: Encode the special effect video frames of each time stamp into a special effect video according to the time sequence, and return the special effect video to the current user.

[0091] When the user issues an export instruction, the special effects video can be exported to the user terminal.

[0092] This embodiment describes the process of performing special effects processing on the entire video. For example, a user can import a dance video through the client. After the dance video is uploaded to the server, the server decodes the dance video to obtain video frames, and then performs special effects processing and virtual digital human driving processing on each video frame, thereby generating a special effects video of a virtual digital human dancing on a virtual stage. The user can export the special effects video. In the whole process, all the user needs to do is import the video to be processed and export the special effects video, which effectively reduces the amount of calculation and resource occupation of the user.

[0093] Embodiment 4

[0094] Figure 6 This is a flowchart of a special effects processing method provided in Embodiment 4 of the present application. This method embodiment, based on Embodiment 1 or Embodiment 2, describes a scenario in which real-time special effects processing is performed on a live video stream. Figure 6 As shown, this embodiment may include the following steps:

[0095] Step 401, in response to a control signal sent by a host user, determining a target special effect identifier corresponding to the control signal.

[0096] Step 402: Acquire a target special effect material set corresponding to the target special effect identifier, wherein the target special effect material set includes a plurality of target special effect materials having timestamp information.

[0097] Step 403: Acquire a target special effect material having a synchronized timestamp with the current video frame from the target special effect material set, wherein the current video frame is a video frame corresponding to the live video stream.

[0098] In one example, the live video stream may include action footage of the current host user.

[0099] Step 404: Generate a special effect video frame of the current timestamp based on the target special effect material and the current video frame.

[0100] In this embodiment, if a virtual live broadcast is involved, the driven virtual image is the virtual image of the anchor user.

[0101] Step 405: Encode the special effect video frame into a special effect video stream.

[0102] During implementation, an audio and video encoder may be used to encode special effects video frames into special effects video streams. During encoding, an audio stream may also be obtained for encoding and merging operations. The obtained special effects video stream is an audio and video stream suitable for live broadcast of a virtual image.

[0103] Step 406: Push the special effect video stream to the host user and the audience users.

[0104] In one implementation, the special effects video stream can be transmitted back to the anchor user in real time through webRTC technology (Web Real-Time Communications, a real-time communication technology), so that the special effects video stream can be displayed on the anchor user's visual interface.

[0105] For each audience user, the special effects video stream can be uploaded to the Content Delivery Network (CDN) for distribution to the clients of different audience users for display on the audience side.

[0106] In one embodiment, after step 406, this embodiment may further include the following steps:

[0107] In response to receiving an adjustment signal sent by any user of the special effects video stream, an adjustment parameter corresponding to the adjustment signal is determined; the special effects video frame to be distributed to the user is adjusted according to the adjustment parameter, and the adjusted special effects video stream is sent to the user.

[0108] In practice, if the user who initiates the adjustment signal is a viewer user, the viewer user may be a viewer user with certain permissions. Specifically, viewer users may be divided into ordinary viewer users and authorized viewer users, and authorized viewer users have more operation permissions on the received video stream. For example, authorized viewer users have a panoramic and free observation angle to watch the video stream, and have the permission to switch the viewing angle; while ordinary viewer users do not have this permission. For another example, ordinary viewer users and authorized viewer users both have the permission to adjust the video viewing angle, but more viewing angles are open to authorized viewer users.

[0109] In addition, the user who initiates the adjustment signal may also be the anchor user, who may also adjust the received special effects video stream. To some extent, the anchor user and the authorized audience user have similar rights, or may even have more adjustment rights than the authorized audience user.

[0110] In one implementation, an adjustment toolbar may be included in the user interface for playing a video stream, and the adjustment toolbar may include adjustment tools that match the user's authority. The user triggers one or more of the adjustment tools to send an adjustment signal, and after the client detects the adjustment signal triggered by the user, the adjustment signal is sent to the current server. For example, the adjustment signal may include a viewing angle adjustment signal, and the viewing angle adjustment signal is used to indicate the viewing angle that the user wants to watch. The viewing angle adjustment signal may include viewing angle information such as front, back, left, and right, window size, and specific camera script triggering.

[0111] When the server receives an adjustment signal sent by a user, it can obtain the adjustment parameters corresponding to the adjustment signal. In one implementation, the correspondence between various viewing angle adjustment signals and corresponding adjustment parameters can be pre-configured on the server side, and the corresponding adjustment parameters can be obtained by searching for the currently received adjustment signal. In other implementations, the adjustment parameters corresponding to the current viewing angle adjustment signal can also be calculated by a pre-set underlying algorithm. Exemplarily, the adjustment parameters may include, but are not limited to, viewpoint position, viewing angle, and viewport size.

[0112] After the adjustment parameters are determined, the special effects video frame to be sent to the current user can be obtained, the special effects video frame to be sent to the current user can be adjusted according to the adjustment parameters, and the adjusted, encoded special effects video stream can be sent to the user. The adjustment may include, for example, switching the camera script to the camera script corresponding to the user-triggered viewing angle, adjusting the viewpoint position, viewing angle, and the size of the observation viewport.

[0113] In this embodiment, in a virtual live broadcast scene, the host user can drive the corresponding virtual image to move in a specified special effects scene when hosting, forming a live special effects video stream, and distribute the live special effects video stream to each audience user. At the same time, the host user can also see the live special effects video stream, thereby enriching the live broadcast scene and improving the live broadcast effect.

[0114] Embodiment 5

[0115] Figure 7 This is a flowchart of a special effects processing method provided in Embodiment 5 of the present application. This method embodiment, based on Embodiment 1 or Embodiment 2, describes another scenario of performing real-time special effects processing on a live video stream. Figure 7 As shown, this embodiment may include the following steps:

[0116] Step 501, in response to a control signal sent by an audience user, determining a target special effect identifier corresponding to the control signal.

[0117] Step 502: Acquire a target special effect material set corresponding to the target special effect identifier, wherein the target special effect material set includes a plurality of target special effect materials having timestamp information.

[0118] Step 503: Acquire a target special effect material having a synchronized timestamp with the current video frame from the target special effect material set, wherein the current video frame is a video frame corresponding to the live video stream.

[0119] In one example, the live video stream may include action footage of the current viewer user.

[0120] Step 504: Generate a special effect video frame with a current timestamp based on the target special effect material and the current video frame.

[0121] In this embodiment, if a virtual live broadcast is involved, the driven virtual image is the virtual image of the audience user.

[0122] Step 505: encode the special effect video frame into a special effect video stream.

[0123] Step 506: Send the special effects video stream to the viewer user, other viewer users, and the anchor user.

[0124] In one embodiment, after step 506, the following steps may also be included:

[0125] In response to receiving an adjustment signal sent by any user of the special effects video stream, an adjustment parameter corresponding to the adjustment signal is determined; the special effects video frame to be sent to the user is adjusted according to the adjustment parameter, and the adjusted special effects video stream is sent to the user.

[0126] This embodiment is different from the fourth embodiment in that it describes the scene of virtual driving and special effect rendering by the audience user with a virtual character from the audience user side. Together with the fourth embodiment, this embodiment can realize the interaction between the anchor user and the audience user in the virtual special effect scene, and even realize the driving and rendering live broadcast of multiple users in the same scene, which enhances the interactivity of the live broadcast and enriches the fun of the live broadcast.

[0127] Embodiment 6

[0128] Figure 8 This is a structural diagram of a special effect processing device provided in Embodiment 6 of the present application. The device is set in a server and may include the following modules:

[0129] The target special effect identifier determination module 601 is used to determine the target special effect identifier corresponding to the control signal in response to the control signal sent by the user;

[0130] A target special effect material acquisition module 602 is used to acquire a target special effect material set corresponding to the target special effect identifier, wherein the target special effect material set includes a plurality of target special effect materials with timestamp information;

[0131] A synchronous special effect material determination module 603 is used to obtain a target special effect material having a synchronous time stamp with the current video frame from the target special effect material set;

[0132] The special effect processing module 604 is used to generate a special effect video frame with a current timestamp based on the target special effect material and the current video frame.

[0133] In one embodiment, the special effect processing module 604 may further include the following modules:

[0134] A posture information acquisition module is used to obtain the posture information of the current video frame;

[0135] A virtual image determination module, used to determine the virtual image corresponding to the user;

[0136] The special effects video frame generation module is used to generate a special effects video frame of a current timestamp based on the posture information, the virtual image and the target special effects material.

[0137] In one embodiment, the special effect video frame generation module is specifically used for:

[0138] The posture information, the virtual image and the target special effect material are input into a pre-generated rendering model, the rendering model determines material parameters according to the target special effect material, and uses the material parameters to add corresponding special effects to the virtual image driven by the posture information, and outputs special effect video frames.

[0139] In one embodiment, the posture information acquisition module is specifically used to:

[0140] The current video frame is input into a pre-generated posture recognition model, and the posture information output by the posture recognition model after posture recognition of the video frame is obtained.

[0141] In one embodiment, the target special effect material acquisition module 602 is specifically used to:

[0142] Determining a target material server corresponding to the target special effect identifier;

[0143] Generate a material request based on the target special effect identifier, and send the material request to the target material server;

[0144] Receive a target special effect material set returned by the target material server based on the material request.

[0145] In one embodiment, the current video frame is a video frame in a video file;

[0146] The device may further include the following modules:

[0147] The special effect video generation module is used to encode the special effect video frames of each time stamp into a special effect video in time sequence after generating the special effect video frame of the current timestamp based on the target special effect material and the current video frame, and return the special effect video to the user.

[0148] In one embodiment, the current video frame is a video frame corresponding to a live video stream;

[0149] The device may further include the following modules:

[0150] A video encoding module, used for encoding the special effect video frame into a special effect video stream;

[0151] A first streaming module, configured to stream the special effects video stream to the anchor user and the audience user if the user is the anchor user;

[0152] The second streaming module is used to send the special effect video stream to the viewer user, other viewer users and the anchor user if the user is a viewer user.

[0153] In one embodiment, the device may further include an adjustment module, configured to:

[0154] In response to receiving an adjustment signal sent by any user of the special effect video stream, determining an adjustment parameter corresponding to the adjustment signal;

[0155] The special effects video frame to be sent to the user is adjusted according to the adjustment parameters, and the adjusted special effects video stream is sent to the user.

[0156] A special effects processing device provided in an embodiment of the present application can execute a special effects processing method provided in any one of embodiments 1 to 5 of the present application, and has functional modules and beneficial effects corresponding to the execution method.

[0157] Embodiment 7

[0158] Fig. 9 The schematic diagram of the structure of the server 10 that can be used to implement the method embodiment of the present application is shown. Fig. 9 As shown, the server 10 includes at least one processor 11, and a storage device connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the storage device stores one or more computer programs that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the server 10 can also be stored.

[0159] In some embodiments, the method in any one of Embodiments 1 to 5 may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the server 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method in any one of Embodiments 1 to 5 described above may be performed.

[0160] In some embodiments, the method in any one of embodiments one to five may be implemented as a computer program product, which includes computer executable instructions, which, when executed, are used to perform one or more steps of the method in any one of embodiments one to five described above.

Claims

1. A special effect processing method, characterized in that: The method is applied in a server, and the method comprises: In response to a control signal sent by a user, determining a target special effect identifier corresponding to the control signal; Acquire a target special effect material set corresponding to the target special effect identifier, wherein the target special effect material set includes a plurality of target special effect materials with timestamp information; wherein the special effect material set is represented in the form of a special effect material video, and each frame in the special effect material video is called a special effect material with timestamp information; Acquire a target special effect material having a synchronized timestamp with the current video frame from the target special effect material set; Generate a special effects video frame with a current timestamp based on the target special effects material and the current video frame; The step of acquiring a target special effect material set corresponding to the target special effect identifier includes: Determining a target material server corresponding to the target special effect identifier; Generate a material request based on the target special effect identifier, and send the material request to the target material server; Receive a target special effect material set returned by the target material server based on the material request.

2. The method according to claim 1, characterized in that: The generating a special effect video frame with a current timestamp based on the target special effect material and the current video frame includes: Get the pose information of the current video frame; Determining a virtual image corresponding to the user; Based on the posture information, the virtual image and the target special effect material, a special effect video frame of the current timestamp is generated.

3. The method according to claim 2, characterized in that The step of generating a special effects video frame at a current timestamp based on the posture information, the virtual image, and the target special effects material includes: The posture information, the virtual image and the target special effect material are input into a pre-generated rendering model, the rendering model determines material parameters according to the target special effect material, and uses the material parameters to add corresponding special effects to the virtual image driven by the posture information, and outputs special effect video frames.

4. The method according to claim 2, characterized in that: The obtaining of the pose information of the current video frame includes: The current video frame is input into a pre-generated posture recognition model, and the posture information output by the posture recognition model after posture recognition of the video frame is obtained.

5. The method according to claim 1, characterized in that The current video frame is a video frame in a video file; After generating a special effect video frame with a current timestamp based on the target special effect material and the current video frame, the method further includes: The special effect video frames of each time stamp are encoded into a special effect video in time sequence, and the special effect video is returned to the user.

6. The method according to claim 1, characterized in that The current video frame is a video frame corresponding to the live video stream; After generating a special effect video frame with a current timestamp based on the target special effect material and the current video frame, the method further includes: Encoding the special effects video frame into a special effects video stream; If the user is a host user, the special effects video stream is pushed to the host user and the audience users; If the user is a viewer user, the special effects video stream is sent to the viewer user, other viewer users and the anchor user.

7. The method according to claim 6, characterized in that The method further comprises: In response to receiving an adjustment signal sent by any user of the special effect video stream, determining an adjustment parameter corresponding to the adjustment signal; The special effects video frame to be sent to the user is adjusted according to the adjustment parameters, and the adjusted special effects video stream is sent to the user.

8. A special effect processing device, characterized in that: The device is arranged in a server, and comprises: A target special effect identifier determination module, configured to determine a target special effect identifier corresponding to a control signal sent by a user; A target special effect material acquisition module is used to acquire a target special effect material set corresponding to the target special effect identifier, wherein the target special effect material set includes a plurality of target special effect materials with timestamp information; wherein the special effect material set is represented in the form of a special effect material video, and each frame in the special effect material video is called a special effect material with timestamp information; A synchronous special effect material determination module, used to obtain a target special effect material having a synchronous time stamp with the current video frame from the target special effect material set; A special effect processing module, used for generating a special effect video frame of a current timestamp based on the target special effect material and the current video frame; The target special effect material acquisition module is specifically used for: Determining a target material server corresponding to the target special effect identifier; Generate a material request based on the target special effect identifier, and send the material request to the target material server; Receive a target special effect material set returned by the target material server based on the material request.

9. A server, characterized in that: The server comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising computer executable instructions, wherein the computer executable instructions are used to implement the method according to any one of claims 1 to 7 when executed.

Images