Methods, apparatus, devices, media, and program products for special effect generation

Abstract

Embodiments of the present disclosure provide a method, device, equipment, medium and program product for special effect generation. The method comprises: acquiring a plurality of reference part key points, the plurality of reference part key points being extracted from a reference image of a reference part and the part shape corresponding to the plurality of reference part key points matching a shape of a region in a page where a special effect is to be rendered; generating special effect content based on the plurality of reference part key points and a special effect processing algorithm corresponding to the special effect, the special effect content matching the part shape; and presenting the special effect content in the region of the page.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for generating special effects. Background Technology

[0002] More and more applications are now offering live streaming functionality. During a live stream, viewers can interact with the streamer to enhance the interactive experience and increase engagement between the streamer and viewers. For example, live streams can create interactive events (such as live chat sessions or game streams), allowing viewers to engage in various activities. Viewers can send virtual resources, such as special effects and virtual gifts, to the streamer or other participants to improve the interactive experience. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for generating special effects is provided. The method includes: acquiring multiple reference part keypoints, wherein the multiple reference part keypoints are extracted from a reference image of the reference parts and the shape of the parts corresponding to the multiple reference part keypoints matches the shape of an area on a page for which a special effect is to be rendered; generating special effect content based on the multiple reference part keypoints and a special effect processing algorithm corresponding to the special effect, wherein the special effect content matches the part shape; and presenting the special effect content in an area of ​​the page.

[0004] In a second aspect of this disclosure, an apparatus for generating special effects is provided. The apparatus includes: an acquisition module configured to acquire a plurality of reference part keypoints, wherein the plurality of reference part keypoints are extracted from a reference image of the reference parts and the shape of the parts corresponding to the plurality of reference part keypoints matches the shape of an area on a page to be rendered with a special effect; a generation module configured to generate special effect content based on the plurality of reference part keypoints and a special effect processing algorithm corresponding to the special effect, wherein the special effect content matches the shape of the parts; and a presentation module configured to present the special effect content in an area of ​​the page.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The medium stores computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0008] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A schematic diagram illustrating a process for generating special effects according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram illustrating an example of extracting key points of a reference region according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic diagram illustrating examples of coordinate transformations according to some embodiments of the present disclosure is shown;

[0014] Figure 5 Example interfaces according to some embodiments of this disclosure are shown;

[0015] Figure 6 A flowchart illustrating a process for generating special effects according to some embodiments of this disclosure is shown;

[0016] Figure 7 A schematic structural block diagram of an apparatus for special effects generation according to some embodiments of the present disclosure is shown; and

[0017] Figure 8 An electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0020] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0021] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0022] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0023] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0024] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0026] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In environment 100, server 130 is located, and one or more users 110-1, 110-2, 110-3, ..., 110-N can watch live streams (sometimes referred to as "live sessions") and participate in live interactive events through their respective associated terminal devices 120-1, 120-2, ..., 120-N. For ease of discussion, users 110-1, 110-2, ..., 110-N can be collectively referred to as user 110 or individually, and terminal devices 120-1, 120-2, ..., 120-N can be collectively referred to as terminal device 120 or individually. Among all users 110, one or more users, such as user 110-1, can initiate a live stream through their associated terminal device 120-1.

[0027] In some scenarios, the user 110-1 who initiates the live stream can also be referred to as the streamer, streamer user, streamer participant, streamer, or stream administrator. In some embodiments, the participants in the live interactive event may include the streamer user 110-1 and one or more other participants 110-N. Other participants 110-N may be guest participants who do not start their own live streams but join the current streamer participant's live interactive event (e.g., a live chat). In some embodiments, multiple streamer users who have started their own live streams may be allowed to interact with each other, so the participants in the live interactive event may include multiple streamer participants. Additionally, other users 110 may also include viewers in the live stream (also referred to as audience users).

[0028] In some embodiments, the terminal device 120 may have an application installed that can provide live streaming services, or may have access to a website that can provide live streaming services. The user 110 can operate the terminal device 120 to access the corresponding application or website. Accordingly, the terminal device 120 may present a corresponding live streaming interface, which may provide live streaming content, such as audio live streaming content or video live streaming content.

[0029] In some embodiments, the terminal device 120 can also communicate with the server 130 via the network 132 to provide live streaming services through the server 130. For example, the server 130 can provide services related to live streaming interactions, such as viewers sending special effects (also referred to as "virtual gifts") to the broadcaster and / or guests. In some embodiments, the server 130 can store data generated during the live streaming process, including live streaming interaction data generated during live streaming interactions.

[0030] Terminal device 120 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 120 may also support any type of user-facing interface (such as "wearable" circuitry). Server 110 can be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc. Server 130 may, for example, include computing systems / servers such as mainframes, edge computing nodes, computing devices in cloud environments, etc. Server-side equipment can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server-side equipment may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0031] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0032] Typically, virtual resources such as special effects and virtual gifts sent by viewers to users participating in interactive events within a live streaming environment (e.g., a live streaming room) are a hallmark of live streaming rooms. Users participating in interactive events can include the host who initiated the live stream and guests invited by the host. Accordingly, a live streaming room can also be called a "live chat room," "live session," etc.

[0033] Currently, in some live streaming formats that display body parts (such as the face and hands), special effects are displayed in a 3D rendering space based on these body parts of the streamer. This enhances interactivity in the live stream and increases interaction between the streamer and viewers. For example, after an effect is sent, the corresponding effect is displayed at the location of the face in the video. The position and wrapping effect of the effect are determined by detecting key facial features in the video frames and rendering them in 3D space. During the design process, designers can design the effect display based on a head model, ensuring the effect correctly wraps around the face in the virtual environment.

[0034] However, in live streaming scenarios that don't display body parts, such as audio-based live streams, the interface won't capture and present video containing body parts like the face; instead, it typically only displays the user's avatar. Therefore, it's impossible to display the body parts associated with special effects on the interface, nor can the effects be displayed correctly and in real-time. This makes it difficult to enhance the interactive experience of live streams without showing body parts, negatively impacting the user experience.

[0035] In view of this, according to embodiments of the present disclosure, a scheme for generating special effects is provided. According to this scheme, multiple reference part key points are obtained. These multiple reference part key points are extracted from reference images of the reference parts, and the shapes of the parts corresponding to the multiple reference part key points are matched with the shapes of the areas on the page where the special effects are to be rendered. Based on the multiple reference part key points and the corresponding special effects processing algorithm, special effects content is generated, and the special effects content matches the part shapes. Then, the special effects content is presented in the area of ​​the page.

[0036] In this way, in live streaming scenarios where body parts are not displayed, multiple reference body part key points are identified, ensuring that the shapes outlined by these key points match the shapes of the areas on the page where special effects are to be rendered. Based on these key points, the generated special effects content can match the body part shapes, thus rendering good special effects on the page. This allows for the reuse of existing special effects in live streaming scenarios that can display user body parts, and it also allows the special effects to fit the page area, satisfying the presentation requirements of different special effects in different display positions on the page.

[0037] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0038] Figure 2 A schematic diagram of a process 200 for special effects generation according to some embodiments of the present disclosure is shown. For ease of discussion, reference will be made to... Figure 1 These embodiments are described in the context of environment 100. In some examples, these embodiments may be implemented in... Figure 1 The server is located at 130. In other examples, these embodiments may also be implemented at terminal device 120, or through the cooperation of terminal device 120 and server 130. The following specific embodiments are exemplified by implementation at terminal device 120.

[0039] like Figure 2 As shown, the terminal device 120 acquires multiple reference part key points (210). The multiple reference part key points are extracted from the reference image 213 of the reference part (215). The shape of the part corresponding to the multiple reference part key points matches the shape of the area of ​​the special effect to be rendered on the page.

[0040] As an example, the reference part may be a part of a person, animal, robot, etc. The part may be, for example, a face, torso, legs, etc. It should be understood that the reference part can be any suitable part of any object having body parts. Multiple reference part key points may be key points extracted from key parts of the reference parts in reference image 213. In some embodiments, the multiple reference part key points include multiple reference facial key points, and the reference image includes a reference image of a reference face, and the part shape corresponding to the multiple reference part key points includes the facial shape corresponding to the multiple reference facial key points.

[0041] Taking a human face as a reference area as an example, key facial features could include the eyes, nose, mouth, and facial contours. Multiple reference area keypoints could include, for example, 106 or other numbers of keypoints. It should be understood that these are merely examples and not intended as any limitation. Of course, one or more keypoints can be extracted for each key feature. Furthermore, the shapes of the areas corresponding to multiple reference area keypoints can be adjusted to match the shape of the area on the page where the effect will be rendered. For ease of understanding, the following will combine... Figure 3 We will discuss in detail how to adjust the shape of the parts corresponding to multiple reference key points and how to extract multiple reference key points.

[0042] It should be noted that this article uses a human face as an example to discuss the embodiments of this disclosure. However, it should be understood that in other embodiments, the face may also include other faces besides human faces, such as animal faces, robot faces, etc.

[0043] In some embodiments, the page to be rendered may include a live streaming interface. The live streaming interface can be presented via a terminal device 120. The terminal device 120 may have an application installed that provides live streaming functionality. For example, the application may support the streamer in creating live interactive events, inviting viewers to participate as guests, and allowing viewers (also known as live streamers) to watch the live stream. Furthermore, during live interactive events, viewers can interact with the streamer and / or guests; for example, viewers can send virtual resources such as special effects or virtual gifts to the streamer and / or guests. For live streams such as multi-person live streams, radio-style live streams, and voice Q&A live streams, the streamer and / or guests' faces may not be displayed on the corresponding live streaming interface; instead, their avatars may be displayed. For ease of discussion, this type of live stream can be simply referred to as a faceless live stream. Of course, in addition to user identifiers like avatars, other user identifiers such as text identifiers and image identifiers may also be displayed on the live streaming interface.

[0044] It should be understood that the page for rendering special effects may also include any suitable interface other than the live streaming interface. The rendered special effects can be displayed on user 110's terminal device 120. In other embodiments, server 130 may provide the rendered special effects to terminal device 120 for display.

[0045] In some embodiments, the specific area on the page where the special effects are to be placed may include an area for displaying the user's avatar. For example, in a live streaming interface, the area for displaying the user's avatar may be defined as a circular area or other shapes. For the faceless live streams discussed above, in order to display the same or similar special effects as those displayed on a face, when rendering in a page or area without a face, multiple reference body key points can be defined first. The shapes of the body parts corresponding to these facial key points are then matched with the shapes of the area where the special effects are to be rendered, such as matching the shape of the user's avatar in the live streaming interface. This allows for the direct reuse of existing special effects used in live streaming interactive events with faces, enabling the use of these effects in some video-less scenarios without modification. By matching the shapes of the body parts corresponding to multiple reference body key points with the shapes of the area where the special effects are to be rendered on the page, the special effects can fit the user's avatar, thus still presenting high-quality special effects in such faceless live streaming scenarios.

[0046] The embodiments of this disclosure can be implemented not only for rendering special effects on user avatars in a live streaming interface, but also for rendering special effects in any appropriate area of ​​any other appropriate interface (e.g., on any appropriate page that includes user avatars, or in any appropriate area of ​​a page that does not necessarily include user avatars).

[0047] Continue to refer to Figure 2 In some embodiments, in order to obtain multiple reference feature key points, the terminal device 120 may determine a reference image 213 for extracting the multiple reference feature key points. Then, based on the shape of the region in the page, the terminal device 120 may perform adjustments (214) on the region in the reference image 213 to make the adjusted facial region shape match the shape of the region. Furthermore, the terminal device 120 can extract multiple reference feature key points from the adjusted region (215).

[0048] Figure 3 A schematic diagram of Example 300 for extracting key points of a reference region according to some embodiments of the present disclosure is shown. Example 300 can be implemented in Figure 1 In environment 100. Example 300 can be implemented through procedure 200.

[0049] Example 300 includes an example of a reference image 213 for a reference region. Reference image 213 contains an example face. As an example, the shape of the example face in reference image 213 can be an initial, unadjusted shape. Taking the area on the page that is the user's avatar in the live stream interface as an example, if the shape of the area is circular, the face in reference image 213 can be adjusted to obtain a face like the one in example image 310. For ease of understanding, the adjusted face is presented in example image 310 with circle 315 as a reference. It should be understood that the adjustment of the face in reference image 213 is discussed here only with the example of a circular area shape. If the shape of the specific area to be rendered in the actual application is a square, triangle, or other shape, the face will be adjusted accordingly to match the square, triangle, or other shape.

[0050] By adjusting the shape of a reference part in the reference image to match the shape of the region, the shapes of the parts corresponding to multiple key points extracted from the adjusted part can conform to the shape of the region. For example, in Example 300, by performing a key point extraction operation on the adjusted part in Example Image 310, the shapes of the parts corresponding to multiple example reference key points in Example Image 320 are also circular. This is beneficial for rendering visually appealing effects on page areas.

[0051] Return to reference Figure 2 In some embodiments, to determine the reference image 213 of the reference region, the terminal device 120 may acquire an initial image 211 containing the region. Then, the terminal device 120 may scale (212) the initial image 211 to obtain the reference image 213 of the reference region. The distances between reference points of the region and multiple edges of the reference image 213 of the reference region may each satisfy a distance threshold. For example, the initial image 211 may include an initial facial image. For example, the reference points of the face may include any suitable reference points such as the center of the eyebrows or the center point of the face.

[0052] Because special effects are designed to include entrance and exit animations, or to include other pattern elements that need to be displayed together, a safe distance needs to be maintained around the reference image 213 when generating the reference part. To maintain this safe distance, the initial image 211 needs to be scaled.

[0053] Combination Figure 3After scaling the initial image 211, a scaled initial image 211' is obtained, as shown in the reference image 213 of the reference part. After scaling the initial image 211, the position coordinates of the reference point of the part within the reference image 213 can be determined. For example, it can be required that the center point of the part in the scaled initial image 211' is equidistant from both the left and right edges of the reference image 213. As another example, taking the face as an example, it can be required that the distance between the center of the eyebrows and the upper edge of the reference image 213 meets a preset distance threshold. It should be understood that these are merely examples and are not intended to impose any limitations. The distance threshold can be defined by any suitable value based on the size of the reference image 213 of the reference part and the desired special effect; no restrictions are imposed here.

[0054] In some embodiments, for a reference image 213 of a reference region, the position of at least one of a plurality of reference region key points in the reference image 213 of the reference region may conform to a corresponding positional condition. As an example, the positional condition may indicate a predetermined position of at least one reference region key point in the reference image of the reference region. For example, the positional condition may include the reference region key point corresponding to the tip of the nose being located at the center of the reference image 213 of the reference region. As another example, the positional condition may include the T-shaped facial region centered on the eyes and nose being located in the central region of the reference image 213 of the reference region. It should be understood that these are merely examples and are not intended to be limiting.

[0055] Alternatively or additionally, in some embodiments, for a reference image 213 of a reference region, the relative relationship between the distance between at least two of the multiple reference region key points and the size of the reference image 213 of the reference region may conform to corresponding size conditions. As an example, the size conditions may indicate a predetermined relative relationship between the distance between at least two reference region key points and the size of the reference image 213 of the reference region. For example, the size conditions may include the distance between reference region key points corresponding to both ears being one-third of the width of the reference image 213 of the reference region. As another example, the size conditions may include the distance between the reference region key point corresponding to the highest point of the facial hairline and the reference region key point corresponding to the lowest point of the chin being one-third of the height of the reference image 213 of the reference region. It should be understood that these are merely examples and are not intended to be limiting; specific size ratios can be configured according to actual application needs.

[0056] Alternatively or additionally, in some embodiments, when selecting or generating the reference image 213 of the reference region, the image quality of the face in the reference image 213 of the reference region can meet quality conditions. As an example, quality conditions may include predetermined requirements for the image quality of the face in the reference image 213 of the reference region. For example, quality conditions may include that the facial region in the reference image 213 of the reference region needs to conform to the special effects display effect. As another example, quality conditions may include that multiple reference region key points of the face in the reference image 213 of the reference region can be identified by a special effects processing algorithm to obtain corresponding facial information.

[0057] Therefore, in Example 300, in the extraction results of multiple reference part key points in Example Image 320, the virtual face corresponding to the multiple reference part key points is in the center, which can perfectly match the circular user avatar, and achieve a better overlay of the interactive effect on the user avatar.

[0058] Return to reference Figure 2 Based on multiple reference body key points and corresponding special effects processing algorithms 220, the terminal device 120 generates special effects content (230). The special effects content matches the body shape. As an example, the special effects content can be associated with the facial contours corresponding to multiple reference panel key points. For example, the special effects content could be a swinging tiger head hat, a rotating headscarf, a headband with swaying rabbit ears, etc., without limitation. These special effects content can also be applied to existing live interactive events that present faces.

[0059] As an example, the special effects processing algorithm 220 can determine how to match special effects to key points of multiple reference parts, thereby generating special effects content. Accordingly, the special effects processing algorithm 220 can also be applied to the generation of special effects content in existing live interactive events that present faces.

[0060] In some embodiments, when generating special effects content, the terminal device 120 can generate at least one frame of special effects image based on multiple reference key points and special effects processing algorithms. Each frame of special effects image may include a portion of the special effects content to be generated, and the portion of the special effects content may match the shape of the part. Then, based on at least one frame of special effects image, the terminal device 120 can generate the special effects content.

[0061] As an example, special effects content can be in the form of animation, presenting dynamic effects. Therefore, for dynamic special effects, multiple frames of special effects images can be generated when generating special effects content, and then combined to form dynamic special effects content.

[0062] Continue to refer to Figure 2The terminal device 120 displays the special effects content in area (240) of the page. Since the rendering positioning of the special effects content may be based on the parts defined by reference key points, and considering that the positioning coordinate systems of multiple reference key points may differ from the positioning coordinate system of the page used for the special effects content, it is also necessary to implement position transformation from multiple reference key points and the special effects content to the rendering area on the page to ensure that the special effects content can be presented in the correct position on the page. The following will combine... Figure 4 Let's discuss in detail the coordinate transformation and the positioning of key points and special effects content for multiple reference parts.

[0063] Figure 4 A schematic diagram of an example 400 for coordinate transformation according to some embodiments of the present disclosure is shown. Example 400 can be implemented in Figure 1 In environment 100. Example 400 can be implemented through process 200. In example 400, the circular shape of region 405 is merely an example; in actual applications, the shape of region 405 can be any suitable shape, and no limitation is intended here. Furthermore, the values ​​regarding coordinates and diameter in example 400 are also merely examples, shown only for ease of understanding, and are not intended to impose any limitations.

[0064] like Figure 4 As shown, in some embodiments, in order to present special effects content in area 405 of page 401, based on the coordinate transformation relationship between the first coordinate system 410 corresponding to multiple reference key points and the second coordinate system 420 of page 401, and the position of area 401 in page 401, the terminal device 120 can present special effects content in area 405 of page 401. That is, in order to render special effects content associated with multiple reference key points at a specified position on page 401, it is necessary to convert the second coordinate system 420 corresponding to the page into the first coordinate system 410 of the rendering space.

[0065] In some embodiments, in order to determine the coordinate transformation relationship, the terminal device 120 can determine the coordinate transformation relationship between the first coordinate system 410 and the second coordinate system 420 based on the size of the page 401, the position of the region 405 in the page 401, and the size of the region 405.

[0066] As an example, we can first obtain the coordinates (x, y) of region 405 on page 401 in the second coordinate system 420, the diameter of region 405, and the width and height of page 401. Then, by calculating the ratio of coordinate x to page width and the ratio of coordinate y to page height, we can obtain the normalized coordinates of x and y. By calculating the ratio of diameter to page width and the ratio of diameter to page height, we can obtain the normalized diameter represented by the values ​​in (0, 1). Coordinate transformations can then be performed.

[0067] In some embodiments, to further present special effects content in region 405 of page 401, based on coordinate transformation relationships and the position of region 405 on page 401, terminal device 120 can determine the positional mapping relationship of region 405 between the first coordinate system 410 and the second coordinate system 420. Based on the positional mapping relationship of region 405 and the relative positional relationship between multiple reference key points in the first coordinate system 410 and region 405, terminal device 120 can present special effects content in the region of page 401 located in the second coordinate system 420. Thus, the positioning of multiple reference key points and special effects content on the page can be achieved.

[0068] Figure 5 Example interfaces 500 according to some embodiments of the present disclosure are shown. Interface 500 may be the example interface presented on page 401. It should be understood that the interfaces shown in the drawings are merely examples, and various interface designs may exist in practice. The various graphic elements in the interface may have different arrangements and different visual representations, one or more of which may be omitted or replaced, and one or more other elements may also be present. The embodiments of the present disclosure are not limited in this respect.

[0069] like Figure 5 As shown, in live streaming interactive events without specific body parts, such as in a co-hosted live stream, the special effect content 520 sent by viewer B to the host can be displayed in the area of ​​the host's avatar 510. Similarly, viewer B can also send special effects to guest users in a co-hosted live stream, thus displaying the corresponding special effects in the area of ​​the guest user's avatar.

[0070] In the step of rendering part effects in a live-stream scene without visible parts, this embodiment uses virtual part texture images to represent parts that may appear in the video. However, this step requires algorithmic computation and consumes machine resources. To improve efficiency, the results obtained from the part keypoint algorithm can be saved in advance and input into the effects package, thereby saving the time spent recognizing part keypoints. Alternatively, by using the initial image to determine reference images for reference parts and extracting multiple reference part keypoints, effects rendering in a live-stream scene without visible reference part keypoints can be achieved.

[0071] Therefore, the embodiments of this disclosure can place virtual body parts on the page where they need to be displayed in a live streaming scenario without body parts, and provide them as input to the special effects processing system to achieve the purpose of rendering facial special effects in a live streaming scenario without body parts. This allows for the reuse of existing special effects in live streaming scenarios that can display user body parts, and also allows the special effects to fit the area of ​​the page.

[0072] Figure 6 A flowchart of a process 600 for special effects generation according to some embodiments of the present disclosure is shown. Process 600 may be implemented at server 130, at terminal device 120, or through collaboration between terminal device 120 and server 130. The following example of process 600 is implemented at terminal device 120.

[0073] In frame 610, terminal device 120 acquires multiple reference part key points. The multiple reference part key points are extracted from the reference image of the reference part, and the shape of the part corresponding to the multiple reference part key points matches the shape of the area of ​​the special effect to be rendered on the page.

[0074] Within frame 620, based on multiple reference key points and corresponding special effects processing algorithms, terminal device 120 generates special effects content that matches the shape of the part.

[0075] In frame 630, terminal device 120 displays special effects content in the area of ​​the page.

[0076] In some embodiments, obtaining multiple reference region key points includes: determining a reference image of a reference region for extracting the multiple reference region key points; adjusting the face in the reference image of the reference region based on the shape of a region in the page so that the shape of the adjusted face matches the shape of the region; and extracting the multiple reference region key points from the adjusted face.

[0077] In some embodiments, determining a reference image of a reference region includes: acquiring an initial image containing a face; scaling the size of the initial image to obtain a reference image of the reference region, wherein the distances between reference points of the face and multiple edges of the reference image of the reference region in the reference image of the reference region respectively satisfy distance thresholds.

[0078] In some embodiments, the reference image of the reference region meets at least one of the following requirements: the position of at least one of the multiple reference region key points in the reference image of the reference region meets the corresponding position condition; the relative relationship between the distance between at least two of the multiple reference region key points and the size of the reference image of the reference region meets the corresponding size condition; and the image quality of the face in the reference image of the reference region meets the quality condition.

[0079] In some embodiments, presenting special effects content in the area of ​​the page includes: presenting special effects content in the area of ​​the page based on the coordinate transformation relationship between the first coordinate system corresponding to multiple reference key points and the second coordinate system corresponding to the page, and the position of the area in the page.

[0080] In some embodiments, the coordinate transformation relationship is determined by: determining the coordinate transformation relationship between the first coordinate system and the second coordinate system based on the size of the page, the position of the region on the page, and the size of the region.

[0081] In some embodiments, presenting special effects content in an area of ​​the page further includes: determining a positional mapping relationship between a first coordinate system and a second coordinate system for the area based on coordinate transformation relationships and the position of the area on the page; and presenting special effects content in an area of ​​the page located in the second coordinate system based on the positional mapping relationship for the area and the relative positional relationship between multiple reference key points in the first coordinate system and the area.

[0082] In some embodiments, generating special effects content includes: generating at least one frame of special effects image based on multiple reference part key points and special effects processing algorithms, each frame of special effects image including a portion of the special effects content to be generated, the portion of special effects content matching the shape of the part; and generating special effects content based on at least one frame of special effects image.

[0083] In some embodiments, the page includes a live streaming interface, and / or includes an area in the page for displaying user avatars.

[0084] Figure 7 A schematic structural block diagram of an apparatus 700 for generating object effects according to some embodiments of the present disclosure is shown. The apparatus 700 may be implemented as or included in a server 130 or a terminal device 120. The various modules / components in the apparatus 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0085] The apparatus 700 includes an acquisition module 710 configured to acquire multiple reference part key points, wherein the multiple reference part key points are extracted from a reference image of the reference part and the shape of the part corresponding to the multiple reference part key points matches the shape of the area to be rendered in the page; a generation module 720 configured to generate special effect content based on the multiple reference part key points and the special effect processing algorithm corresponding to the special effect, wherein the special effect content matches the part shape; and a presentation module 730 configured to present the special effect content in the area of ​​the page.

[0086] In some embodiments, the acquisition module 710 is further configured to determine a reference image of a reference region for extracting multiple reference region key points; perform adjustments on the face in the reference image of the reference region based on the shape of the region in the page so that the shape of the region of the adjusted face matches; and extract multiple reference region key points from the adjusted face.

[0087] In some embodiments, the apparatus 700 is further configured to acquire an initial image containing a face; and to scale the size of the initial image to obtain a reference image of a reference region, wherein the distances between reference points of the face and multiple edges of the reference image of the reference region in the reference image of the reference region respectively satisfy distance thresholds.

[0088] In some embodiments, the reference image of the reference region meets at least one of the following requirements: the position of at least one of the multiple reference region key points in the reference image of the reference region meets the corresponding position condition; the relative relationship between the distance between at least two of the multiple reference region key points and the size of the reference image of the reference region meets the corresponding size condition; and the image quality of the face in the reference image of the reference region meets the quality condition.

[0089] In some embodiments, the presentation module 730 is further configured to present the special effects content in an area of ​​the page based on the coordinate transformation relationship between the first coordinate system corresponding to multiple reference key points and the second coordinate system corresponding to the page, as well as the position of the area in the page.

[0090] In some embodiments, the apparatus 700 further includes a coordinate transformation relationship determination module, configured to determine the coordinate transformation relationship between the first coordinate system and the second coordinate system based on the size of the page, the position of the region in the page, and the size of the region.

[0091] In some embodiments, the presentation module 730 is further configured to determine the positional mapping relationship between the first coordinate system and the second coordinate system for the region based on the coordinate transformation relationship and the position of the region on the page; and to present the special effects content in the region of the page located in the second coordinate system based on the positional mapping relationship for the region and the relative positional relationship between multiple reference key points in the first coordinate system and the region.

[0092] In some embodiments, the generation module 720 is further configured to generate at least one frame of special effects image based on multiple reference part key points and special effects processing algorithms, each frame of special effects image including a portion of the special effects content to be generated, the portion of the special effects content matching the part shape; and to generate special effects content based on at least one frame of special effects image.

[0093] In some embodiments, the page includes a live streaming interface, and / or includes an area in the page for displaying user avatars.

[0094] The modules and / or units included in device 700 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more modules and / or units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the modules and / or units in device 700 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0095] Figure 8 A block diagram is shown illustrating an electronic device 800 in which one or more embodiments of the present disclosure may be implemented. It should be understood that... Figure 8 The electronic device 800 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 1 Terminal equipment 120 or Figure 7 The device 700 can be implemented as or included in one or more electronic devices 800.

[0096] like Figure 8 As shown, electronic device 800 is in the form of a general-purpose computing device. Components of electronic device 800 may include, but are not limited to, one or more processors or processing units 810, memory 820, storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. Processing unit 810 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 820. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 800.

[0097] Electronic device 800 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 820 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 830 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 800.

[0098] Electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 8 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 820 may include computer program product 825 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0099] The communication unit 840 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 800 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 800 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0100] Input device 850 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 860 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 800 can also communicate with one or more external devices (not shown) via communication unit 840 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 800, or with any device that enables electronic device 800 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0101] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0102] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0103] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0104] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0106] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating special effects, comprising: Multiple reference part key points are obtained. The multiple reference part key points are extracted from the reference image of the reference part, and the shape of the part corresponding to the multiple reference part key points matches the shape of the area of ​​the special effect to be rendered on the page. Based on the key points of the multiple reference parts and the special effects processing algorithm corresponding to the special effects, special effects content is generated, and the special effects content matches the shape of the parts. as well as The special effects content will be displayed in the area of ​​the page.

2. The method according to claim 1, wherein obtaining multiple reference site key points includes: Determine the reference image used to extract key points of the plurality of reference regions; Based on the shape of the region on the page, adjustments are made to the parts in the reference image so that the shape of the adjusted parts matches the shape of the region; as well as Extract the key points of the multiple reference locations from the adjusted locations.

3. The method of claim 2, wherein determining the reference image comprises: Obtain an initial image containing the described region; The initial image is scaled to obtain the reference image, wherein the distances between the reference point of the region in the reference image and multiple edges of the reference image respectively satisfy distance thresholds.

4. The method of claim 3, wherein the reference image meets at least one of the following conditions: At least one of the key points of the plurality of reference parts is positioned in the reference image that meets the corresponding positional conditions. The relative relationship between the distance between at least two key points of the plurality of key points of the reference parts and the size of the reference image conforms to the corresponding size conditions. The image quality of the specified region in the reference image meets the quality requirements.

5. The method according to claim 1, wherein presenting the special effects content in the area of ​​the page comprises: Based on the coordinate transformation relationship between the first coordinate system corresponding to the key points of the multiple reference parts and the second coordinate system corresponding to the page, and the position of the region on the page, the special effect content is presented in the region of the page.

6. The method of claim 5, wherein the coordinate transformation relationship is determined by: Based on the size of the page, the position of the region on the page, and the size of the region, the coordinate transformation relationship between the first coordinate system and the second coordinate system is determined.

7. The method of claim 5, wherein presenting the special effects content in the area of ​​the page further comprises: Based on the coordinate transformation relationship and the position of the region on the page, determine the positional mapping relationship of the region between the first coordinate system and the second coordinate system; Based on the positional mapping relationship of the region and the relative positional relationship between the multiple reference key points and the region in the first coordinate system, the special effects content is presented in the region of the page located in the second coordinate system.

8. The method according to claim 1, wherein generating the special effects content includes: Based on the multiple reference key points and the special effects processing algorithm, at least one frame of special effects image is generated. Each frame of special effects image includes a portion of the special effects content to be generated, and the portion of special effects content matches the shape of the part. as well as The special effects content is generated based on the at least one frame of special effects image.

9. The method according to claim 1, wherein the page includes a live streaming interface corresponding to the live stream, and / or The area mentioned on the page includes a region for displaying the user's avatar.

10. An apparatus for generating special effects, comprising: The acquisition module is configured to acquire multiple reference part key points, which are extracted from the reference image of the reference part and the shape of the part corresponding to the multiple reference part key points matches the shape of the area of ​​the special effect to be rendered on the page. The generation module is configured to generate special effects content based on the key points of the multiple reference parts and the special effects processing algorithm corresponding to the special effects, wherein the special effects content matches the shape of the parts. as well as A presentation module is configured to present the special effects content in the area of ​​the page.

11. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processor.

12. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 9.

Images