Method and device for interacting and making special effect, equipment and storage medium

Abstract

The embodiment of the invention relates to a method and device for interaction and special effect production, equipment and a storage medium. The interaction method provided by the invention comprises the following steps: detecting a plurality of reference key points associated with a to-be-mounted virtual object from a target image; determining the mounting position of the virtual object based on the position information of the plurality of reference key points in the target image; and mounting the virtual object to the mounting position in the target image. Based on the above mode, the embodiment of the invention can realize more accurate mounting of the virtual object, and improve the interaction experience of the user.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for interaction and special effect creation. Background Art

[0002] With the development of computer technology, the Internet has become an important platform for creating and sharing media content. In the process of creating media content, video effects and image effects are often used by users. Such effects can enrich the media content created by users and increase the interest of the content. Summary of the invention

[0003] In a first aspect of the present disclosure, an interaction method is provided. The method comprises: detecting a plurality of reference key points associated with a virtual object to be mounted from a target image; determining a mounting position of the virtual object based on position information of the plurality of reference key points in the target image; and mounting the virtual object at the mounting position in the target image.

[0004] In a second aspect of the present disclosure, a method for producing special effects is provided. The method includes: presenting an editing interface for producing a mounting special effect; determining an expected mounting position of a virtual object to be mounted based on input information received in the editing interface; determining a plurality of reference key points associated with the expected mounting position, the plurality of reference key points being used to determine a target mounting position of the virtual object during execution of the mounting special effect; and generating a special effect file corresponding to the mounting special effect based on the plurality of reference key points.

[0005] In a third aspect of the present disclosure, an interactive device is provided. The device includes: a key point determination module configured to detect multiple reference key points associated with a virtual object to be mounted from a target image; a position determination module configured to determine a mounting position of the virtual object based on position information of the multiple reference key points in the target image; and a mounting module configured to mount the virtual object at the mounting position in the target image.

[0006] In a fourth aspect of the present disclosure, a device for making special effects is provided. The device includes: an interface presentation module, configured to present an editing interface for making a mounting special effect; a position marking module, configured to determine an expected mounting position of a virtual object to be mounted based on input information received in the editing interface; a key point marking module, configured to determine a plurality of reference key points associated with the expected mounting position, the plurality of reference key points being used to determine a target mounting position of the virtual object during the execution of the mounting special effect; and a file generation module, configured to generate a special effect file corresponding to the mounting special effect based on the plurality of reference key points.

[0007] In a fifth aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect and / or the second aspect.

[0008] In a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect and / or the second aspect.

[0009] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] Figure 1A and Figure 1B A schematic diagram showing a conventional mounting virtual object is shown;

[0012] Figure 2 A schematic diagram showing an example environment in which embodiments according to the present disclosure may be implemented;

[0013] FIG. 3A to FIG. 3D A schematic diagram of mounting a virtual object according to some embodiments of the present disclosure is shown;

[0014] Figure 4 A schematic diagram of mounting a virtual object according to some other embodiments of the present disclosure is shown;

[0015] FIG. 5A to FIG. 5C A schematic diagram of producing mounting special effects according to some embodiments of the present disclosure is shown;

[0016] Figure 6 A flowchart showing an example interaction process according to some embodiments of the present disclosure is shown;

[0017] Figure 7 A flowchart showing an example process for producing special effects according to some embodiments of the present disclosure is shown;

[0018] Figure 8 A schematic structural block diagram of an interaction device according to some embodiments of the present disclosure is shown;

[0019] Fig. 9 A schematic structural block diagram of an apparatus for producing special effects according to some embodiments of the present disclosure is shown; and

[0020] Fig.10 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0023] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0024] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects are subject to the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user knows and confirms. Accordingly, when implementing each embodiment of the present disclosure, the type, scope of use, usage scenario, etc. of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method can vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.

[0025] In this specification and the embodiments, if personal information processing is involved, it will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or it is necessary to perform a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0026] Some special effects can support mounting some virtual objects at predetermined locations in the image. Figure 1A and Figure 1B As shown, such an example special effect may, for example, support mounting the virtual object 110 at a predetermined position of the image 105 .

[0027] However, traditional mount special effects rely on specific key points (also called feature points) identified in the image. Figure 1A The human body object shown is used as an example. The key point positions that can be identified in the human body object may be limited. For example, the chest position of the human body may not have a matching key point. This may result in that people may not be able to effectively mount the virtual object to the chest position of the human body.

[0028] In addition, in the mounting scheme based on specific key points, the orientation information of the virtual object usually only depends on the orientation of the key points. This results in the mounted virtual object having an inappropriate orientation, for example, not matching the posture of the human body.

[0029] Therefore, such mounting special effects are difficult to meet people's interactive needs.

[0030] On the one hand, an embodiment of the present disclosure proposes an interactive solution. According to the solution, multiple reference key points associated with a virtual object to be mounted can be detected from a target image. Further, the mounting position of the virtual object can be determined based on the position information of the multiple reference key points in the target image. Accordingly, the virtual object can be mounted at the mounting position in the target image.

[0031] Based on the above method, the embodiments of the present disclosure can achieve more accurate mounting of virtual objects and improve the user's interactive experience. In addition, the embodiments of the present disclosure can also enrich the support for mounting virtual objects at positions without key points (for example, parts such as the chest), thereby improving the interactive experience.

[0032] On the other hand, an embodiment of the present disclosure proposes a special effect production scheme. According to the scheme, an editing interface for producing a mounting special effect can be presented. In addition, the expected mounting position of the virtual object to be mounted can be determined based on the input information received in the editing interface. Further, multiple reference key points associated with the expected mounting position can be determined, and the multiple reference key points are used to determine the target mounting position of the virtual object during the execution of the mounting special effect. Accordingly, a special effect file corresponding to the mounting special effect can be generated based on the multiple reference key points.

[0033] Based on the above approach, the embodiments of the present disclosure can support users in creating mounting special effects mounted at non-key point locations, thereby improving the flexibility of users in creating mounting special effects.

[0034] Various example implementations of the solution are described in detail below in conjunction with the accompanying drawings.

[0035] Example Environment

[0036] Figure 2 Schematic diagram of an example environment 200 in which embodiments of the present disclosure can be implemented. Figure 2 As shown, the example environment 200 may include an interaction device 210. As will be described in detail below, the interaction device 210 may apply a mounting special effect to a target image 220 to provide media content 230 with the mounting special effect.

[0037] In some embodiments, such a target image 220 may include, for example, a static image (e.g., a photo) or a dynamic image (e.g., a video) acquired by an image acquisition device. For example, a user may use the image acquisition device carried by the interactive device 210 to take a picture or a video, and may apply the mount special effect to such a picture or video.

[0038] In some other embodiments, such a target image 220 may also include a generated image file (e.g., a picture file or a video file). The user may, for example, select an existing image file locally in the interactive device 210 to apply the mount special effect. Alternatively, the user may, for example, upload an image file to be processed to the interactive device 210 to apply the mount special effect.

[0039] In other embodiments, such a target image 220 may also be, for example, a static image or a dynamic image with a target style generated based on a generated image file. For example, a user may upload a local image or video to the interactive device 210, and the interactive device 210 and / or other appropriate electronic devices may use a style conversion tool to generate another image or video corresponding to the generated image or video. For example, a user may upload a realistic photo, and the generated image may be, for example, a cartoon-style picture. This disclosure is not intended to limit the image conversion process.

[0040] In some embodiments, the interactive device 210 can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable game terminals, VR / AR devices, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDA), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, game devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the user (such as "wearable" circuits, etc.).

[0041] In some other examples, the interactive device 210 may be, for example, an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, as well as big data and artificial intelligence platforms.

[0042] In some embodiments, the interactive device 210 may also apply the mounting special effect to the target image 220 based on the published special effect file 250. Figure 2 As shown, as will be described in detail below, such a special effect file can be generated by the generating device 240, for example.

[0043] It should be understood that although the generating device 240 and the interactive device 210 are Figure 2 Although shown as separate boxes in FIG. 2 , the generating device 240 and the interacting device 210 may be the same or different electronic devices.

[0044] It should be understood that the structure and function of each element in the environment 200 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0045] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0046] Example Mounting Effects

[0047] The following will refer to FIG. 3A to FIG. 3D To describe the special effect interaction process according to an embodiment of the present disclosure.

[0048] Figure 3A Schematic diagram of the interaction process according to some embodiments of the present disclosure is shown. Figure 3A As shown, the interactive device 210 can obtain the target image 220 and detect multiple key points from the target image 220, for example, key points 305-1 to 305-10 (individually or collectively referred to as key points 305).

[0049] It should be understood that the key points 305 may also be of different types depending on the objects included in the target image 220. Taking the target image 220 including a human object as an example, the key points 305 may include detected human key points (eg, bone key points).

[0050] It should be understood that any appropriate key point detection technology may be used to identify the key points 305 in the target image 220 , examples of which may include but are not limited to: Skeleton2d, avatar3d, and the like.

[0051] Furthermore, the interaction device 210 may determine a plurality of reference key points associated with the virtual object to be mounted from the plurality of key points 305 .

[0052] In some embodiments, such multiple reference key points may be determined based on the expected mounting position of the virtual object. Figure 3A As shown, the virtual object corresponds to an expected mounting position 310. Such an expected mounting position 310 does not overlap with the detectable plurality of key points 305.

[0053] In some embodiments, the expected mounting position 310 may include, for example, a specific point, or may also include a specific area. For example, the expected mounting position 310 may indicate the left chest area of ​​the human subject.

[0054] Furthermore, the interaction device 210 may determine, from the plurality of key points 305, a plurality of key points whose distances to the expected mounting position 310 are less than a threshold value, as reference key points. Figure 3AAs an example, the determined reference key points may include, for example, a reference key point 305 - 4 (shoulder point), a reference key point 305 - 2 (neck point), a reference key point 305 - 5 (upper arm point), and a reference key point 305 - 6 (spine point).

[0055] Alternatively or additionally, the multiple reference key points may also include those specified in the process of making the special effect file. As described in detail below with reference to the special effect making process, for example, multiple reference key points to be detected may also be specified in the process of generating the special effect file.

[0056] As another embodiment, the interaction device 210 may also perform detection of multiple desired reference key points, without performing detection of all key points.

[0057] Furthermore, the interaction device 210 may determine the mounting position of the virtual object based on a plurality of reference key points.

[0058] In some embodiments, the interactive device 210 may determine the mounting position based on two reference key points among the multiple reference key points. Figure 3B As shown, the interaction device 210 may determine the mounting position based on the reference key point 305 - 6 and the reference key point 305 - 5 , for example.

[0059] As an example, the interaction device 210 may construct a first reference line 315 - 1 based on the reference key point 305 - 6 and the reference key point 305 - 5 , and determine the midpoint of the first reference line 315 - 1 as the mounting position.

[0060] As another example, the interactive device 210 may also determine the target position in the reference line as the mounting position based on the position description information. Such position description information may indicate, for example, the relationship between the first distance from the target position to be determined to the reference key point 305-6 and the second distance from the target position to the reference key point 305-5.

[0061] For example, the special effect file 250 may maintain position description information, which may indicate that the distance from the target position to be determined to the reference key point 305-6 is equal to the distance from the target position to the reference key point 305-5. In this case, the determined target position may be the midpoint of the reference line.

[0062] As another example, the position description information may indicate a ratio (e.g., 2:1) between the distance from the target position to be determined to the reference key point 305-6 and the distance from the target position to the reference key point 305-5. Thus, the interaction device 210 may determine the target position on the first reference line 315-1 as the mounting position according to the ratio.

[0063] In some embodiments, as will be described in detail below, such position description information may be determined based on interactions during the special effects production process, for example.

[0064] In some embodiments, the interactive device 210 may also determine the mounting position based on a greater number of reference key points. Accordingly, the position description information may also indicate, for example, the influence weight of each reference key point on the mounting position. Further, the interactive device 210 may determine the target position to be mounted based on a weighted interpolation operation of multiple reference key points.

[0065] In some embodiments, the interaction device 210 may also determine the target position to be mounted based on a third reference key point (eg, reference key point 305 - 4 ) among the plurality of reference key points.

[0066] For example, the interaction device 210 may construct the second reference line 315 - 2 based on the reference key point 305 - 4 and the target direction information. In some embodiments, the reference key point 305 - 4 may be the reference key point closest to the expected mounting position 310 .

[0067] In some embodiments, the target direction information used to construct the second reference key point 315 - 2 may include preset first direction information. For example, considering that the upper body of the human body is mostly in a vertical state in the target image 220 , the first direction information may include the vertical direction in the target image 220 .

[0068] As another embodiment, the target direction information used to construct the second reference key point 315-2 may be based on the second direction information determined by detecting the posture of the target object (e.g., human body) in the target image 220. For example, the interaction device 210 may analyze the orientation of the human spine in the target image 220 and determine the orientation as the second direction information.

[0069] Further, the interaction device 210 may determine the mounting position 320 of the virtual object based on the intersection of the first reference line 315 - 1 and the second reference line 315 - 2 . Accordingly, the interaction device 210 may mount the virtual object at the mounting position 320 of the target image 220 .

[0070] Based on the approach discussed above, the embodiments of the present disclosure can determine the mounting position of a virtual object through multiple reference key points, thereby supporting mounting the virtual object to an area that cannot be covered by key point detection, thereby improving the flexibility of special effect interaction.

[0071] In some embodiments, the interaction device 210 may further determine the target orientation of the virtual object based on the position information of the multiple reference key points, and may complete the mounting of the virtual object according to the determined target orientation.

[0072] Specifically, the interaction device 210 may determine at least one reference plane based on multiple reference key points, wherein each reference plane is determined based on at least three reference key points among the multiple reference key points. Figure 3C In the example, the interactive device 210 can, for example, determine the first plane 325-1 where the triangle formed by the reference key point 305-2, the reference key point 305-6 and the reference key point 305-4 is located; and can determine the second plane 325-2 where the triangle formed by the reference key point 305-5, the reference key point 305-6 and the reference key point 305-4 is located.

[0073] Accordingly, the interaction device 210 can determine the orientation information (also referred to as first orientation information) of the first plane 325-1 and the second plane 325-2, and determine the target orientation of the virtual object based on the orientation information. Exemplarily, the interaction device 210 can determine the target orientation of the virtual object based on the mean of the normal directions of the first plane 325-1 and the second plane 325-2.

[0074] In some embodiments, when determining the target orientation of the virtual object, the interaction device 210 may also consider orientation information of a target reference key point among multiple reference key points. Such a target reference key point may be, for example, a reference key point whose distance to the expected mounting position of the virtual object is less than a preset distance, for example, the closest reference key point 305-4.

[0075] Further, the interactive device 210 may obtain the orientation information (also referred to as the second orientation information) of the reference key point 305-4. It should be understood that such orientation information may be determined based on the key point recognition process, for example. Further, the interactive device 210 may use the first orientation information and the second orientation information to determine the target orientation of the virtual object.

[0076] Exemplarily, the interaction device 210 may convert the rotation quaternion of the reference key point 305 - 4 into a direction vector, and normalize the direction vector with the average of the normal vector of the first plane 325 - 1 and the normal vector of the second plane, and finally determine the target orientation of the virtual object.

[0077] Based on this approach, the embodiments of the present disclosure can enable the mounted virtual object to better match the posture of the target object (eg, human body) in the target image, thereby improving the interactive experience of the special effects.

[0078] For example, Figure 3D As shown, the interactive device 210 can mount the virtual object 330 on the target image 220 based on the determined mounting position 320 and the target orientation, thereby completing the interaction of the mounting special effect.

[0079] Can see, with Figure 1B Compared with the mounting special effects shown in the figure, the embodiment of the present disclosure can support mounting in areas without key point coverage on the one hand, and can also improve the mounting authenticity of virtual objects on the other hand. Thus, the interactive experience of the mounting special effects can be improved.

[0080] In some other embodiments, taking the chest as an example of the expected mounting position, the target image acquired by the interactive device 210 may not be able to detect all the reference key points, thus causing mounting failure.

[0081] Therefore, before using multiple reference key points to determine the mounting position, the interactive device 210 can, for example, detect a facial area from the target image, that is, an area corresponding to the face of a person in the target image. Further, the interactive device 210 can determine the proportion of the facial area in the target image. Figure 4 As an example, the interaction device 210 may detect the facial region 410 from the target image 405 , and determine the proportion of the facial region 410 in the target image 405 .

[0082] Further, if the proportion is less than or equal to the threshold, the interaction device 210 may determine that all reference key points can be detected, and may use the process discussed above to complete the mounting of the virtual object.

[0083] On the contrary, if the proportion is greater than the threshold, the interaction device 210 may determine that it is difficult to complete the mounting of the virtual object based on multiple reference key points. In this case, the interaction device 210 may, for example, directly determine the position of the predetermined reference key point among the key points identified from the target image 405 as the mounting position of the virtual object.

[0084] For example, the interactive device 210 may Figure 3A The reference key point 305 - 4 shown is determined as the mounting position. Such a reference key point can be, for example, a key point closest to the expected mounting position of the virtual object.

[0085] Alternatively, such a reference key point may also be a specific key point specified in the generation process of the special effect file.

[0086] In some embodiments, the interactive device 210 may also directly use the orientation information of the reference key point 305 - 5 as the target orientation of the virtual object 415 .

[0087] As another example, in order to ensure the presentation effect of the mounting special effects, when the proportion of the facial area is greater than the threshold, the interactive device 210 can also adjust the size of the virtual object 415 so that it can be fully presented in the target image 405.

[0088] In this way, the embodiments of the present disclosure can further ensure successful mounting in some special scenarios and improve the stability and flexibility of the mounting special effect interaction.

[0089] Sample special effects production

[0090] The following will further describe the process for producing the mount special effects file discussed above.

[0091] Figure 5A The editing interface 500A for making mounting special effects is shown. Such an editing interface 500A can be used, for example, as follows Figure 2 The generating device 240 is shown provided.

[0092] like Figure 5A As shown, the editing interface 500A can be used to make a reference model 505 of the mounting effect. It should be understood that such a reference model 505 can be used to characterize the basic style of the target object to be mounted, and the present disclosure is not intended to limit the specific form of the reference model 505.

[0093] Furthermore, the editing device 500A can also present multiple key points 510 that can be used to produce mounting special effects in association with the reference model 505. Taking the human body as an example of the target object, such multiple key points 510 can include multiple preset human key points.

[0094] Further, if Figure 5B As shown, the generating device 240 may obtain input information received in the editing interface and determine an expected mounting position 515 of the virtual object to be mounted.

[0095] For example, the user may set the expected mounting position 515 by clicking, or may adjust the expected mounting position by moving the indicator of the expected mounting position 515. The present disclosure is not intended to limit the specific interaction form of setting the expected mounting position 515.

[0096] Further, the generating device 240 may determine a plurality of reference key points based on the expected mounting position 515. Such a plurality of reference key points may include, for example, Figure 5B Shown are reference key point 520 - 1 , reference key point 520 - 2 , reference key point 520 - 3 , and reference key point 520 - 4 .

[0097] In some embodiments, the generating device 240 may determine as reference key points a plurality of key points whose distances to the expected mounting position 515 are less than a threshold distance. Additionally, such a threshold distance may be an initially set distance, or may also be a distance determined based on user input.

[0098] For example, the generating device 240 may indicate the range 525 for determining the reference key points through a graphic style during the process of determining the reference key points. The user may also adjust the range 525 through a drag operation, for example, to add or remove one or more reference key points.

[0099] Additionally, the generating device 240 may also highlight a plurality of reference key points (eg, reference key points 520 - 1 to 520 - 4 ) determined in real time, for example, to distinguish them from other key points that are not determined as reference key points.

[0100] Alternatively, in addition to adjusting the range 525, the user may also add or remove one or more reference key points by, for example, selecting the key points.

[0101] After determining such a plurality of reference key points, the generating device 240 may generate a plurality of reference key points based on the reference key points as described above. FIG. 3A to FIG. 3D The discussed mounting logic based on multiple reference key points to determine the mounting position and / or target orientation of the virtual object will not be described in detail in this disclosure.

[0102] In some embodiments, the generating device 240 may also present the real-time mounting effect corresponding to the determined multiple reference key points in the editing interface. Figure 5C As shown, the generating device 240 may, for example, mount a predetermined virtual object 535 onto the sample image 530 according to a plurality of reference key points determined in the editing interface.

[0103] It should be understood that such virtual objects 535 and / or sample images 530 may be added or uploaded by users during the special effects production process.

[0104] Further, after receiving confirmation about the multiple reference key points, the generating device 240 may generate a special effect file corresponding to the mounted special effect based on the multiple reference key points.

[0105] Exemplarily, such a special effect file may maintain identifiers of multiple reference key points for determining the mounting position and / or target orientation of the virtual object, so as to detect multiple reference key points during the special effect execution process to complete the mounting special effect.

[0106] In some embodiments, as discussed above, the generation device 240 may also determine multiple distances from the expected mounting position 515 to multiple reference key points (e.g., reference key point 520-1 to reference key point 520-4), and may generate distance description information based on the multiple distances. The distance description information may be used to indicate the relationship between the multiple distances, and the distance description information is used to determine the target mounting position of the virtual object during the execution of the mounting special effect.

[0107] Such distance description information may, for example, indicate the influence weights of the multiple reference key points 520 - 1 to 520 - 4 on the mounting position to be determined, thereby supporting the determination of the mounting position of the virtual object by weight-based interpolation.

[0108] By maintaining distance description information in a special effect file, embodiments of the present disclosure can support determining the mounting position according to the positions of multiple reference key points during the special effect application process discussed above.

[0109] Example Process

[0110] Figure 6 FIG. 6 is a flow chart showing an example interaction process 600 according to some embodiments of the present disclosure. The process 600 may be implemented at the interaction device 210. Figure 2 600 is described below.

[0111] like Figure 6 As shown, in box 610, the interaction device 210 detects a plurality of reference key points associated with the virtual object to be mounted from the target image.

[0112] In block 620 , the interaction device 210 determines a mounting position of the virtual object based on position information of the plurality of reference key points in the target image.

[0113] In block 630 , the interaction device 210 mounts the virtual object at the mount location in the target image.

[0114] In some embodiments, the mounting position is a target mounting position, and detecting multiple reference key points associated with the virtual object to be mounted from the target image includes: based on the expected mounting position of the virtual object, determining multiple reference key points from the target image, and the distance between the multiple reference key points and the expected mounting position is less than a first threshold.

[0115] In some embodiments, the multiple reference key points include at least a first reference key point and a second reference key point, and determining the mounting position of the virtual object based on the position information of the multiple reference key points in the target image includes: determining a first reference line based on the first reference key point and the second reference key point; and determining the mounting position of the virtual object based on the first reference line.

[0116] In some embodiments, determining the mounting position of the virtual object based on the first reference line includes: determining the target position on the first reference line as the mounting position according to preset distance description information, wherein the distance description information indicates the relationship between a first distance from the target position to a first reference key point and a second distance from the target position to a second reference key point.

[0117] In some embodiments, the multiple reference key points also include a third reference key point, and determining the mounting position of the virtual object based on the first reference line includes: determining the second reference line based on the third reference key point and target direction information; and determining the mounting position based on the intersection of the first reference line and the second reference line.

[0118] In some embodiments, the third reference key point is a reference key point that is closest to the expected mounting position of the virtual object among the multiple reference key points.

[0119] In some embodiments, the target direction information includes: preset first direction information; or second direction information determined based on the posture of the target object in the target image.

[0120] In some embodiments, mounting the virtual object at the mounting position in the target image includes: determining the target orientation of the virtual object based on position information of multiple reference key points; and mounting the virtual object at the mounting position in the target image according to the target orientation.

[0121] In some embodiments, determining the target orientation of a virtual object based on position information of multiple reference key points includes: determining at least one reference plane based on multiple reference key points, wherein each reference plane is determined based on at least three reference key points among the multiple reference key points; and determining the target orientation of the virtual object based on first orientation information of at least one reference plane.

[0122] In some embodiments, determining the target orientation of a virtual object based on first orientation information of at least one reference plane includes: determining second orientation information of a target reference key point among multiple reference key points, wherein the distance from the target reference key point to the expected mounting position of the virtual object is less than a preset distance; and determining the target orientation of the virtual object based on the first orientation information and the second orientation information.

[0123] In some embodiments, determining the mounting position of the virtual object based on the position information of multiple reference key points in the target image includes: detecting a facial area from the target image; and in response to a proportion of the facial area in the target image being less than or equal to a threshold, determining the mounting position of the virtual object based on the position information of multiple reference key points in the target image.

[0124] In some embodiments, process 600 further includes: in response to a proportion of the facial area in the target image being greater than a threshold, determining a position of a predetermined reference key point among the plurality of reference key points as a mounting position of the virtual object.

[0125] In some embodiments, process 600 further includes: determining an orientation of a predetermined reference key point among the plurality of reference key points as a target orientation of the virtual object.

[0126] In some embodiments, the predetermined reference key point is a reference key point that is closest to the expected mounting position of the virtual object among the multiple reference key points.

[0127] In some embodiments, the target image includes: a static image or a dynamic image captured by an image capture device; an uploaded image file; or an image corresponding to the target style generated based on the uploaded image file.

[0128] Figure 7 FIG. 7 is a flowchart showing an example process 700 for producing special effects according to some embodiments of the present disclosure. The process 700 may be implemented at the generating device 240. Figure 2 700 is described below.

[0129] like Figure 7 As shown, in block 710 , the generating device 240 presents an editing interface for producing mounting special effects.

[0130] In block 720 , the generating device 240 determines an expected mounting position of the virtual object to be mounted based on the input information received in the editing interface.

[0131] In block 730 , the generation device 240 determines a plurality of reference key points associated with the expected mounting position, and the plurality of reference key points are used to determine a target mounting position of the virtual object during the execution of the mounting special effect.

[0132] In block 740 , the generating device 240 generates a special effect file corresponding to the mounted special effect based on the plurality of reference key points.

[0133] In some embodiments, the editing interface presents a reference model and multiple key points corresponding to the reference model, and determining multiple reference key points associated with the expected mounting position includes: determining multiple reference key points from the multiple key points whose distance to the expected mounting position is less than a threshold distance.

[0134] In some embodiments, the threshold distance is determined based on user input.

[0135] In some embodiments, process 700 further includes: highlighting multiple reference key points among the multiple key points in the editing interface.

[0136] In some embodiments, process 700 also includes: determining multiple distances from the expected mounting position to multiple reference key points; and generating distance description information based on the multiple distances, the distance description information is used to indicate the relationship between the multiple distances, and the distance description information is used to determine the target mounting position of the virtual object during the execution of the mounting special effect.

[0137] Example devices and equipment

[0138] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 8 Schematic block diagram of an example interactive apparatus 800 according to some embodiments of the present disclosure is shown. The apparatus 800 may be implemented as or included in the interactive device 210. Each module / component in the apparatus 800 may be implemented by hardware, software, firmware or any combination thereof.

[0139] like Figure 8 As shown, the device 800 includes a key point determination module 810, which is configured to detect multiple reference key points associated with a virtual object to be mounted from a target image; a position determination module 820, which is configured to determine the mounting position of the virtual object based on the position information of the multiple reference key points in the target image; and a mounting module 830, which is configured to mount the virtual object to the mounting position in the target image.

[0140] In some embodiments, the mounting position is a target mounting position, and the key point determination module 810 is further configured to: determine multiple reference key points from the target image based on the expected mounting position of the virtual object, and the distances from the multiple reference key points to the expected mounting position are less than a first threshold.

[0141] In some embodiments, the multiple reference key points include at least a first reference key point and a second reference key point, and the position determination module 820 is further configured to: determine a first reference line based on the first reference key point and the second reference key point; and determine the mounting position of the virtual object based on the first reference line.

[0142] In some embodiments, the position determination module 820 is further configured to: determine the target position on the first reference line as the mounting position according to preset distance description information, wherein the distance description information indicates the relationship between a first distance from the target position to the first reference key point and a second distance from the target position to the second reference key point.

[0143] In some embodiments, the multiple reference key points also include a third reference key point, and the position determination module 820 is further configured to: determine the second reference line based on the third reference key point and the target direction information; and determine the mounting position based on the intersection of the first reference line and the second reference line.

[0144] In some embodiments, the third reference key point is a reference key point that is closest to the expected mounting position of the virtual object among the multiple reference key points.

[0145] In some embodiments, the target direction information includes: preset first direction information; or second direction information determined based on the posture of the target object in the target image.

[0146] In some embodiments, the mounting module 830 is further configured to: determine a target orientation of the virtual object based on position information of a plurality of reference key points; and mount the virtual object to a mounting position in the target image according to the target orientation.

[0147] In some embodiments, the mounting module 830 is further configured to: determine at least one reference plane based on multiple reference key points, wherein each reference plane is determined based on at least three reference key points among the multiple reference key points; and determine the target orientation of the virtual object based on the first orientation information of the at least one reference plane.

[0148] In some embodiments, the mounting module 830 is further configured to: determine second orientation information of a target reference key point among multiple reference key points, wherein the distance from the target reference key point to the expected mounting position of the virtual object is less than a preset distance; and determine the target orientation of the virtual object based on the first orientation information and the second orientation information.

[0149] In some embodiments, the position determination module 820 is further configured to: detect a facial area from the target image; and in response to the proportion of the facial area in the target image being less than or equal to a threshold, determine the mounting position of the virtual object based on the position information of multiple reference key points in the target image.

[0150] In some embodiments, the position determination module 820 is further configured to: in response to a proportion of the facial area in the target image being greater than a threshold, determine the position of a predetermined reference key point among the multiple reference key points as the mounting position of the virtual object.

[0151] In some embodiments, the mounting module 830 is further configured to: determine the orientation of a predetermined reference key point among the multiple reference key points as the target orientation of the virtual object.

[0152] In some embodiments, the predetermined reference key point is a reference key point that is closest to the expected mounting position of the virtual object among the multiple reference key points.

[0153] In some embodiments, the target image includes: a static image or a dynamic image captured by an image capture device; an uploaded image file; or an image corresponding to the target style generated based on the uploaded image file.

[0154] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Fig. 9 A schematic structural block diagram of an example apparatus 900 for producing special effects according to some embodiments of the present disclosure is shown. The apparatus 900 may be implemented as or included in the generating device 240. Each module / component in the apparatus 900 may be implemented by hardware, software, firmware, or any combination thereof.

[0155] like Fig. 9 As shown, the apparatus 900 includes an interface presentation module 910, configured to present an editing interface for producing a mounting special effect; a position marking module 920, configured to generate a device 240 to determine an expected mounting position of a virtual object to be mounted based on input information received in the editing interface; a key point marking module 930, configured to determine a plurality of reference key points associated with the expected mounting position, the plurality of reference key points being used to determine a target mounting position of the virtual object during execution of the mounting special effect; and a file generation module 940, configured to generate a special effect file corresponding to the mounting special effect based on a plurality of reference key points.

[0156] In some embodiments, the editing interface presents a reference model and a plurality of key points corresponding to the reference model, and the key point marking module 930 is further configured to: determine a plurality of reference key points whose distance to the expected mounting position is less than a threshold distance from the plurality of key points.

[0157] In some embodiments, the threshold distance is determined based on user input.

[0158] In some embodiments, the interface presenting module 910 is further configured to: highlight and present multiple reference key points among the multiple key points in the editing interface.

[0159] In some embodiments, the file generation module 940 is further configured to: determine multiple distances from the expected mounting position to multiple reference key points; and generate distance description information based on the multiple distances, wherein the distance description information is used to indicate the relationship between the multiple distances, and the distance description information is used to determine the target mounting position of the virtual object during the execution of the mounting special effect.

[0160] Fig.10 1 shows a block diagram of an electronic device 1000 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Fig.10 The electronic device 1000 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Fig.10 The electronic device 1000 shown may be used to implement the electronic device 110 of FIG. 1 .

[0161] like Fig.10As shown, the electronic device 1000 is in the form of a general electronic device. The components of the electronic device 1000 may include, but are not limited to, one or more processors or processing units 1010, a memory 1020, a storage device 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060. The processing unit 1010 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 1020. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 1000.

[0162] The electronic device 1000 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 1000, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1020 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1030 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the electronic device 1000.

[0163] The electronic device 1000 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig.10 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. Memory 1020 may include a computer program product 1025 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0164] The communication unit 1040 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 1000 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0165] The input device 1050 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 1060 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 1000 may also communicate with one or more external devices (not shown) through the communication unit 1040 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 1000, or communicate with any device that allows the electronic device 1000 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0166] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0167] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0168] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0169] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0170] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0171] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. An interactive method, comprising: Detecting a plurality of reference key points associated with a virtual object to be mounted from the target image; Determining a mounting position of the virtual object based on position information of the multiple reference key points in the target image; as well as The virtual object is mounted at the mounting position in the target image.

2. The method according to claim 1, wherein the mounting position is a target mounting position, and detecting a plurality of reference key points associated with the virtual object to be mounted from the target image comprises: Based on the expected mounting position of the virtual object, the multiple reference key points are determined from the target image, and the distances from the multiple reference key points to the expected mounting position are less than a first threshold.

3. The method according to claim 1, wherein the plurality of reference key points include at least a first reference key point and a second reference key point, and determining the mounting position of the virtual object based on position information of the plurality of reference key points in the target image comprises: Determine a first reference line based on the first reference key point and the second reference key point; as well as Based on the first reference line, the mounting position of the virtual object is determined.

4. The method according to claim 3, wherein determining the mounting position of the virtual object based on the first reference line comprises: According to preset distance description information, a target position on the first reference line is determined as the mounting position, wherein the distance description information indicates a relationship between a first distance from the target position to the first reference key point and a second distance from the target position to the second reference key point.

5. The method according to claim 3, wherein the plurality of reference key points further include a third reference key point, and determining the mounting position of the virtual object based on the first reference line comprises: Determining a second reference line based on the third reference key point and the target direction information; as well as The mounting position is determined based on an intersection of the first reference line and the second reference line.

6. The method according to claim 5, wherein the third reference key point is a reference key point among the multiple reference key points that is closest to the expected mounting position of the virtual object.

7. The method according to claim 5, wherein the target direction information comprises: Preset first direction information; or The second direction information is determined based on the posture of the target object in the target image.

8. The method according to claim 1, wherein mounting the virtual object at the mounting position in the target image comprises: Determining a target orientation of the virtual object based on the position information of the plurality of reference key points; as well as The virtual object is mounted at the mounting position in the target image according to the target orientation.

9. The method according to claim 8, wherein determining the target orientation of the virtual object based on the position information of the plurality of reference key points comprises: Based on the plurality of reference key points, determining at least one reference plane, wherein each reference plane is determined based on at least three reference key points among the plurality of reference key points; as well as The target orientation of the virtual object is determined based on the first orientation information of the at least one reference plane.

10. The method according to claim 9, wherein determining the target orientation of the virtual object based on the first orientation information of the at least one reference plane comprises: Determining second orientation information of a target reference key point among the multiple reference key points, where a distance from the target reference key point to an expected mounting position of the virtual object is less than a preset distance; as well as The target orientation of the virtual object is determined based on the first orientation information and the second orientation information.

11. The method according to claim 1, wherein determining the mounting position of the virtual object based on the position information of the multiple reference key points in the target image comprises: Detecting a facial region from the target image; as well as In response to the proportion of the facial area in the target image being less than or equal to a threshold, a mounting position of the virtual object is determined based on position information of the multiple reference key points in the target image.

12. The method according to claim 11, further comprising: In response to the proportion of the facial area in the target image being greater than the threshold, determining the position of a predetermined reference key point among the plurality of reference key points as the mounting position of the virtual object; as well as Determining the orientation of the predetermined reference key point among the plurality of reference key points as the target orientation of the virtual object; The predetermined reference key point is a reference key point among the multiple reference key points that is closest to the expected mounting position of the virtual object.

13. A method for generating special effects, comprising: Presents the editing interface for generating mount effects; Determining an expected mounting position of the virtual object to be mounted based on the input information received in the editing interface; Determining a set of reference key points associated with the expected mounting position, wherein the set of reference key points is used to determine a target mounting position of the virtual object during the execution of the mounting special effect; as well as A special effects file corresponding to the mounting special effects is generated based on the set of reference key points.

14. The method according to claim 13, wherein the editing interface presents a reference model and a plurality of key points corresponding to the reference model, and determining a set of reference key points associated with the expected mounting position comprises: The set of reference key points whose distances to the expected mounting position are less than a threshold distance is determined from the plurality of key points, wherein the threshold distance is determined based on user input.

15. The method according to claim 14, further comprising: In the editing interface, the group of reference key points among the plurality of key points is highlighted.

16. The method according to claim 13, further comprising: Determining a plurality of distances from the expected mounting position to the plurality of reference key points; as well as Based on the distances, distance description information is generated to indicate the relationship between the multiple distances, and the distance description information is used to determine the target mounting position of the virtual object during the execution of the mounting special effect.

17. An interactive device, comprising: A key point determination module is configured to detect a plurality of reference key points associated with a virtual object to be mounted from a target image; a position determination module, configured to determine a mounting position of the virtual object based on position information of the multiple reference key points in the target image; as well as The mounting module is configured to mount the virtual object to the mounting position in the target image.

18. A device for generating special effects, comprising: An interface presentation module, configured to present an editing interface for generating mounting special effects; a position marking module, configured to determine an expected mounting position of the virtual object to be mounted based on the input information received in the editing interface; A key point marking module is configured to determine a set of reference key points associated with the expected mounting position, wherein the set of reference key points is used to determine a target mounting position of the virtual object during the execution of the mounting special effect; as well as The special effect generation module is configured to generate a special effect file corresponding to the mounted special effect based on the set of reference key points.

19. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform a method according to any one of claims 1 to 12 or 13 to 16.

20. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 12 or 13 to 16.

Images