Method and apparatus for interaction and producing special effect, and device and storage medium

Abstract

The embodiments of the present disclosure relate to a method and apparatus for interaction and producing a special effect, and a device and a storage medium. The method for interaction provided herein comprises: detecting, from among a target image, a plurality of reference key points associated with a virtual object to be mounted; determining a mounting location for the virtual object on the basis of location information of the plurality of reference key points in the target image; and mounting the virtual object at the mounting location in the target image. On the basis of the above method, the embodiments of the present disclosure can realize more accurate mounting of a virtual object, thus improving the interaction experience of a user.

Description

Method, device, equipment and storage medium for interaction and special effect production

[0001] This application claims priority to the Chinese invention patent application with application number 202310807910.X filed on July 3, 2023 and titled “Methods, devices, equipment and storage media for interaction and production of special effects”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for interaction and producing special effects. Background Art

[0003] With the development of computer technology, the internet has become a vital platform for creating and sharing media content. During the creation of media content, video and image effects are frequently used. These effects can enrich user-generated content and enhance its appeal.

[0004] Summary of the Invention

[0005] In a first aspect of the present disclosure, an interaction method is provided. The method includes: detecting multiple reference key points associated with a virtual object to be mounted from a target image; determining a mounting position of the virtual object based on positional information of the multiple reference key points in the target image; and mounting the virtual object at the mounting position in the target image.

[0006] In a second aspect of the present disclosure, a method for creating a special effect is provided. The method includes: presenting an editing interface for creating a mount special effect; determining an expected mount position of a virtual object to be mounted based on input information received in the editing interface; determining multiple reference key points associated with the expected mount position, the multiple reference key points being used to determine a target mount position of the virtual object during execution of the mount special effect; and generating a special effect file corresponding to the mount special effect based on the multiple reference key points.

[0007] In a third aspect of the present disclosure, an interactive device is provided. The device includes: a key point determination module configured to detect multiple reference key points associated with a virtual object to be mounted from a target image; a position determination module configured to determine a mounting position of the virtual object based on positional information of the multiple reference key points in the target image; and a mounting module configured to mount the virtual object at the mounting position in the target image.

[0008] In a fourth aspect of the present disclosure, a device for producing special effects is provided. The device includes: an interface presentation module configured to present an editing interface for producing a mounting special effect; a position marking module configured to determine an expected mounting position of a virtual object to be mounted based on input information received in the editing interface; a key point marking module configured to determine a plurality of reference key points associated with the expected mounting position, the plurality of reference key points being used to determine a target mounting position of the virtual object during execution of the mounting special effect; and a file generation module configured to generate a special effect file corresponding to the mounting special effect based on the plurality of reference key points.

[0009] In a fifth aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first and / or second aspects.

[0010] In a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect and / or the second aspect.

[0011] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0013] 1A and 1B are schematic diagrams showing conventional mounting of virtual objects;

[0014] FIG2 shows a schematic diagram of an example environment in which embodiments according to the present disclosure may be implemented;

[0015] 3A to 3D illustrate schematic diagrams of mounting a virtual object according to some embodiments of the present disclosure;

[0016] FIG4 shows a schematic diagram of mounting a virtual object according to some other embodiments of the present disclosure;

[0017] 5A to 5C are schematic diagrams showing how to create a mount special effect according to some embodiments of the present disclosure;

[0018] FIG6 shows a flowchart of an example interaction process according to some embodiments of the present disclosure;

[0019] FIG7 shows a flowchart of an example process for producing special effects according to some embodiments of the present disclosure;

[0020] FIG8 shows a schematic structural block diagram of an interaction device according to some embodiments of the present disclosure;

[0021] FIG9 shows a schematic structural block diagram of an apparatus for producing special effects according to some embodiments of the present disclosure; and

[0022] FIG10 illustrates a block diagram of an electronic device capable of implementing various embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0024] It should be noted that the titles of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and any type of embodiment may be included under any section / subsection. Furthermore, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or in different sections / subsections.

[0025] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0026] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects shall comply with the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user is aware of and confirms them. Accordingly, when implementing the various embodiments of the present disclosure, the types, scope of use, and usage scenarios of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method may vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.

[0027] If this specification and the solutions in the examples involve the processing of personal information, such processing will be done only with a legitimate basis (such as with the consent of the subject of personal information or as necessary for the performance of a contract) and only within the prescribed or agreed scope. A user's refusal to process personal information other than that required for basic functions will not affect the user's use of basic functions.

[0028] Some special effects can support mounting some virtual objects at predetermined locations in an image. As shown in FIG1A and FIG1B , such an example special effect can support mounting a virtual object 110 at a predetermined location in an image 105 .

[0029] However, traditional mounting effects rely on specific key points (also known as feature points) identified in the image. However, taking the human object shown in Figure 1A as an example, the key point locations that can be identified in the human object may be limited. For example, there may be no matching key points at the chest of the human body. This may result in people being unable to effectively mount virtual objects to the chest of the human body.

[0030] Furthermore, in mounting solutions based on specific keypoints, the orientation of the virtual object typically depends solely on the orientation of the keypoints. This can result in the mounted virtual object having an inappropriate orientation, for example, not matching the human body's posture.

[0031] Therefore, such mounting special effects are difficult to meet people's interactive needs.

[0032] In one aspect, embodiments of the present disclosure provide an interactive solution. According to this solution, multiple reference key points associated with a virtual object to be mounted can be detected from a target image. Furthermore, the mounting position of the virtual object can be determined based on the positional information of the multiple reference key points in the target image. Accordingly, the virtual object can be mounted at the mounting position in the target image.

[0033] Based on the above approach, the embodiments of the present disclosure can achieve more accurate mounting of virtual objects, improving the user's interactive experience. In addition, the embodiments of the present disclosure can also enrich the support for mounting virtual objects in locations without key points (for example, parts such as the chest), thereby improving the interactive experience.

[0034] On the other hand, embodiments of the present disclosure propose a special effects production solution. According to this solution, an editing interface for producing a mounting special effect can be presented. Furthermore, based on input information received in the editing interface, an expected mounting position of a virtual object to be mounted can be determined. Furthermore, multiple reference key points associated with the expected mounting position can be determined, and these multiple reference key points are used to determine the target mounting position of the virtual object during the execution of the mounting special effect. Accordingly, a special effects file corresponding to the mounting special effect can be generated based on the multiple reference key points.

[0035] Based on the above approach, the embodiments of the present disclosure can support users in creating mounting effects at non-key point locations, thereby improving the flexibility of users in creating mounting effects.

[0036] Various example implementations of this solution are described in detail below in conjunction with the accompanying drawings.

[0037] Sample Environment

[0038] FIG2 illustrates a schematic diagram of an example environment 200 in which embodiments of the present disclosure can be implemented. As shown in FIG2 , example environment 200 may include an interactive device 210. As will be described in detail below, interactive device 210 may apply a mount effect to a target image 220 to provide media content 230 with the mount effect.

[0039] In some embodiments, such target image 220 may include, for example, a static image (e.g., a photo) or a dynamic image (e.g., a video) acquired using an image acquisition device. For example, a user may use the image acquisition device carried by the interactive device 210 to take a picture or video, and may apply the mount special effect to such a picture or video.

[0040] In some other embodiments, the target image 220 may also include a generated image file (e.g., a picture file or a video file). The user may select an image file already stored locally on the interactive device 210 to apply the mount effect. Alternatively, the user may upload an image file to be processed to the interactive device 210 to apply the mount effect.

[0041] In other embodiments, such a target image 220 may also be a static image or dynamic image generated based on a generated image file and having a target style. For example, a user may upload a local image or video to the interactive device 210, and the interactive device 210 and / or other appropriate electronic devices may utilize style conversion tools to generate another image or video corresponding to the generated image or video. For example, the user may upload a realistic photo, and the generated image may be, for example, a cartoon-style picture. This disclosure is not intended to limit the image conversion process.

[0042] In some embodiments, the interactive device 210 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a handheld computer, a portable game terminal, a VR / AR device, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the user (such as a "wearable" circuit, etc.).

[0043] In some other examples, the interactive device 210 can also be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms.

[0044] In some embodiments, the interactive device 210 may also apply the mount special effect to the target image 220 based on the published special effect file 250. As shown in FIG2 , such a special effect file may be generated by a generating device 240, for example, as will be described in detail below.

[0045] It should be understood that although the generating device 240 and the interacting device 210 are shown as separate blocks in FIG. 2 , the generating device 240 and the interacting device 210 may be the same or different electronic devices.

[0046] It should be understood that the structure and function of each element in the environment 200 are described for illustrative purposes only and do not imply any limitation on the scope of the present disclosure.

[0047] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0048] Example mount effects

[0049] The special effect interaction process according to an embodiment of the present disclosure will be described below with reference to FIG. 3A to FIG. 3D .

[0050] Figure 3A shows a schematic diagram of an interaction process according to some embodiments of the present disclosure. As shown in Figure 3A, the interaction device 210 can obtain a target image 220 and detect multiple key points from the target image 220, such as key points 305-1 to key points 305-10 (individually or collectively referred to as key points 305).

[0051] It should be understood that the key points 305 may be of different types depending on the objects included in the target image 220. Taking the target image 220 as an example, the key points 305 may include detected human key points (eg, skeletal key points).

[0052] It should be understood that any appropriate keypoint detection technology may be used to identify the keypoints 305 in the target image 220 , examples of which may include but are not limited to Skeleton2d, Avatar3d, and the like.

[0053] Furthermore, the interaction device 210 may determine a plurality of reference key points associated with the virtual object to be mounted from the plurality of key points 305 .

[0054] In some embodiments, such multiple reference key points can be determined based on the expected mounting position of the virtual object. As shown in Figure 3A, the virtual object corresponds to the expected mounting position 310. Such expected mounting position 310 does not overlap with the detectable multiple key points 305.

[0055] In some embodiments, the expected mounting position 310 may include, for example, a specific point, or may also include a specific area. For example, the expected mounting position 310 may indicate the left chest area of ​​the human subject.

[0056] Furthermore, the interaction device 210 may determine, from the plurality of key points 305, a plurality of key points whose distances to the expected mounting position 310 are less than a threshold value, as reference key points. Taking FIG. 3A as an example, the determined reference key points may include, for example, reference key point 305-4 (shoulder point), reference key point 305-2 (neck point), reference key point 305-5 (upper arm point), and reference key point 305-6 (spine point).

[0057] Alternatively or additionally, the multiple reference key points may also include those specified in the process of making the special effect file. As will be described in detail below with reference to the special effect making process, for example, multiple reference key points to be detected may also be specified in the process of generating the special effect file.

[0058] As another embodiment, the interactive device 210 may also perform detection on multiple desired reference key points, without performing detection on all key points.

[0059] Furthermore, the interaction device 210 may determine the mounting position of the virtual object based on a plurality of reference key points.

[0060] In some embodiments, the interactive device 210 may determine the mounting position based on two reference key points among the multiple reference key points. As shown in FIG3B , the interactive device 210 may determine the mounting position based on reference key point 305 - 6 and reference key point 305 - 5 .

[0061] As an example, the interaction device 210 may construct a first reference line 315 - 1 based on the reference key point 305 - 6 and the reference key point 305 - 5 , and determine the midpoint of the first reference line 315 - 1 as the mounting position.

[0062] As another example, the interaction device 210 may also determine the target position in the reference line as the mounting position based on the position description information. Such position description information may, for example, indicate the relationship between a first distance from the target position to be determined to the reference key point 305-6 and a second distance from the target position to the reference key point 305-5.

[0063] For example, the special effect file 250 may include position description information, which may indicate that the distance between the target position to be determined and the reference key point 305-6 is equal to the distance between the target position and the reference key point 305-5. In this case, the determined target position may be the midpoint of the reference line.

[0064] As another example, the position description information may indicate a ratio (e.g., 2:1) between the distance from the target position to be determined to the reference key point 305-6 and the distance from the target position to the reference key point 305-5. Thus, the interaction device 210 may determine the target position on the first reference line 315-1 as the mounting position based on the ratio.

[0065] In some embodiments, as will be described in detail below, such position description information may be determined based on interactions during the special effects production process, for example.

[0066] In some embodiments, the interactive device 210 may determine the mounting location based on a greater number of reference key points. Accordingly, the location description information may further indicate the weight of each reference key point's influence on the mounting location. Furthermore, the interactive device 210 may determine the target location to be mounted based on a weighted interpolation operation of multiple reference key points.

[0067] In some embodiments, the interaction device 210 may also determine the target location to be mounted based on a third reference key point (eg, reference key point 305 - 4 ) among the multiple reference key points.

[0068] For example, the interaction device 210 may construct the second reference line 315 - 2 based on the reference key point 305 - 4 and the target direction information. In some embodiments, the reference key point 305 - 4 may be the reference key point closest to the expected mounting position 310 .

[0069] In some embodiments, the target direction information used to construct the second reference key point 315-2 may include, for example, preset first direction information. For example, considering that the upper body of a person is mostly in a vertical position in the target image 220, the first direction information may include the vertical direction in the target image 220.

[0070] As another embodiment, the target direction information used to construct the second reference key point 315-2 may be based on second direction information determined by detecting the posture of a target object (e.g., a human body) in the target image 220. For example, the interaction device 210 may analyze the orientation of the human spine in the target image 220 and determine the orientation as the second direction information.

[0071] Furthermore, the interaction device 210 may determine the mounting position 320 of the virtual object based on the intersection of the first reference line 315 - 1 and the second reference line 315 - 2 . Accordingly, the interaction device 210 may mount the virtual object at the mounting position 320 of the target image 220 .

[0072] Based on the method discussed above, the embodiments of the present disclosure can determine the mounting position of the virtual object through multiple reference key points, thereby supporting the mounting of the virtual object to an area that cannot be covered by key point detection, thereby improving the flexibility of special effect interaction.

[0073] In some embodiments, the interaction device 210 may further determine the target orientation of the virtual object based on the position information of multiple reference key points, and may complete the mounting of the virtual object according to the determined target orientation.

[0074] Specifically, the interaction device 210 may determine at least one reference plane based on multiple reference key points, wherein each reference plane is determined based on at least three of the multiple reference key points. For example, in the example of FIG3C , the interaction device 210 may determine a first plane 325-1 based on a triangle formed by reference key points 305-2, 305-6, and 305-4; and may determine a second plane 325-2 based on a triangle formed by reference key points 305-5, 305-6, and 305-4.

[0075] Accordingly, the interaction device 210 may determine the orientation information (also referred to as first orientation information) of the first plane 325-1 and the second plane 325-2, and determine the target orientation of the virtual object based on the orientation information. For example, the interaction device 210 may determine the target orientation of the virtual object based on the average of the normal directions of the first plane 325-1 and the second plane 325-2.

[0076] In some embodiments, when determining the target orientation of the virtual object, the interaction device 210 may also consider orientation information of a target reference key point among the multiple reference key points. Such a target reference key point may, for example, be a reference key point whose distance to the expected mounting position of the virtual object is less than a preset distance, such as the closest reference key point 305-4.

[0077] Furthermore, the interaction device 210 may obtain orientation information (also referred to as second orientation information) of the reference key point 305-4. It should be understood that such orientation information may be determined, for example, based on a key point recognition process. Furthermore, the interaction device 210 may utilize the first orientation information and the second orientation information to determine the target orientation of the virtual object.

[0078] Exemplarily, the interactive device 210 may convert the rotation quaternion of the reference key point 305 - 4 into a direction vector, and normalize the direction vector with the mean of the normal vector of the first plane 325 - 1 and the normal vector of the second plane to ultimately determine the target orientation of the virtual object.

[0079] Based on this approach, the embodiments of the present disclosure can enable the mounted virtual object to better match the posture of the target object (e.g., human body) in the target image, thereby improving the interactive experience of special effects.

[0080] For example, as shown in FIG3D , the interactive device 210 may mount the virtual object 330 onto the target image 220 based on the determined mounting position 320 and target orientation, thereby completing the interaction of the mounting special effect.

[0081] It can be seen that compared with the mounting effect shown in FIG1B , the embodiment of the present disclosure can support mounting in areas without key point coverage, and can also improve the mounting authenticity of virtual objects, thereby improving the interactive experience of the mounting effect.

[0082] In some other embodiments, taking the chest as an example, the target image acquired by the interactive device 210 may not be able to detect all the reference key points, thus causing the mounting to fail.

[0083] Therefore, before determining the mounting position using multiple reference key points, the interaction device 210 can, for example, detect a facial region from the target image, i.e., the region corresponding to the face of a person in the target image. Furthermore, the interaction device 210 can determine the proportion of the facial region within the target image. Using Figure 4 as an example, the interaction device 210 can detect a facial region 410 from the target image 405 and determine the proportion of the facial region 410 within the target image 405.

[0084] Furthermore, if the proportion is less than or equal to the threshold, the interaction device 210 may determine that all reference key points can be detected, and may use the process discussed above to complete the mounting of the virtual object.

[0085] On the other hand, if the percentage is greater than a threshold, the interaction device 210 may determine that it is difficult to complete the mounting of the virtual object based on the multiple reference key points. In this case, the interaction device 210 may, for example, directly determine the position of a predetermined reference key point among the key points identified in the target image 405 as the mounting position of the virtual object.

[0086] For example, the interaction device 210 may determine the reference key point 305 - 4 shown in FIG3A as the mounting position. Such a reference key point may be, for example, the key point closest to the expected mounting position of the virtual object.

[0087] Alternatively, such a reference key point may also be a specific key point specified during the generation process of the special effect file.

[0088] In some embodiments, the interactive device 210 may also directly use the orientation information of the reference key point 305 - 5 as the target orientation of the virtual object 415 .

[0089] As another example, in order to ensure the presentation effect of the mounting special effect, when the proportion of the facial area is greater than the threshold, the interactive device 210 can also adjust the size of the virtual object 415 so that it can be fully presented in the target image 405.

[0090] In this way, the embodiments of the present disclosure can further ensure successful mounting in some special scenarios and improve the stability and flexibility of the mounting special effect interaction.

[0091] Example special effects production

[0092] The following further describes the process for producing the mount effects file discussed above.

[0093] FIG5A shows an editing interface 500A for producing a mounting special effect. Such an editing interface 500A may be provided by the generating device 240 shown in FIG2 , for example.

[0094] 5A , the editing interface 500A can be used to create a reference model 505 for mounting special effects. It should be understood that such a reference model 505 can be used to represent the basic style of the target object to be mounted, and the present disclosure is not intended to limit the specific form of the reference model 505.

[0095] Furthermore, the editing device 500A can also present multiple key points 510 that can be used to create a mounting special effect in association with the reference model 505. Taking the human body as an example of the target object, such multiple key points 510 can include multiple preset human body key points.

[0096] Furthermore, as shown in FIG5B , the generating device 240 may obtain input information received in the editing interface and determine an expected mounting position 515 of the virtual object to be mounted.

[0097] For example, the user may set the expected mounting position 515 by clicking, or may adjust the expected mounting position by moving the indicator of the expected mounting position 515. The present disclosure is not intended to limit the specific interaction form of setting the expected mounting position 515.

[0098] Furthermore, the generating device 240 may determine a plurality of reference key points based on the expected mounting position 515. Such a plurality of reference key points may include, for example, a reference key point 520-1, a reference key point 520-2, a reference key point 520-3, and a reference key point 520-4 as shown in FIG5B.

[0099] In some embodiments, the generation device 240 may determine as reference key points a plurality of key points whose distances to the expected mounting position 515 are less than a threshold distance. Additionally, such a threshold distance may be an initially set distance, or may be a distance determined based on user input.

[0100] For example, the generating device 240 may indicate the range 525 used to determine the reference key points through a graphic style during the process of determining the reference key points. The user may also adjust the range 525 by, for example, dragging, to add or remove one or more reference key points.

[0101] Additionally, the generating device 240 may also highlight a plurality of reference key points (eg, reference key points 520 - 1 to 520 - 4 ) determined in real time, for example, to distinguish them from other key points that are not determined as reference key points.

[0102] Alternatively, in addition to adjusting the range 525 , the user may also add or remove one or more reference key points by, for example, selecting the key points.

[0103] After determining such multiple reference key points, the generation device 240 can determine the mounting position and / or target orientation of the virtual object based on the mounting logic based on multiple reference key points as discussed above with reference to Figures 3A to 3D. This disclosure will not be repeated here.

[0104] In some embodiments, the generation device 240 may also present a real-time mounting effect corresponding to the determined multiple reference key points in the editing interface. As shown in FIG5C , the generation device 240 may, for example, mount a predetermined virtual object 535 onto the sample image 530 based on the multiple reference key points determined in the editing interface.

[0105] It should be understood that such virtual objects 535 and / or sample images 530 may be added or uploaded by the user during the special effects production process.

[0106] Furthermore, after receiving confirmation about the multiple reference key points, the generating device 240 may generate a special effect file corresponding to the mounted special effect based on the multiple reference key points.

[0107] Exemplarily, such a special effect file may maintain identifiers of multiple reference key points for determining the mounting position and / or target orientation of the virtual object, so as to detect the multiple reference key points during the special effect execution process to complete the mounting special effect.

[0108] In some embodiments, as discussed above, the generation device 240 may also determine multiple distances from the expected mount position 515 to multiple reference key points (e.g., reference key point 520-1 to reference key point 520-4), and may generate distance description information based on the multiple distances. This distance description information may be used to indicate the relationship between the multiple distances, and the distance description information is used to determine the target mount position of the virtual object during the execution of the mount special effect.

[0109] Such distance description information may, for example, indicate the influence weights of the multiple reference key points 520 - 1 to 520 - 4 on the mounting position to be determined, thereby supporting determination of the mounting position of the virtual object by weight-based interpolation.

[0110] By maintaining distance description information in the special effects file, embodiments of the present disclosure can support determining the mounting position according to the positions of multiple reference key points during the special effects application process discussed above.

[0111] Example Process

[0112] FIG6 shows a flow chart of an example interaction process 600 according to some embodiments of the present disclosure. The process 600 may be implemented at the interaction device 210. The process 600 is described below with reference to FIG2.

[0113] As shown in FIG. 6 , in block 610 , the interaction device 210 detects a plurality of reference key points associated with a virtual object to be mounted from a target image.

[0114] In block 620 , the interaction device 210 determines a mounting position of the virtual object based on position information of the plurality of reference key points in the target image.

[0115] In block 630 , the interaction device 210 mounts the virtual object at the mount location in the target image.

[0116] In some embodiments, the mounting position is a target mounting position, and detecting multiple reference key points associated with the virtual object to be mounted from the target image includes: based on the expected mounting position of the virtual object, determining multiple reference key points from the target image, and the distance between the multiple reference key points and the expected mounting position is less than a first threshold.

[0117] In some embodiments, the multiple reference key points include at least a first reference key point and a second reference key point, and determining the mounting position of the virtual object based on the position information of the multiple reference key points in the target image includes: determining a first reference line based on the first reference key point and the second reference key point; and determining the mounting position of the virtual object based on the first reference line.

[0118] In some embodiments, determining the mounting position of the virtual object based on the first reference line includes: determining the target position on the first reference line as the mounting position according to preset distance description information, wherein the distance description information indicates the relationship between the first distance from the target position to the first reference key point and the second distance from the target position to the second reference key point.

[0119] In some embodiments, the multiple reference key points also include a third reference key point, and determining the mounting position of the virtual object based on the first reference line includes: determining the second reference line based on the third reference key point and target direction information; and determining the mounting position based on the intersection of the first reference line and the second reference line.

[0120] In some embodiments, the third reference key point is a reference key point that is closest to the expected mounting position of the virtual object among the multiple reference key points.

[0121] In some embodiments, the target direction information includes: preset first direction information; or second direction information determined based on the posture of the target object in the target image.

[0122] In some embodiments, mounting the virtual object at a mounting position in the target image includes: determining a target orientation of the virtual object based on position information of multiple reference key points; and mounting the virtual object at the mounting position in the target image according to the target orientation.

[0123] In some embodiments, determining the target orientation of a virtual object based on position information of multiple reference key points includes: determining at least one reference plane based on multiple reference key points, wherein each reference plane is determined based on at least three reference key points among the multiple reference key points; and determining the target orientation of the virtual object based on first orientation information of at least one reference plane.

[0124] In some embodiments, determining the target orientation of a virtual object based on first orientation information of at least one reference plane includes: determining second orientation information of a target reference key point among multiple reference key points, where the distance from the target reference key point to the expected mounting position of the virtual object is less than a preset distance; and determining the target orientation of the virtual object based on the first orientation information and the second orientation information.

[0125] In some embodiments, determining the mounting position of the virtual object based on the position information of multiple reference key points in the target image includes: detecting a facial area from the target image; and in response to the proportion of the facial area in the target image being less than or equal to a threshold, determining the mounting position of the virtual object based on the position information of multiple reference key points in the target image.

[0126] In some embodiments, process 600 further includes: in response to a proportion of the facial region in the target image being greater than a threshold, determining a position of a predetermined reference key point among the plurality of reference key points as a mounting position of the virtual object.

[0127] In some embodiments, process 600 further includes determining an orientation of a predetermined reference key point among the plurality of reference key points as a target orientation of the virtual object.

[0128] In some embodiments, the predetermined reference key point is a reference key point that is closest to the expected mounting position of the virtual object among the multiple reference key points.

[0129] In some embodiments, the target image includes: a static image or a dynamic image captured by an image capture device; an uploaded image file; or an image corresponding to the target style generated based on the uploaded image file.

[0130] 7 shows a flow chart of an example process 700 for producing special effects according to some embodiments of the present disclosure. Process 700 may be implemented at a generation device 240. Process 700 is described below with reference to FIG.

[0131] As shown in FIG. 7 , in block 710 , the generating device 240 presents an editing interface for producing mounting special effects.

[0132] In block 720 , the generating device 240 determines an expected mounting position of the virtual object to be mounted based on the input information received in the editing interface.

[0133] In block 730 , the generation device 240 determines a plurality of reference key points associated with the expected attachment position, where the plurality of reference key points are used to determine a target attachment position of the virtual object during execution of the attachment special effect.

[0134] In block 740 , the generating device 240 generates a special effect file corresponding to the mounted special effect based on the plurality of reference key points.

[0135] In some embodiments, the editing interface presents a reference model and multiple key points corresponding to the reference model, and determining the multiple reference key points associated with the expected mounting position includes: determining, from the multiple key points, multiple reference key points whose distance to the expected mounting position is less than a threshold distance.

[0136] In some embodiments, the threshold distance is determined based on user input.

[0137] In some embodiments, process 700 further includes: highlighting a plurality of reference key points among the plurality of key points in the editing interface.

[0138] In some embodiments, process 700 also includes: determining multiple distances from the expected mounting position to multiple reference key points; and generating distance description information based on the multiple distances, the distance description information is used to indicate the relationship between the multiple distances, and the distance description information is used to determine the target mounting position of the virtual object during the execution of the mounting special effect.

[0139] Example devices and equipment

[0140] Embodiments of the present disclosure also provide corresponding apparatuses for implementing the above-described methods or processes. FIG8 shows a schematic block diagram of an example interactive apparatus 800 according to certain embodiments of the present disclosure. Apparatus 800 may be implemented as or included in interactive device 210. Each module / component in apparatus 800 may be implemented by hardware, software, firmware, or any combination thereof.

[0141] As shown in Figure 8, the device 800 includes a key point determination module 810, which is configured to detect multiple reference key points associated with the virtual object to be mounted from the target image; a position determination module 820, which is configured to determine the mounting position of the virtual object based on the position information of the multiple reference key points in the target image; and a mounting module 830, which is configured to mount the virtual object to the mounting position in the target image.

[0142] In some embodiments, the mounting position is a target mounting position, and the key point determination module 810 is further configured to: determine multiple reference key points from the target image based on the expected mounting position of the virtual object, and the distance between the multiple reference key points and the expected mounting position is less than a first threshold.

[0143] In some embodiments, the multiple reference key points include at least a first reference key point and a second reference key point, and the position determination module 820 is further configured to: determine a first reference line based on the first reference key point and the second reference key point; and determine the mounting position of the virtual object based on the first reference line.

[0144] In some embodiments, the position determination module 820 is further configured to: determine the target position on the first reference line as the mounting position based on preset distance description information, wherein the distance description information indicates the relationship between the first distance from the target position to the first reference key point and the second distance from the target position to the second reference key point.

[0145] In some embodiments, the multiple reference key points also include a third reference key point, and the position determination module 820 is further configured to: determine the second reference line based on the third reference key point and the target direction information; and determine the mounting position based on the intersection of the first reference line and the second reference line.

[0146] In some embodiments, the third reference key point is a reference key point that is closest to the expected mounting position of the virtual object among the multiple reference key points.

[0147] In some embodiments, the target direction information includes: preset first direction information; or second direction information determined based on the posture of the target object in the target image.

[0148] In some embodiments, the mounting module 830 is further configured to: determine a target orientation of the virtual object based on position information of a plurality of reference key points; and mount the virtual object to a mounting position in the target image according to the target orientation.

[0149] In some embodiments, the mounting module 830 is further configured to: determine at least one reference plane based on multiple reference key points, wherein each reference plane is determined based on at least three reference key points among the multiple reference key points; and determine the target orientation of the virtual object based on the first orientation information of the at least one reference plane.

[0150] In some embodiments, the mounting module 830 is further configured to: determine the second orientation information of a target reference key point among multiple reference key points, where the distance from the target reference key point to the expected mounting position of the virtual object is less than a preset distance; and determine the target orientation of the virtual object based on the first orientation information and the second orientation information.

[0151] In some embodiments, the position determination module 820 is further configured to: detect a facial area from the target image; and in response to the proportion of the facial area in the target image being less than or equal to a threshold, determine the mounting position of the virtual object based on the position information of multiple reference key points in the target image.

[0152] In some embodiments, the position determination module 820 is further configured to: in response to a proportion of the facial area in the target image being greater than a threshold, determine the position of a predetermined reference key point among the multiple reference key points as the mounting position of the virtual object.

[0153] In some embodiments, the mounting module 830 is further configured to: determine the orientation of a predetermined reference key point among the multiple reference key points as the target orientation of the virtual object.

[0154] In some embodiments, the predetermined reference key point is a reference key point that is closest to the expected mounting position of the virtual object among the multiple reference key points.

[0155] In some embodiments, the target image includes: a static image or a dynamic image captured by an image capture device; an uploaded image file; or an image corresponding to the target style generated based on the uploaded image file.

[0156] Embodiments of the present disclosure also provide corresponding apparatuses for implementing the above-described methods or processes. FIG9 shows a schematic block diagram of an example apparatus 900 for producing special effects according to certain embodiments of the present disclosure. Apparatus 900 may be implemented as or included in a generation device 240. Each module / component in apparatus 900 may be implemented by hardware, software, firmware, or any combination thereof.

[0157] As shown in Figure 9, the device 900 includes an interface presentation module 910, which is configured to present an editing interface for producing a mounting special effect; a position marking module 920, which is configured to generate a device 240 to determine the expected mounting position of the virtual object to be mounted based on the input information received in the editing interface; a key point marking module 930, which is configured to determine a plurality of reference key points associated with the expected mounting position, and the plurality of reference key points are used to determine the target mounting position of the virtual object during the execution of the mounting special effect; and a file generation module 940, which is configured to generate a special effect file corresponding to the mounting special effect based on the plurality of reference key points.

[0158] In some embodiments, the editing interface presents a reference model and multiple key points corresponding to the reference model, and the key point marking module 930 is further configured to: determine multiple reference key points whose distance to the expected mounting position is less than a threshold distance from the multiple key points.

[0159] In some embodiments, the threshold distance is determined based on user input.

[0160] In some embodiments, the interface presentation module 910 is further configured to: highlight and present multiple reference key points among the multiple key points in the editing interface.

[0161] In some embodiments, the file generation module 940 is further configured to: determine multiple distances from the expected mounting position to multiple reference key points; and generate distance description information based on the multiple distances, the distance description information being used to indicate the relationship between the multiple distances, and the distance description information being used to determine the target mounting position of the virtual object during the execution of the mounting special effect.

[0162] FIG10 shows a block diagram of an electronic device 1000 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 1000 shown in FIG10 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 1000 shown in FIG10 may be used to implement the electronic device 110 of FIG1 .

[0163] As shown in FIG10 , electronic device 1000 is in the form of a general electronic device. Components of electronic device 1000 may include, but are not limited to, one or more processors or processing units 1010, memory 1020, storage device 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060. Processing unit 1010 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 1020. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to enhance the parallel processing capabilities of electronic device 1000.

[0164] The electronic device 1000 typically includes a plurality of computer storage media. Such media can be any accessible media that is accessible to the electronic device 1000, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1020 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1030 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the electronic device 1000.

[0165] The electronic device 1000 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 10 , a disk drive for reading from or writing to a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 1020 may include a computer program product 1025 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0166] The communication unit 1040 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 1000 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 500 can operate in a networked environment using a logical connection to one or more other servers, a network personal computer (PC), or another network node.

[0167] Input device 1050 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 1060 may be one or more output devices, such as a display, a speaker, or a printer. Electronic device 1000 may also communicate with one or more external devices (not shown) via communication unit 1040 as needed, such as storage devices, display devices, or the like, with one or more devices that allow a user to interact with electronic device 1000, or with any device that allows electronic device 1000 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0168] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0169] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0170] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0171] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0172] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0173] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. An interactive method, comprising: Detecting a plurality of reference key points associated with a virtual object to be mounted from the target image; Determining a mounting position of the virtual object based on position information of the multiple reference key points in the target image; as well as The virtual object is mounted at the mounting position in the target image.

2. The method according to claim 1, wherein the mounting position is a target mounting position, and detecting a plurality of reference key points associated with the virtual object to be mounted from the target image comprises: Based on the expected mounting position of the virtual object, the multiple reference key points are determined from the target image, and the distances from the multiple reference key points to the expected mounting position are less than a first threshold.

3. The method according to claim 1, wherein the plurality of reference key points include at least a first reference key point and a second reference key point, and determining the mounting position of the virtual object based on position information of the plurality of reference key points in the target image comprises: Determine a first reference line based on the first reference key point and the second reference key point; as well as Based on the first reference line, the mounting position of the virtual object is determined.

4. The method according to claim 3, wherein determining the mounting position of the virtual object based on the first reference line comprises: According to preset distance description information, a target position on the first reference line is determined as the mounting position, wherein the distance description information indicates a relationship between a first distance from the target position to the first reference key point and a second distance from the target position to the second reference key point.

5. The method according to claim 3, wherein the plurality of reference key points further include a third reference key point, and determining the mounting position of the virtual object based on the first reference line comprises: Determining a second reference line based on the third reference key point and the target direction information; as well as The mounting position is determined based on an intersection of the first reference line and the second reference line.

6. The method according to claim 5, wherein the third reference key point is a reference key point among the multiple reference key points that is closest to the expected mounting position of the virtual object.

7. The method according to claim 5, wherein the target direction information comprises: Preset first direction information; or The second direction information is determined based on the posture of the target object in the target image.

8. The method according to claim 1, wherein mounting the virtual object at the mounting position in the target image comprises: Determining a target orientation of the virtual object based on the position information of the plurality of reference key points; as well as The virtual object is mounted at the mounting position in the target image according to the target orientation.

9. The method according to claim 8, wherein determining the target orientation of the virtual object based on the position information of the plurality of reference key points comprises: Based on the plurality of reference key points, determining at least one reference plane, wherein each reference plane is determined based on at least three reference key points among the plurality of reference key points; as well as The target orientation of the virtual object is determined based on the first orientation information of the at least one reference plane.

10. The method according to claim 9, wherein determining the target orientation of the virtual object based on the first orientation information of the at least one reference plane comprises: Determining second orientation information of a target reference key point among the multiple reference key points, where a distance from the target reference key point to an expected mounting position of the virtual object is less than a preset distance; as well as Determine the virtual object based on the first orientation information and the second orientation information The target direction.

11. The method according to claim 1, wherein determining the mounting position of the virtual object based on the position information of the multiple reference key points in the target image comprises: Detecting a facial region from the target image; as well as In response to the proportion of the facial area in the target image being less than or equal to a threshold, a mounting position of the virtual object is determined based on position information of the multiple reference key points in the target image.

12. The method according to claim 11, further comprising: In response to the proportion of the facial area in the target image being greater than the threshold, determining the position of a predetermined reference key point among the plurality of reference key points as the mounting position of the virtual object; as well as Determining the orientation of the predetermined reference key point among the plurality of reference key points as the target orientation of the virtual object; The predetermined reference key point is a reference key point among the multiple reference key points that is closest to the expected mounting position of the virtual object.

13. A method for generating special effects, comprising: Presents the editing interface for generating mount effects; Determining an expected mounting position of the virtual object to be mounted based on the input information received in the editing interface; Determining a set of reference key points associated with the expected mounting position, wherein the set of reference key points is used to determine a target mounting position of the virtual object during the execution of the mounting special effect; as well as A special effects file corresponding to the mounting special effects is generated based on the set of reference key points.

14. The method according to claim 13, wherein the editing interface presents a reference model and a plurality of key points corresponding to the reference model, and determining a set of reference key points associated with the expected mounting position comprises: The set of reference key points whose distances to the expected mounting position are less than a threshold distance is determined from the plurality of key points, wherein the threshold distance is determined based on user input.

15. The method according to claim 14, further comprising: In the editing interface, the group of reference key points among the plurality of key points is highlighted.

16. The method according to claim 13, further comprising: Determining a set of distances from the expected mounting position to the set of reference key points; as well as Based on the set of distances, distance description information is generated to indicate a relationship between the set of distances, and the distance description information is used to determine the target mounting position of the virtual object during execution of the mounting special effect.

17. An interactive device, comprising: A key point determination module is configured to detect a plurality of reference key points associated with a virtual object to be mounted from a target image; a position determination module, configured to determine a mounting position of the virtual object based on position information of the multiple reference key points in the target image; as well as The mounting module is configured to mount the virtual object to the mounting position in the target image.

18. A device for generating special effects, comprising: An interface presentation module, configured to present an editing interface for generating mounting special effects; a position marking module, configured to determine an expected mounting position of the virtual object to be mounted based on the input information received in the editing interface; A key point marking module is configured to determine a set of reference key points associated with the expected mounting position, wherein the set of reference key points is used to determine a target mounting position of the virtual object during the execution of the mounting special effect; as well as The special effect generation module is configured to generate a special effect file corresponding to the mounted special effect based on the set of reference key points.

19. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform a method according to any one of claims 1 to 12 or 13 to 16.

20. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 12 or 13 to 16.

Images