Method, device, electronic device, and storage medium for determining special effect video

Abstract

An embodiment of the present application discloses a method, apparatus, electronic device and storage medium for determining a special effect animation, the method including: adding a virtual special effect model to a target object in response to a special effect trigger operation; controlling a target virtual part of the virtual special effect model corresponding to the target object to move based on collected audio information; and adjusting a target entity part of the target object to a predetermined state to obtain a target special effect.

Description

[Technical Field]

[0001] This application claims priority from Chinese Patent Application No. 202210362558.9 filed with the China Patent Office on April 7, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the field of computer technology, for example, to a method, device, electronic device, and storage medium for determining special effect video. [Background technology]

[0003] With the development of network technology, more and more applications have entered into users' lives, and the series of software that allows users to shoot short videos is especially popular among users.

[0004] When shooting corresponding videos or images based on software for shooting short videos, some special effects processing is often performed to make the video more interesting, but traditional special effects do not have sufficiently rich content and the display is relatively simple, resulting in an insufficient user viewing experience and usage experience. Summary of the Invention

[0005] The present application provides a method, apparatus, electronic device and storage medium for determining special effect video, thereby making video content richer and more interesting, and further improving user experience.

[0006] An embodiment of the present application provides a method for determining a special effect animation, the method including: adding a virtual special effect model to a target object in response to a special effect trigger operation; controlling a target virtual part of the virtual special effect model corresponding to the target object to move based on collected audio information; and adjusting a target entity part of the target object to a predetermined state to obtain a target special effect.

[0007] An embodiment of the present application provides a special effect animation determination device, which includes: a special effect adding module configured to add a virtual special effect model to a target object in response to a special effect trigger operation; and a target special effect animation determination module configured to control a target virtual part of the virtual special effect model corresponding to the target object to move based on collected audio information, and adjust a target entity part of the target object to a predetermined state to obtain a target special effect.

[0008] An embodiment of the present application further provides an electronic device, which includes at least one processor and a storage device in which at least one program is stored, and when the at least one program is executed by the at least one processor, the at least one processor implements the method for determining a special effect video described in any one of the embodiments of the present application.

[0009] An embodiment of the present application further provides a storage medium including computer-executable instructions, which, when executed by a computer processor, are used to perform the method for determining a special effect video described in any one of the embodiments of the present application. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a flow chart of a method for determining special effect animation provided by an embodiment of the present application; [Figure 2] 1 is a flow chart of a method for determining special effect animation provided by an embodiment of the present application; [Figure 3] 1 is a flow chart of a special effect animation determining device provided by an embodiment of the present application; [Figure 4] 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, the embodiments of the present application will be described with reference to the drawings. It should be understood that the drawings show some embodiments of the present application, but the present application can be realized in many forms and should not be construed as being limited to the embodiments described herein. It should be noted that the drawings and embodiments of the present application are used for illustrative purposes only and do not limit the scope of protection of the present application.

[0012] It should be noted that the steps described in the method embodiments of the present application may be performed in a different order and / or in parallel, and method embodiments may include additional steps and / or omit performing illustrated steps. The scope of the present application is not limited in this respect.

[0013] As used herein, the term "comprises" and variations thereof are open-ended, i.e., "including, but not limited to." The term "based on" means "based at least in part on." The term "in one embodiment" refers to "at least one embodiment," the term "another embodiment" refers to "at least one other embodiment," and the term "some embodiments" refers to "at least some embodiments." Relevant definitions of other terms are provided below.

[0014] It should be noted that the concepts of "first", "second", etc. referred to in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0015] It should be noted that those skilled in the art will understand that the modifications "one" and "multiple" referred to in this application are intended to be exemplary and not limiting, and should be understood as "one or more" unless the context clearly dictates otherwise.

[0016] The names of messages or information exchanged between devices in the examples of this application are for illustrative purposes only and are not used to limit the scope of these messages or information.

[0017] Before describing examples of the present application, an exemplary description of an application scenario can be provided: Particular embodiments of the present application can be applied to any screen that requires special effect display, such as a short video shooting scenario, or any traditional video shooting scenario.

[0018] In the embodiment of the present application, the algorithm implementing this tool can be placed in the terminal device, or integrated into the conventional video shooting scenario to enhance the enjoyment of video shooting. For other embodiments, please refer to the following description.

[0019] 1 is a flow chart of a method for determining a special effect video provided by an embodiment of the present application. This embodiment can be applied to any image display scenario supported by the Internet, such as performing special effect processing on real-time captured video or performing special effect processing on recorded video. The method can be executed by a special effect image processing device, which can be implemented in the form of software and / or hardware, and the hardware can be electronic equipment such as a mobile terminal, a personal computer (PC) end, or a server. Any image display scenario is usually realized by the cooperation of a client and a server, and the method provided by this embodiment can be executed by the server end, the client, or the cooperation of a client and a server end.

[0020] As shown in FIG. 1, the method of this embodiment includes the following steps:

[0021] S110, adding a virtual special effect model to the target object in response to a special effect trigger operation.

[0022] The device for performing the special effect image processing method provided by the embodiments of the present application may be integrated into application software supporting special effect image processing functions, and the software may be installed in an electronic device, which may optionally be a mobile terminal or a PC. The application software is a type of software for image / video processing, and as long as the application software can realize image / video processing, it will not be described one by one in this specification. It may also be an application specially developed to implement software for adding and displaying special effects, or special effects can be added to the corresponding page through an integrated page on the PC.

[0023] It should be noted that the embodiments of the present application can be developed as a special effect tool, and can be integrated into software capable of shooting videos, such as a camera with a terminal device, or into corresponding short video shooting software, so that the special effect tool can be triggered to generate special effect videos when a user shoots a video, broadcasts live, or generates an interesting interactive interface.

[0024] The target object is an object in the frame-in screen, or any object that needs to have a corresponding virtual special effect model added. The virtual special effect model can be a model to be added, which can be any cartoon model or any pet model designed in the development stage, and the model can be displayed in the form of an image, an augmented reality (AR) model, or a three-dimensional model. For example, various AR special effect models of cats are pre-created.

[0025] The user can trigger the corresponding application software to enter the homepage, and then click the shooting control to enter the shooting page. By triggering the special effect tool corresponding to the special effect displayed on the shooting page, if the display page includes a target object, any one virtual special effect model can be added to the target object; if the user selects a control model without triggering a virtual special effect model, a default virtual special effect model can be added to the target object; if the user selects a virtual special effect model, a virtual special effect model selected by the user can be added.

[0026] In this embodiment, responding to the special effect trigger operation includes at least one of detecting that the frame-in screen includes a target object, detecting a control that triggers a target special effect tool, detecting that a body movement of the target object in the frame-in screen matches a body movement of the added special effect, or detecting that audio information triggers a wake word for adding a special effect.

[0027] The control may be a button displayed on the display interface of the application software, and the triggering of the button is characterized as generating a special effect animation. The button may be a physical button or a virtual button. In practical applications, when a user triggers the button, it can be considered that a special effect animation corresponding to the special effect tool is generated. The added special effect may specifically match the special effect triggered by the user. Furthermore, audio information is collected based on a microphone array arranged on the terminal device, and the audio information is analyzed and processed. If the processing result contains the words "add special effect," it indicates that the special effect adding function is triggered. Based on the content of the audio information, it is determined whether to add a special effect to avoid user interaction with the display page, thereby improving the intelligence of the special effect addition. Another implementation method is as follows: According to the field of view of the mobile terminal, it is determined whether the body movement of a target subject within the field of view matches a predetermined body movement, and if so, it indicates that a special effect adding operation is triggered. For example, if the predetermined body movement is a "victory" pose and the target subject's body movement triggers the "victory" pose, it indicates that a special effect trigger operation is triggered. Alternatively, after triggering the special effect tool, as long as it is detected that a facial image is included within the field of view of the photographing device, the special effect trigger operation is triggered, i.e., as long as it is detected that the frame-in screen includes a target object, it can be considered that a corresponding virtual special effect model needs to be added to the target object.

[0028] It should be noted that in actual applications, the number of target objects may be one or more. For example, if the special effect video expected by the user contains only one user, the video may be filmed in an environment with low human traffic density. Naturally, when filming in an environment with high human traffic density, there may be multiple objects in the frame-in screen. If the multiple objects in the frame-in screen are all target objects, virtual special effect models can be added to all target objects. If there are multiple objects in the frame-in screen but only one object is the target object, the target object can be marked in advance. If there are multiple target objects for filming, virtual special effect models can be added only to the marked target object. Alternatively, the marked target object is displayed on the display interface, and other objects are not displayed.

[0029] In this embodiment, before adding a virtual special effect model to the target object, the method further includes determining a virtual special effect model corresponding to the target object from a virtual special effect library, where the virtual special effect library includes at least one virtual special effect model to be selected.

[0030] It should be noted that in the development stage, multiple virtual special effect models can be designed and stored in a virtual special effect library. The multiple virtual special effect models stored in the virtual special effect library can be character models, animal models, cartoon pattern models, or any scenery models, and the virtual special effect models can be used as virtual special effect models to be selected. The virtual special effect model added to the target object can be a model among the virtual special effect models to be selected.

[0031] After responding to the special effect trigger operation, a virtual special effect model can be randomly selected from the virtual special effect library and added to the target object. Furthermore, a default virtual special effect model can be set, and when a special effect tool is detected and the display interface detects that the target object is included, if the user does not select a virtual special effect model at the same time, the default virtual special effect model can be added to the target object. Another situation is as follows: after a special effect tool is detected, a virtual model selection menu can be popped up, and the user can trigger a virtual special effect model displayed in the menu to add the triggered virtual special effect model to the target object.

[0032] In this embodiment, determining a virtual special effect model corresponding to the target object from a virtual special effect library includes determining a virtual special effect model corresponding to the target object based on basic attribute information of the target object.

[0033] In practical applications, in order to make the added virtual special effect model optimal for the target object, a virtual special effect model suitable for the target object can be selected from the virtual special effect library.

[0034] The basic attribute information may be basic information of the user, and optionally, the basic information may correspond to icon information registered by the user. For example, if the icon is a landscape, a landscape virtual special effect model can be selected. If the icon is a cartoon pattern, a virtual special effect model corresponding to the cartoon pattern can be selected from the virtual special effect library.

[0035] It should be noted that the virtual special effect model may be preset in the development stage or generated in real time according to actual needs, and optionally, before adding the virtual special effect model to the target object, the method further includes obtaining an image to be processed that has been uploaded in advance, and determining a virtual special effect model based on at least one display object in the image to be processed.

[0036] The image to be processed is an image previously captured by a user. The image may be a photograph of animals and plants, or a landscape. The animals and plants in the image to be processed are used as display objects. A virtual special effect model can be generated based on the display objects, allowing customization of the virtual special effect model.

[0037] Before adding the virtual special effects model to the target object, an image upload control can be triggered to upload the image to be processed to the software. The software can identify the display object in the image to be processed and reconstruct the virtual special effects model corresponding to the display object. After the reconstruction is complete, the virtual special effects model can be added to the target object.

[0038] In practical application, there may be one or more target objects in the frame-in screen, and if there are multiple target objects, all of the objects may be used as target objects, and the target object may be determined based on a predetermined selection condition. For the method, please refer to the following description.

[0039] Optionally, the number of target objects includes at least one, and adding the virtual special effect model to the target object includes: adding a virtual special effect model corresponding to each of the at least one target object, or adding a virtual special effect model to a target object among the at least one target object that satisfies a predetermined condition; determining a target object corresponding to the audio information, and adding a virtual special effect model to the target object, and displaying the virtual special effect models corresponding to the other target objects transparently.

[0040] In this embodiment, when there are multiple target objects, adding virtual special effect models to the target objects may be such that corresponding virtual special effect models can be added to multiple target objects in the frame-in image, or the same virtual special effect model can be added. The method for determining the virtual special effect model may refer to the following description. A pre-marked object can be used as a target object to add a virtual special effect model to the target object. Another method is as follows: A plurality of objects in the display interface are obtained in real time, and a target object corresponding to audio information is determined in real time. A virtual special effect model is added to the target object corresponding to the audio information. Another method is as follows: After determining a virtual special effect model corresponding to each target object, audio information can be obtained in real time, and a virtual special effect model of the target object corresponding to the audio information can be displayed, while the virtual special effect models of other objects that do not emit sound can be displayed transparently.

[0041] S120, based on the collected audio information, control the movement of a target virtual part of a virtual special effect model corresponding to the target object, and adjust the target entity part of the target object to a predetermined state to obtain a target special effect.

[0042] The target virtual part is a part on the virtual special effects model. The target entity part is a five-visual part on the face of the target object. Optionally, the target virtual part is a mouth part of the virtual special effects model. The target entity part is a mouth part of the target object. The predetermined state may be any one of a smiling state and a closed state.

[0043] When the audio information is collected, the opening and closing of the mouth part of the virtual special effect part corresponding to the target object can be controlled. At the same time, the mouth part of the target object is adjusted to a smiling state. Adjusting the mouth part to a smiling state can be achieved through a pre-trained neural network. Based on the above method, the final target special effect video can be generated. When it is detected that a shooting stop condition is triggered, the final target special effect video is obtained.

[0044] In this embodiment, adjusting the target entity portion to a predetermined state may be adjusting the target entity portion from a first state to the predetermined state based on an entity portion adjustment model corresponding to the predetermined state.

[0045] The entity-part adjustment model is obtained by pre-training and is used to adjust a target entity part in a facial image from an open or closed state to a smiling state. The first state is a mouth state corresponding to any facial expression, such as open, smiling, etc. The predetermined state is an expected output state during model training, for example, the default state is a smiling state.

[0046] When the target object speaks, the target virtual part can be controlled to move, and at the same time the target entity part of the target object can be adjusted to a smiling state.

[0047] For example, when a target object emits audio information, the target object can move based on the mouth part of the virtual model of the audio information emitted, and at the same time, the mouth part of the target object can be adjusted to a smiling state to achieve a ventriloquism effect.

[0048] In the embodiment of the present application, in response to a special effect trigger operation, a virtual special effect model is added to a target object, and according to the collected audio information, the target virtual part of the virtual special effect model corresponding to the target object is controlled to move, and the target entity part of the target object is adjusted to a predetermined state to obtain a target special effect, which makes the video shooting content rich and interesting, and also achieves the effect of ventriloquism.

[0049] 2 is a flow chart of a method for determining a special effect video provided by an embodiment of the present application. Based on the above embodiment, it can be explained that "based on the collected audio information, the target virtual part of the virtual special effect model corresponding to the target object is controlled to move, and the target entity part of the target object is adjusted to a predetermined state to obtain a target special effect." For specific embodiments, please refer to the description of this embodiment. The same or corresponding technical terms as those in the above embodiment will not be described again in this specification.

[0050] As shown in FIG. 2, the method includes the following steps:

[0051] S210, in response to a special effect trigger operation, determining a special effect adding position corresponding to the target object, and adding the virtual special effect to the special effect adding position.

[0052] The special effect adding position includes any position on the trunk of the body of the target object, or any position in a predetermined vicinity of the center of the target object. Optionally, the any position may be a shoulder or any position a certain distance away from the trunk of the user. The predetermined region may be any position a certain range away from the edge contour of the user.

[0053] In this embodiment, determining a special effect adding position corresponding to a target object and adding a virtual special effect to the special effect adding position includes determining hand information of the target object, and in response to the hand information matching a predetermined hand shape, determining the special effect adding position as a hand part, and adding a virtual special effect model to the hand part.

[0054] The hand shape can be a hand swing shape. Many swing shapes can be preset and used as the predetermined hand shape.

[0055] A hand part of a target object is detected in real time, and if the shape of the hand part matches a predetermined hand shape, it is determined that the special effect adding position is the hand part. The determined virtual special effect model can be added to the hand part. In this case, the hand part is not visible to the user in the display interface. That is, the virtual special effect part occludes the hand part.

[0056] S220, controlling the movement of a target virtual part of a virtual special effect model corresponding to the target object according to the collected audio information.

[0057] Optionally, in response to collecting the audio information, a target virtual portion of the virtual special effects model corresponding to the target object is controlled to move, or in response to detecting that the detected audio information includes motion-triggering content, a target virtual portion of the virtual special effects model corresponding to the target object is controlled to move.

[0058] As long as audio information is collected, a target object corresponding to the audio information can be determined. The mouth part of the virtual special effect model added to the hand part of the target object can be driven to move. Furthermore, audio information of at least one target object can be obtained in real time to determine whether the audio information triggers mouth movement content. For example, if the triggered movement content is a word such as ventriloquism, the target virtual part of the virtual special effect part model can be controlled to move.

[0059] In this embodiment, controlling the movement of the target virtual part of the virtual special effect model corresponding to the target object may include three situations, and the corresponding implementation methods are described below. The first implementation method is as follows: the target virtual part is hidden and then displayed, and the target entity part is transferred as the target virtual part based on the pre-trained entity part transfer model, and the movement information of the target virtual part corresponds to the mouth movement information of the audio information.

[0060] Based on the entity-part transfer model obtained by pre-training, the mouth part of the target object in the face image can be cut out and transferred to the target virtual part, where the motion information of the target virtual part is consistent with the mouth motion information of the target entity part.

[0061] The second implementation method is as follows: obtain mouth movement information of the target entity part, and control the movement of the target virtual part according to the mouth movement information, so that the movement of the target virtual part is consistent with the mouth movement information.

[0062] The mouth movement information of the target entity part is obtained in real time, and the movement information of the target virtual part in the virtual special effect model is controlled to match the mouth movement information, so as to achieve the effect that audio information is output from the virtual special effect model.

[0063] The third realization method is as follows: Obtaining the movement information of the hand part, and controlling the target virtual part to move based on the movement information.

[0064] The movement information of the hand part can be detected in real time, and when the hand part moves, the target virtual part can be controlled to move, thereby achieving the effect of audio information being emitted from the target virtual part.

[0065] In practical applications, a situation may occur in which a user issues audio information but the user's hand does not move. In this case, in response to not obtaining the movement information of the hand, the target virtual part is controlled to move based on the audio information, and the movement information of the target virtual part corresponds to the mouth movement information of the audio information.

[0066] Dual driving of the target virtual part can be performed based on the audio information and the hand part. When the movement of the hand part is detected, the target virtual part is controlled to move based on the hand part, and reaches the target part and corresponding audio information is output. Alternatively, when the movement of the hand part is not detected, the target part can be directly controlled to move based on the audio information. The movement information of the target virtual part corresponds to the mouth movement information corresponding to the audio information.

[0067] S230, adjusting the target entity portion of the target object to a predetermined state.

[0068] In the process of controlling the movement of the target virtual part, the target entity part can be adjusted to a smiling state.

[0069] In the embodiment of the present application, after responding to a special effect trigger operation, a virtual special effect model can be added to the target object; after collecting audio information, the mouth part corresponding to the target object can be transferred to the target virtual part of the virtual special effect model, and the movement information of the target virtual part can be controlled to correspond to the mouth movement of the audio information; and the state of the mouth part of the target object can be adjusted to a predetermined state, so that the special effect video content is rich and interesting, and the effect of ventriloquism can be achieved.

[0070] 3 is a structural block diagram of a special effect image processing device provided by an embodiment of the present application, which can execute the special effect image processing method provided by any embodiment of the present application, and has corresponding function modules for executing the method. As shown in FIG. 3, the device includes a special effect adding module 310 and a target special effect video determining module 320.

[0071] The special effect adding module 310 is configured to add a virtual special effect model to a target object in response to a special effect trigger operation, and the target special effect animation determining module 320 is configured to control the movement of a target virtual part of the virtual special effect model corresponding to the target object based on the collected audio information, and adjust the target entity part of the target object to a predetermined state to obtain a target special effect.

[0072] Based on the above embodiment, responding to the special effect trigger operation includes at least one of detecting that the frame-in screen includes a target object, detecting a control that triggers a target special effect tool, detecting that the body movement of the target object in the frame-in screen matches the body movement of the added special effect, or detecting that audio information triggers a wake word for adding a special effect.

[0073] Based on the above embodiment, the special effect model adding module is further configured to determine a virtual special effect model corresponding to the target object from a virtual special effect library, where the virtual special effect library includes at least one virtual special effect model to be selected.

[0074] Based on the above embodiment, the special effect model adding module is configured to determine a virtual special effect model corresponding to the target object based on the basic attribute information of the target object.

[0075] Based on the above embodiment, the special effect model adding module is configured to obtain a pre-uploaded image to be processed, and determine a virtual special effect model based on at least one display object in the image to be processed.

[0076] Based on the above embodiment, the number of the target objects includes at least one, and the special effect model adding module is configured to add a virtual special effect model respectively corresponding to the at least one target object, or add a virtual special effect model to a target object among the at least one target object that satisfies a predetermined condition, determine a target object corresponding to the audio information, and add a virtual special effect model to the target object, and display virtual special effect models corresponding to other target objects among the at least one target object as transparent.

[0077] Based on the above embodiment, the special effect model adding module is configured to determine a special effect adding position corresponding to the target object, and add the virtual special effect to the special effect adding position, wherein the special effect adding position includes any position on the trunk of the body of the target object, or any position in a predetermined vicinity of the center of the target object.

[0078] Based on the above embodiment, the special effect model addition module is configured to determine hand information of the target object, and in response to the hand information matching a predetermined hand shape, determine the special effect addition portion as a hand portion, and add the virtual special effect model to the hand portion.

[0079] Based on the above embodiment, the target special effect animation determination module is configured to control a target virtual portion of the virtual special effect model corresponding to the target object to move in response to collecting audio information, or to control a target virtual portion of the virtual special effect model corresponding to the target object to move in response to detecting that the detected audio information includes motion trigger content.

[0080] Based on the above embodiment, the target special effect animation determination module is configured to hide and display the target virtual part, and transfer the target entity part to the target virtual part as the target virtual part based on a pre-trained entity part transfer model, and movement information of the target virtual part corresponds to mouth movement information of the audio information.

[0081] Based on the above embodiment, the target special effect animation determination module is configured to obtain mouth movement information of the target entity part, and control the target virtual part to move based on the mouth movement information, and the movement information of the target virtual part is consistent with the mouth movement information.

[0082] According to the above embodiment, the target special effect animation determining module is configured to obtain the movement information of the hand part, and control the target virtual part to move according to the movement information.

[0083] Based on the above embodiment, the target special effect animation determination module is further configured to, in response to not acquiring the movement information of the hand part, control the target virtual part to move based on the audio information, where the movement information of the target virtual part corresponds to the mouth movement information of the audio information.

[0084] Based on the above embodiment, the target special effect animation determination module is configured to adjust the target entity part from a first state to the predetermined state based on an entity part adjustment model corresponding to the predetermined state.

[0085] Based on the above embodiment, the target entity part is the mouth part of the target object, the target virtual part is the mouth part of the virtual special effect, and the predetermined state is a closing state or a smiling state of the mouth part.

[0086] In the embodiment of the present application, in response to a special effect trigger operation, a virtual special effect model is added to a target object, and according to the collected audio information, the target virtual part of the virtual special effect model corresponding to the target object is controlled to move, and the target entity part of the target object is adjusted to a predetermined state to obtain a target special effect, which makes the video shooting content rich and interesting, and also achieves the effect of ventriloquism.

[0087] The special effect image processing device provided by the embodiments of the present application can execute the special effect image processing method provided by any embodiment of the present application, and has functional modules corresponding to the execution of the method.

[0088] It should be noted that the multiple units and modules included in the above device are only divided according to functional logic, and are not limited to the above division as long as they can implement the corresponding functions; furthermore, the names of the multiple functional units are only for the convenience of distinguishing them from each other, and are not used to limit the scope of protection of the embodiments of the present application.

[0089] FIG. 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. Hereinafter, reference will be made to FIG. 4, which shows a structural schematic diagram applicable to an electronic device (e.g., a terminal device or server in FIG. 4) 400 for implementing an embodiment of the present application. The terminal device according to the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (Portable Media Players, PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG. 4 is merely an example and does not limit the functionality and scope of use of the embodiment of the present application.

[0090] 4, electronic device 400 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 401, which can perform various appropriate operations and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage device 408 into random access memory (RAM) 403. RAM 403 stores various programs and data necessary for the operation of electronic device 400. Processing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0091] Typically, devices such as input devices 406 including a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc., output devices 407 including a liquid crystal display (LCD), speaker, vibrator, etc., storage devices 408 including magnetic tape, hard disk, etc., and communication devices 409 may be connected to the I / O interface 405. The communication devices 409 may enable the electronic device 400 to communicate and exchange data with other devices wirelessly or via wires. While FIG. 4 illustrates the electronic device 400 with a variety of devices, it should be understood that not all of the devices shown are required to be implemented or available. Instead, more or fewer devices may be implemented or provided.

[0092] According to an embodiment of the present application, the processes described above with reference to the flowcharts may be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication device 409, or installed from the memory 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-described functions defined in the methods according to the embodiment of the present application are performed.

[0093] The names of messages or information exchanged between devices in the examples of this application are for illustrative purposes only and are not used to limit the scope of these messages or information.

[0094] The electronic device provided by the embodiments of the present application and the special effect image processing method provided by the above embodiments belong to the same inventive concept, and for technical content not described in detail in this embodiment, reference can be made to the above embodiments.

[0095] An embodiment of the present application provides a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, the special effect video processing method provided by the above embodiment is implemented.

[0096] It should be noted that the computer-readable medium in this application may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. The computer-readable storage medium may include an electrical connection with one or more leads, a portable computer floppy disk, a hard disk, RAM, ROM, an Erasable Programmable Read-Only Memory (EPROM) or flash memory, optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code. Such a propagated data signal can take many forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium can be any computer-readable medium, other than a computer-readable storage medium, that can transmit, propagate, or carry a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied in a computer-readable medium can be transmitted using any suitable medium, including, but not limited to, wire, fiber optic cable, radio frequency (RF), or the like, or any suitable combination thereof.

[0097] In some embodiments, clients and servers may communicate using any now known or later developed network protocol, such as HyperText Transfer Protocol (HTTP), and may communicate digital data in any form or medium (e.g., communications network) interconnection. Examples of communications networks include local area networks (LANs), wide area networks (WANs), international networks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any now known or later developed networks.

[0098] The computer-readable medium may be included in the electronic device, or may exist independently of the electronic device.

[0099] The computer-readable medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device adds a virtual special effect model to a target object in response to a special effect trigger operation, controls a target virtual part of the virtual special effect model corresponding to the target object to move based on the collected audio information, and adjusts a target entity part of the target object to a predetermined state to obtain a target special effect.

[0100] Computer program code for carrying out the operations of the present application may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages such as Java, Smalltalk, C++, etc., as well as conventional procedural programming languages such as "C" or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In situations involving remote computers, the remote computer may be connected to the user's computer via any type of network, such as a LAN or WAN, or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to many embodiments of the present application. In this regard, each block in the flowcharts or block diagrams may represent a module, program segment, or portion of code, which includes one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions shown in the blocks may be executed in a different order than that shown in the drawings. For example, two blocks shown one after the other may actually be executed substantially in parallel or may be executed in the reverse order, depending on the functionality involved. Furthermore, each block of the block diagrams and / or block diagrams, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of specialized hardware and computer instructions.

[0102] The units included in the embodiments of the present application may be implemented in software or hardware. The names of the units may not limit the units themselves in some cases, for example, the first obtaining unit may also be described as "a unit for obtaining at least two Internet Protocol addresses."

[0103] The functions described herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, examples of the types of hardware logic components that may be used include Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Parts (ASSPs), Systems on Chips (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0104] In the context of the present application, a machine-readable medium may be a tangible medium that can contain or store a program used by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine signal medium or a machine-readable storage medium. A machine-readable medium includes, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples of a machine-readable storage medium may include one or more wire-based electrical connections, portable computer disks, hard disks, RAM, ROM, EPROM or flash memory, optical fiber, CD-ROM, optical storage device, magnetic storage device, or any suitable combination thereof. According to one or more embodiments of the present application, [Example 1] provides a method for determining a special effect animation, the method including: adding a virtual special effect model to a target object in response to a special effect trigger operation; controlling a target virtual portion of the virtual special effect model corresponding to the target object to move based on collected audio information; and adjusting a target entity portion of the target object to a predetermined state to obtain a target special effect.

[0105] According to one or more embodiments of the present application, [Example 2] provides a method for determining a special effect video, and optionally, responding to a special effect trigger operation includes at least one of detecting that a target object is included in the frame-in screen, detecting a control that triggers a target special effect tool, detecting that a body movement of the target object in the frame-in screen matches a body movement of an added special effect, or detecting that audio information triggers a wake word for adding a special effect.

[0106] According to one or more embodiments of the present application, [Example 3] provides a method for determining a special effect video, optionally further including, before adding a virtual special effect model to the target object, determining a virtual special effect model corresponding to the target object from a virtual special effect library, wherein the virtual special effect library includes at least one virtual special effect model to be selected.

[0107] According to one or more embodiments of the present application, [Example 4] provides a method for determining a special effect video, and optionally, determining a virtual special effect model corresponding to the target object from a virtual special effect library includes determining a virtual special effect model corresponding to the target object based on basic attribute information of the target object.

[0108] According to one or more embodiments of the present application, [Example 5] provides a method for determining a special effect video, optionally further including, before adding a virtual special effect model to a target object, acquiring an image to be processed that has been uploaded in advance, and determining a virtual special effect model based on at least one display object in the image to be processed.

[0109] According to one or more embodiments of the present application, [Example 6] provides a method for determining a special effect video, optionally, the number of the target objects includes at least one, and adding a virtual special effect model to the target object includes adding a virtual special effect model corresponding to each of the at least one target object, or adding a virtual special effect model to a target object among the at least one target object that satisfies a predetermined condition; determining a target object corresponding to the audio information, adding a virtual special effect model to the target object, and displaying virtual special effect models corresponding to other target objects among the at least one target object transparently.

[0110] According to one or more embodiments of the present application, [Example 7] provides a method for determining a special effect animation, and optionally, adding a virtual special effect model to a target object includes determining a special effect addition position corresponding to the target object and adding the virtual special effect to the special effect addition position, wherein the special effect addition position includes any position in the trunk of the body of the target object or any position in a predetermined vicinity centered on the target object.

[0111] According to one or more embodiments of the present application, [Example 8] provides a method for determining a special effect animation, and optionally, determining a special effect addition position corresponding to the target object and adding the virtual special effect to the special effect addition position includes determining hand information of the target object, and in response to the hand information matching a predetermined hand appearance, determining the special effect addition position as a hand part, and adding the virtual special effect model to the hand part.

[0112] According to one or more embodiments of the present application, [Example 9] provides a method for determining a special effect animation, and optionally, controlling a target virtual portion of a virtual special effect model corresponding to the target object to move based on collected audio information includes controlling a target virtual portion of a virtual special effect model corresponding to the target object to move in response to collecting audio information, or controlling a target portion of a virtual special effect model corresponding to the target object to move in response to detecting that the detected audio information includes movement trigger content.

[0113] According to one or more embodiments of the present application, [Example 10] provides a method for determining a special effect video, which optionally further includes hiding and displaying the target virtual part, and transferring the target entity part to the target virtual part as the target virtual part based on a pre-trained entity part transfer model, wherein the movement information of the target virtual part corresponds to the mouth movement information of the audio information.

[0114] According to one or more embodiments of the present application, [Example 11] provides a method for determining a special effect animation, and optionally, controlling the movement of a target virtual part of a virtual special effect model corresponding to the target object includes obtaining mouth movement information of the target entity part and controlling the movement of the target virtual part based on the mouth movement information, and the movement information of the target virtual part is consistent with the mouth movement information.

[0115] According to one or more embodiments of the present application, [Example 12] provides a method for determining a special effect animation, optionally, the virtual special effect model is positioned at a hand portion of the target object, and controlling the target virtual portion of the virtual special effect model corresponding to the target object to move includes obtaining movement information of the hand portion and controlling the target virtual portion to move based on the movement information.

[0116] According to one or more embodiments of the present application, [Example 13] provides a method for determining a special effect animation, which optionally further includes, in response to not obtaining movement information of the hand part, controlling the target virtual part to move based on the audio information, and the movement information of the target virtual part corresponds to mouth movement information of the audio information.

[0117] According to one or more embodiments of the present application, [Example 14] provides a method for determining a special effect video, and optionally, adjusting a target entity part of the target object to a predetermined state includes adjusting the entity part from a first state to the predetermined state based on an entity part adjustment model corresponding to the predetermined state.

[0118] According to one or more embodiments of the present application, [Example 15] provides a method for determining a special effect animation, optionally, wherein the target entity part is a mouth part of the target object, the target virtual part is a mouth part of the virtual special effect, and the predetermined state is a closed state or a smiling state of the mouth part.

[0119] According to one or more embodiments of the present application, [Example 16] provides a special effect animation determination device, the device including: a special effect adding module configured to add a virtual special effect model to a target object in response to a special effect trigger operation; and a target special effect animation determination module configured to control the movement of a target virtual part of the virtual special effect model corresponding to the target object based on collected audio information, and adjust a target entity part of the target object to a predetermined state to obtain a target special effect.

[0120] The above description is merely a description of the embodiments and applied technical principles of the present application. For those skilled in the art, the disclosure scope related to this application is not limited to the technical solution formed by a specific combination of the above technical features, but also needs to cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above concept. For example, the technical solution can be formed by (but is not limited to) replacing the above features with technical features having similar functions that are applied to this application.

[0121] It should be noted that, although various operations are shown in a particular order, this should not be understood as requiring the operations to be performed in the particular order or sequential order shown. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several implementation details are included in the above description, these should not be construed as limiting the scope of the present application. Some features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Claims

1. A method for determining a special effect video, comprising: In response to a special effect trigger operation, adding a virtual special effect model to a target object, wherein the virtual special effect model is a character model, an animal model, a cartoon pattern model, or a scenery model, and the target object is an object in a frame-in screen; A method for determining a special effect animation, comprising: controlling the movement of a target virtual part of a virtual special effect model corresponding to the target object based on the collected audio information; and adjusting the target entity part of the target object to a predetermined state to obtain a target special effect, wherein the target entity part is a five sense parts on the face of the target object.

2. Responding to special effect trigger operations includes: detecting that the target object is included in the frame-in screen; detecting a control that triggers a target special effects tool; Detecting that the body movement of the target object in the frame-in screen matches the body movement of the added special effect; or The method of claim 1 , comprising at least one of detecting audio information to trigger a wake word for adding special effects.

3. Before adding the virtual special effect model to the target object, determining a virtual special effects model corresponding to the target object from a virtual special effects library; The method of claim 1 , wherein the virtual special effects library includes at least one virtual special effects model from which to select.

4. determining a virtual special effects model corresponding to the target object from the virtual special effects library; The method of claim 3 , further comprising determining a virtual special effects model corresponding to the target object based on basic attribute information of the target object.

5. Before adding the virtual special effect model to the target object, The method of claim 1 , further comprising: obtaining a previously uploaded image to be processed; and determining a virtual special effects model based on at least one display object in the image to be processed.

6. The number of the target objects includes at least one, and adding a virtual special effect model to the target object includes: adding a virtual special effects model corresponding to each of the at least one target object; or adding a virtual special effects model to a target object of the at least one target object that satisfies a predetermined condition; 2. The method of claim 1, further comprising: determining a target object corresponding to the audio information; adding a virtual special effects model to the target object; and displaying virtual special effects models corresponding to other target objects of the at least one target object as transparent.

7. Adding a virtual special effects model to a target object determining a special effect addition location corresponding to the target object; and adding the virtual special effect model to the special effect addition location; The method of claim 1 , wherein the special effect addition location comprises any location on the trunk of the body of the target object, or any location in a predetermined vicinity of a center of the target object.

8. determining a special effect adding position corresponding to the target object and adding the virtual special effect model to the special effect adding position; determining hand information of the target object, and determining the special effect adding position as a hand part in response to the hand information matching a predetermined hand shape; and adding the virtual special effects model to the hand region.

9. controlling a target virtual portion of a virtual special effect model corresponding to the target object to move based on the collected audio information; controlling movement of a target virtual portion of a virtual special effects model corresponding to the target object in response to collecting the audio information; or 10. The method of claim 1, further comprising controlling a target virtual portion of a virtual special effects model corresponding to the target object to move in response to the detected audio information including motion trigger content.

10. Controlling the movement of a target virtual portion of a virtual special effect model corresponding to the target object includes:

2. The method of claim 1, comprising: hiding and displaying the target virtual part, and transferring the target entity part to the target virtual part based on a pre-trained entity part transfer model, wherein movement information of the target virtual part corresponds to a mouth movement of the target entity part.

11. Controlling the movement of a target virtual portion of a virtual special effect model corresponding to the target object includes: acquiring mouth movement information of the target entity part, and controlling the target virtual part to move based on the mouth movement information; The method of claim 1 , wherein the target virtual body part movement information is consistent with the mouth movement information.

12. The virtual special effect model is positioned at a hand portion of the target object, and the target virtual portion of the virtual special effect model corresponding to the target object is controlled to move, The method of claim 1 , further comprising acquiring motion information of the hand part and controlling the target virtual part to move based on the motion information.

13. and controlling the target virtual part to move based on the audio information in response to not acquiring the movement information of the hand part. The method of claim 12 , wherein the target virtual body part movement information corresponds to mouth movement information of the audio information.

14. adjusting the target entity portion of the target object to a predetermined state; The method of claim 1 , further comprising adjusting the target entity part from a first state to a predetermined state based on an entity part adjustment model corresponding to the predetermined state.

15. The method of claim 1 , wherein the target entity part is a mouth part of the target object, the target virtual part is a mouth part of a virtual special effect, and the predetermined state is a closed state or a smiling state of the mouth part.

16. A special effect video determining device, comprising: a special effect adding module configured to add a virtual special effect model to a target object in response to a special effect trigger operation, the virtual special effect model being a character model, an animal model, a cartoon pattern model, or a scenery model, and the target object being an object in a frame-in screen; A special effect animation determination device including: a target special effect animation determination module configured to control the movement of a target virtual part of a virtual special effect model corresponding to the target object based on the collected audio information, and to adjust the target entity part of the target object to a predetermined state to obtain a target special effect, wherein the target entity part is a five-sense part on the face of the target object.

17. An electronic device, at least one processor; a storage device configured to store at least one program; An electronic device, wherein when said at least one program is executed by said at least one processor, said at least one processor implements the method for determining special effect moving images according to any one of claims 1 to 15.

18. 16. A storage medium containing computer-executable instructions, the computer-executable instructions, when executed by a computer processor, used to perform the method for determining special effects video of any one of claims 1 to 15.

Images