Video generation method and device, electronic equipment and storage medium

Abstract

The invention provides a video generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining and displaying a target image, the target image comprising at least one object; in response to a trigger operation for the target image, determining a trigger position corresponding to the trigger operation, and determining a target object corresponding to the trigger position from the target image; separating the target object from the target image to obtain a target object layer, and generating an animation special effect associated with the target object based on the target object layer; and inputting the target object layer into the preset model to obtain the description information matched with the target object layer, and generating the encyclopedia video associated with the target object based on the animation special effect associated with the target object and the description information, so that the richness of the presentation mode and the presentation content of the target object can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video generating method, a video generating device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of computer technology, users' demand for encyclopedic information has gradually increased. Information retrieval through terminals is also one of the main means for users to obtain information. For example, users can search for encyclopedic knowledge (such as history, culture, science, technology, etc.) on some websites through terminals.

[0003] However, the information obtained through the above methods is usually static information, and the information presentation method and content are relatively simple. Users can only absorb information in a one-way manner, resulting in a lack of interactivity in the information acquisition process, which affects the user experience. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a video generation method, device, electronic device and storage medium, which can generate an encyclopedia video of a target object, thereby improving the presentation method and richness of the presentation content for the target object.

[0005] The present disclosure provides a video generation method, including:

[0006] Acquire and display a target image, wherein the target image includes at least one object;

[0007] In response to a trigger operation on the target image, determining a trigger position corresponding to the trigger operation, and determining a target object corresponding to the trigger position from the target image;

[0008] Separating the target object from the target image to obtain a target object layer, and generating an animation effect associated with the target object based on the target object layer;

[0009] The target object layer is input into a preset model to obtain description information matching the target object, and based on the animation effects associated with the target object and the description information, an encyclopedia video associated with the target object is generated.

[0010] In a possible implementation, separating the target object from the target image to obtain a target object layer, and generating an animation effect associated with the target object based on the target object layer includes:

[0011] The target object is separated from the target image to obtain a background layer and the target object layer, and an animation effect associated with the target object is generated based on the background layer, the target object layer and preset virtual materials.

[0012] In a possible implementation, the preset virtual material includes a virtual facial material, and the virtual facial material includes at least one of a virtual eye, a virtual mouth, and a virtual leg.

[0013] In a possible implementation, generating an animation effect associated with the target object based on the background layer, the target object layer and preset virtual materials includes:

[0014] Inputting the background layer into a preset image restoration model to obtain a restored background layer;

[0015] Based on the restored background layer, the target object layer and the preset virtual material, an animation effect associated with the target object is generated.

[0016] In a possible implementation manner, inputting the target object layer into a preset model to obtain description information matching the target object includes:

[0017] Inputting the target object layer into a preset text information generation model to obtain text information matching the target object layer;

[0018] The text information is input into a preset description information generation model to obtain description information matching the target object.

[0019] In a possible implementation, the generating an encyclopedia video associated with the target object based on the animation special effects associated with the target object and the description information includes:

[0020] Based on speech synthesis technology, the description information is converted into audio information, and based on the animation special effects and the audio information, an encyclopedia video associated with the target object is generated.

[0021] In a possible implementation manner, determining the target object corresponding to the trigger position from the target image includes:

[0022] Performing image segmentation processing on the target image to obtain multiple semantic regions; each semantic region corresponds to an object;

[0023] Pixel information of the trigger position is determined, and based on the pixel information of the trigger position and the multiple semantic areas, a target object corresponding to the trigger position is determined.

[0024] The present disclosure provides a video generation device, including:

[0025] An image acquisition module, used to acquire and display a target image, wherein the target image includes at least one object;

[0026] an object determination module, configured to determine, in response to a trigger operation on the target image, a trigger position corresponding to the trigger operation, and determine a target object corresponding to the trigger position from the target image;

[0027] An animation generation module, used for separating the target object from the target image to obtain a target object layer, and generating an animation effect associated with the target object based on the target object layer;

[0028] The video generation module is used to input the target object layer into a preset model to obtain description information matching the target object, and generate an encyclopedia video associated with the target object based on the animation effects associated with the target object and the description information.

[0029] In a possible implementation, the animation generation module is specifically used to:

[0030] The target object is separated from the target image to obtain a background layer and the target object layer, and an animation effect associated with the target object is generated based on the background layer, the target object layer and preset virtual materials.

[0031] In a possible implementation, the preset virtual material includes a virtual facial material, and the virtual facial material includes at least one of a virtual eye and a virtual mouth.

[0032] In a possible implementation, the animation generation module is specifically used to:

[0033] Inputting the background layer into a preset image restoration model to obtain a restored background layer;

[0034] Based on the restored background layer, the target object layer and the preset virtual material, an animation effect associated with the target object is generated.

[0035] In a possible implementation manner, the video generation module is specifically used to:

[0036] Inputting the target object layer into a preset text information generation model to obtain text information matching the target object layer;

[0037] The text information is input into a preset description information generation model to obtain description information matching the target object.

[0038] In a possible implementation manner, the video generation module is specifically used to:

[0039] Based on speech synthesis technology, the description information is converted into audio information, and based on the animation special effects and the audio information, an encyclopedia video associated with the target object is generated.

[0040] In a possible implementation manner, the object determination module is specifically configured to:

[0041] Performing image segmentation processing on the target image to obtain multiple semantic regions; each semantic region corresponds to a target object;

[0042] Pixel information of the trigger position is determined, and based on the pixel information of the trigger position and the multiple semantic areas, a target object corresponding to the trigger position is determined.

[0043] An embodiment of the present disclosure also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the video generation method described in any one of the above possible implementation modes are performed.

[0044] The embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the video generating method described in any possible implementation manner described above are executed.

[0045] In the disclosed embodiment, the target object is separated from the target image to obtain a target object layer, and an animation effect is generated based on the target object layer. At the same time, the target object layer is input into a preset model to obtain description information of the target object, and an encyclopedia video associated with the target object is generated based on the animation effect and the description information. In this way, the presentation method and the richness of the presentation content for the target object can be improved, and the relevant information of the target object can be actively output to the user through the encyclopedia video, which is conducive to improving the interactivity of the user in the process of acquiring information while reducing the degree of initiative of the user in acquiring information.

[0046] In addition, since the target object is selected by the user, the generated encyclopedia video is more targeted, which is conducive to improving the user's interactive experience.

[0047] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0049] Figure 1 A flowchart of a video generation method provided by some embodiments of the present disclosure is shown;

[0050] Figure 2 A schematic diagram showing a target image provided by some embodiments of the present disclosure is shown;

[0051] Figure 3 A schematic diagram showing animation special effects provided by some embodiments of the present disclosure is shown;

[0052] Figure 4 A schematic diagram showing animation special effects provided by other embodiments of the present disclosure is shown;

[0053] Figure 5 A schematic diagram showing a screen of an encyclopedia video provided by some embodiments of the present disclosure;

[0054] Figure 6 A schematic diagram showing animation special effects provided by some other embodiments of the present disclosure;

[0055] Figure 7 A schematic diagram showing a deformation processing process provided by some embodiments of the present disclosure;

[0056] Figure 8 A schematic diagram showing a video generating device provided by some embodiments of the present disclosure is shown;

[0057] Fig. 9 A schematic diagram of an electronic device provided by some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0059] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0060] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0061] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0062] In addition, the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0063] With the development of computer technology, users' demand for encyclopedic information has gradually increased. Information retrieval through terminals is also one of the main means for users to obtain information. For example, users can search for encyclopedic knowledge (such as history, culture, science, technology, etc.) on some websites through terminals.

[0064] Research has found that the information obtained through the above methods is usually static information, and the way and content of information presentation are relatively simple. Users can only absorb information in a one-way manner, resulting in a lack of interactivity in the information acquisition process, which affects the user experience.

[0065] Based on the above research, the present disclosure provides a video generation method, device, electronic device and storage medium, which can acquire and display a target image, wherein the target image includes at least one object; in response to a trigger operation on the target image, determine a trigger position corresponding to the trigger operation, and determine a target object corresponding to the trigger position from the target image; separate the target object from the target image to obtain a target object layer, and generate an animation effect associated with the target object based on the target object layer; input the target object layer into a preset model to obtain description information matching the target object, and generate an encyclopedia video associated with the target object based on the animation effect associated with the target object and the description information. In this way, the presentation method and richness of the presentation content for the target object can be improved, and the relevant information of the target object can be actively output to the user through the encyclopedia video, which is conducive to improving the interactivity of the user in the process of acquiring information while reducing the degree of initiative of the user in acquiring information.

[0066] To facilitate understanding of the present embodiment, the execution subject of the video generation method provided by the embodiment of the present disclosure is first introduced in detail. The execution subject of the video generation method provided by the embodiment of the present disclosure is an electronic device, and the electronic device may include a terminal device or a server. For example, the electronic device is a server, and the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data and artificial intelligence platforms. In other embodiments, if the electronic device is a terminal device, the terminal device may be a mobile device, a user terminal, a handheld device, a computing device, a wearable device, etc. In addition, the video generation method can also be implemented by a processor calling a computer-readable instruction stored in a memory.

[0067] The video generation method provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings. Figure 1 FIG. 1 is a flowchart of a video generation method provided by an embodiment of the present disclosure, the method comprising steps S101 to S103, wherein:

[0068] S101, acquiring and displaying a target image, wherein the target image includes at least one object.

[0069] Optionally, the target image may be an image uploaded by a user or a frame of an image in a video uploaded by a user. In a specific implementation, a user may input an image or a video in a preset upload interface to obtain the target image and display it through an electronic device.

[0070] Optionally, the target image may also be an image taken by the user in real time or a frame of image in a video taken by the user in real time. In a specific implementation, the user may use the recognition function (such as a scanning function or a shooting function) of the target application installed on the electronic device to shoot the physical object that the user currently wants to know in real time. In this way, the target image can be acquired and displayed through the electronic device.

[0071] The target image includes at least one object. Here, since the target image can be any image, the object in the target image can be any type of object, such as a person, an animal, a plant, an object, etc.

[0072] Exemplarily, if the target image is a road image, the objects included in the target image may be motor vehicles, non-motor vehicles, pedestrians, etc., wherein motor vehicles may be different types of vehicles, such as cars, trucks, motorcycles, buses, etc., which are not limited here.

[0073] For another example, if the target image is an image taken in a botanical garden, the object in the target image may be a plant, wherein the plant may be a seed plant, a moss plant, a fern plant, an algae plant, etc., such as a tree, a shrub, a grass, a lichen, or a potted plant.

[0074] For another example, if the target image is an image taken in a zoo, the object in the target image may be an animal, such as a tiger, an elk, a lion, a wolf, etc., which is not limited here.

[0075] For example, see Figure 2 , which is a schematic diagram of a target image provided by some embodiments of the present disclosure. Figure 2 As shown in FIG. 2 , the target image 20 includes a plurality of objects, such as an object 21 (cactus), an object 22 (pine tree), an object 23 (willow tree), and an object 24 (potted plant).

[0076] S102: In response to a trigger operation on the target image, determine a trigger position corresponding to the trigger operation, and determine a target object corresponding to the trigger position from the target image.

[0077] Here, when the target image is displayed through an electronic device, the user can perform a trigger operation on any position of the displayed target image, so that the trigger position corresponding to the trigger operation can be determined. Here, the trigger position refers to the position in the screen coordinate system. In this way, after determining the trigger position, the target object corresponding to the trigger position can be determined from the target image.

[0078] For example, please refer again to Figure 2 , if the trigger position is point 25, it can be determined that the target object corresponding to point 25 is object 21, that is, the cactus.

[0079] Optionally, when determining the target object corresponding to the trigger position from the target image, the target image can be firstly segmented to obtain multiple semantic areas, and then the pixel information of the trigger position is determined; and based on the pixel information of the trigger position and the multiple semantic areas, the target object corresponding to the trigger position is determined.

[0080] Among them, each semantic region corresponds to an object.

[0081] Image segmentation is the technology and process of dividing an image into several specific regions with unique properties and proposing objects of interest. From a mathematical point of view, image segmentation is the process of dividing a digital image into non-overlapping regions.

[0082] Here, the target image can be segmented based on a preset image segmentation model. In this embodiment, a semantic segmentation model is used to perform image segmentation on the target image. Specifically, the semantic segmentation model assigns a semantic label to each pixel of the target image, that is, classifies it into a specific category, thereby obtaining multiple semantic regions.

[0083] It can be understood that since the trigger position corresponding to each trigger operation has corresponding coordinates in the screen coordinate system, the pixel information corresponding to the coordinates in the target image can be determined based on the coordinates. Since each pixel in the target image corresponds to a semantic area, and each semantic area corresponds to an object, the target object corresponding to the trigger position can be determined based on the pixel information. In this way, the accuracy of determining the target object can be improved.

[0084] In other implementations, the target image may be segmented based on other semantic segmentation models, or may be segmented based on any one of an instance segmentation model, a bounding box segmentation model, and a contour segmentation model, which is not limited here.

[0085] S103, separating the target object from the target image to obtain a target object layer, and generating an animation effect associated with the target object based on the target object layer.

[0086] It can be understood that after the target object is determined, the target object can be separated from the target image along the contour of the target object to obtain a target object layer.

[0087] Among them, animation special effects can include sound effects and visual effects. Sound effects can be audio or background music added to the target object, and visual effects can refer to visual effects added to the target object through special technical means, such as virtual materials, filters, light and shadow effects, and motion effects (such as the target object moving, jumping, and shaking in the image) added to the target object. In this way, the presentation effect of the target object can be richer.

[0088] Optionally, the virtual material may include a virtual facial material, and the virtual facial material may include at least one of a virtual eye and a virtual mouth, and may also include virtual eyebrows, virtual ears, virtual noses, etc., which are not limited here. The method of adding the virtual facial material will be introduced later. In other embodiments, the virtual material may also include virtual legs, virtual arms, or virtual decorations (such as a virtual hat), etc.

[0089] For example, see Figure 3 , which is a schematic diagram of animation special effects provided by some embodiments of the present disclosure. Figure 3 As shown in , taking a frame of animation special effects picture 30 as an example, the target object layer included in the picture 30 is the layer of the object 21 (ie, the cactus), and the virtual eye 26 is attached to the object 21.

[0090] The virtual material may be obtained by searching from a preset virtual material library, or may be generated based on a material generation tool, which is not limited here.

[0091] Here, a multi-frame animation can be generated based on the target object layer, and animation effects associated with the target object can be generated based on the multi-frame animation. For example, a virtual mouth in a closed state can be added to the target object layer to generate a first frame of animation, and a virtual mouth in an open state can be added to the target object layer to generate a second frame of animation. In this way, animation effects can be generated based on the first frame of animation and the second frame of animation.

[0092] In some implementations, after obtaining the target object layer, the target object layer may be deformed, and based on the deformed target object layer, an animation effect associated with the target object may be generated.

[0093] Here, deforming the target object layer refers to adjusting the shape of the target object layer, for example, stretching or shrinking the target object layer. The deformation process of the target object layer will be described in detail later.

[0094] For example, see Figure 4 , which is a schematic diagram of animation special effects provided by other embodiments of the present disclosure. Figure 4As shown in , still taking a frame of animation special effects screen as an example, the target object layer included in the screen is the layer of object 21 (that is, cactus), and a virtual eye 26 is attached to object 21. It can be seen from the figure that the shape of object 21 is the shape after stretching and deformation processing, and as the shape of object 21 changes, the shape of virtual eye 26 will also be stretched and changed accordingly.

[0095] In this implementation, by deforming the target object layer, the target object can be made to present different forms, thereby improving the richness of the animation special effects.

[0096] S104, inputting the target object layer into a preset model to obtain description information matching the target object, and generating an encyclopedia video associated with the target object based on the animation effects associated with the target object and the description information.

[0097] The description information may refer to information used to describe the target object.

[0098] Specifically, as introduced in the previous text, the target object can be a person, an animal, a plant, an object, etc. In some embodiments, if the target object is a person, the descriptive information that matches the target object may include information such as the person's profile, related persons, the person's experience, and the person's works; if the target object is an animal, the descriptive information that matches the target object may include the animal's name, animal habits, and living environment; if the target object is a plant, the descriptive information that matches the target object may include the plant name, category, growth environment, etc.; if the target object is an object, the descriptive information that matches the target object may include information such as the object name, usage method, and material, which is not limited here.

[0099] For example, if the target object layer is "cat", the target object layer is input into the preset model to obtain descriptive information matching "cat". For example, the descriptive information can be "I am a cat. I live in a wide range, from tropical rainforests, desert hills to cold grasslands and plateaus. I can be distributed on every land except Antarctica. Don't be fooled by my cute appearance, I have sharp claws and teeth."

[0100] For example, take over Figure 3 For example, if the target object layer is a cactus, the description information that matches the cactus may be "I am a cactus. I am named so because I look like a palm with fingers put together and I have thorns. I like warm, dry, sunny and well-ventilated environments. In addition to my ornamental value, I also have medicinal value of promoting qi and blood circulation, cooling blood and stopping bleeding, and detoxifying and reducing swelling."

[0101] In this way, after obtaining the animation effects and description information associated with the target object, an encyclopedia video associated with the target object can be generated. In this way, the presentation method and richness of the presentation content for the target object can be improved, and the relevant information of the target object can be actively output to the user through the encyclopedia video, which reduces the degree of user initiative in acquiring information and is conducive to improving the interactivity of the user in the process of acquiring information.

[0102] For example, see Figure 5 , which is a schematic diagram of a screen of an encyclopedia video provided by some embodiments of the present disclosure. Figure 5 As shown in , the screen 50 of the encyclopedia video associated with the target object displays the target object 21 (i.e., a cactus) and a virtual eye 26 attached to the target object 21. In addition, the screen can also display description information 27, wherein the description information can be displayed in the form of subtitles or barrage.

[0103] It should be noted that, since the encyclopedia video includes multiple frames, the posture of the target object 21 and the displayed description information may be different in different frames. In addition, the display style and display method of the description information may be set according to actual needs. For example, the font, font size, color and other information of the description information may be preset, and the display effects of the description information (such as fade-in and fade-out method, etc.) may also be preset.

[0104] In addition, in an embodiment of the present disclosure, when an encyclopedia video associated with the target object is generated based on the animation effects associated with the target object and the description information, the description information can be converted into audio information based on speech synthesis technology, and the encyclopedia video associated with the target object can be generated based on the animation effects and the audio information.

[0105] Among them, the text-to-speech (TTS) technology is used to convert text into speech. Specifically, during the conversion process, corresponding text processing (such as sentence segmentation, determination of stress position), acoustic modeling (such as determination of pitch, volume, etc.), timbre selection (such as timbre of different emotions, male voice, female voice, child voice, etc.) and noise reduction can be performed.

[0106] It can be understood that after determining the description information, the description information can be converted into voice information based on speech synthesis technology, so that the animation effects associated with the target object can be combined with the voice information to generate an encyclopedia video associated with the target object.

[0107] After the encyclopedia video is generated, it can be played on electronic devices. At this time, when the user is watching the encyclopedia video, he can not only watch the animation effects but also hear the corresponding voice information. In this way, the user can obtain the description information of the target object from both visual and auditory senses.

[0108] In some embodiments, the description information can be displayed in a combination of text form (such as subtitles, barrage) and voice form (voice broadcast), so that the user can understand the target object both visually and auditorily, thereby deepening the user's understanding of the target object.

[0109] Optionally, when the target object layer is input into a preset model to obtain description information matching the target object, the following steps (1) to (2) may be included:

[0110] (1) Inputting the target object layer into a preset text information generation model to obtain text information matching the target object layer.

[0111] Here, the text information may be simple text information describing the target object layer.

[0112] Exemplarily, if the target object layer is a "cactus" layer, the layer is input into the text information generation model, and the obtained text information is "cactus".

[0113] Among them, the preset text information generation model can be a multimodal model, which is a deep learning model that can process and fuse multiple different types of data or information modalities, such as text, images, voice, sensor data, etc.

[0114] In this embodiment, the multimodal model architecture may include any one of a multimodal neural network, a transformer architecture, and a multimodal generative adversarial network (GANs), without specific limitation herein.

[0115] (2) Inputting the text information into a preset description information generation model to obtain description information matching the target object.

[0116] Among them, the preset description information generation model can be a deep learning model, such as a big language model. A big language model refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. The big language model can handle a variety of natural language tasks, such as text classification, question and answer, dialogue, etc.

[0117] In this way, after the text information is input into the preset description information generation model, the description information matching the target object can be obtained, so that the description information is more comprehensive and closer to the target object.

[0118] For example, if the text information is "cactus", the text information is input into the description information generation model to obtain the description information "I am a cactus. I am named so because I look like a palm with fingers put together and have thorns. I like warm, dry, sunny and well-ventilated environments. In addition to my ornamental value, I also have medicinal value of promoting qi and blood circulation, cooling blood and stopping bleeding, and detoxifying and reducing swelling."

[0119] In some embodiments, when separating the target object from the target image to obtain the target object layer, and generating an animation effect associated with the target object based on the target object layer, the target object can be separated from the target image to obtain a background layer and the target object layer, and generating an animation effect associated with the target object based on the background layer, the target object layer and preset virtual materials.

[0120] It can be understood that since the target image may include other objects and a background in addition to the target object, the target object can be separated from the target image to obtain a background layer and a target object layer. Then, based on the background layer, the target object layer and the virtual material, an animation effect associated with the target object can be generated. In this way, the animation effect can be made more complete.

[0121] Here, it can be understood that the background layer is fixed in the animation special effects, and the target object layer can be suspended on the background layer.

[0122] For example, see Figure 6 , which is a schematic diagram of animation special effects provided by some other embodiments of the present disclosure. Figure 6 As shown in , taking a frame of animation special effects as an example, the frame of animation special effects 60 includes a target object layer 21, a virtual material 26, and objects 22, 23, and 24 forming a background layer.

[0123] After generating animation effects associated with the target object based on the background layer, the target object layer and the virtual material, an encyclopedia video can be further generated based on the animation effects and the description information, wherein the target object layer, the background layer, the virtual material and the description information are displayed in the screen of the encyclopedia video.

[0124] Optionally, in order to make the background layer more complete, in this embodiment, the background layer can also be input into a preset image restoration model to obtain a restored background layer, and then based on the restored background layer, the target object layer and the preset virtual material, an animation effect associated with the target object is generated.

[0125] Image restoration refers to the process of reconstructing the missing content of an image based on the remaining content of the image through computer processing methods such as deep learning models.

[0126] Commonly used image restoration models may include interpolation-based methods, such as bilinear interpolation, bicubic interpolation, etc.; methods based on generative adversarial networks (GANs); methods based on convolutional neural networks (CNNs) and methods based on image decomposition. In this embodiment, the image restoration model may be any one of the above restoration models, and is not specifically limited here.

[0127] It can be understood that after the target object is separated from the target image, there is no image information at the initial position of the target object. Therefore, in order to avoid missing information in the generated animation special effects, the background layer can be restored to obtain a restored background layer. In this way, based on the restored background layer, the target object layer and the virtual material, a complete animation special effect can be generated, thereby improving the viewing experience of the animation special effects.

[0128] The following is a detailed introduction to the deformation process of the target object layer.

[0129] Specifically, when deforming the target animation layer, a bounding rectangle can be created based on the target object layer, wherein the bounding rectangle includes multiple vertices, and then a target vertex is determined from the multiple vertices, and the target vertex is used as a fixed anchor point, and finally other vertices in the bounding rectangle except the fixed anchor point are controlled to move to deform the target animation layer.

[0130] The target vertex may be any one or more of the multiple vertices, which is not limited here, and the fixed anchor point refers to a point whose position is fixed.

[0131] For example, see Figure 7 , which is a schematic diagram of the deformation processing process provided by some embodiments of the present disclosure. Figure 7 As shown in , the target object layer 71 has a circumscribed rectangle 72, and the circumscribed rectangle 72 has four vertices (721-724). Vertex 721 and vertex 722 can be used as fixed anchor points, and then vertex 723 and vertex 724 can be controlled to move (for example, move in the direction indicated by arrow 73), so that the target object layer is deformed.

[0132] In other implementations, the target object layer may be deformed in other ways, which are not limited here.

[0133] The following is an introduction to how to add virtual facial elements.

[0134] Specifically, the size of the virtual facial element may be determined based on the size of the circumscribed rectangle, and an adapted position may be allocated to each virtual facial element.

[0135] For the virtual eye, the size of the virtual eye may be adjusted to half the size of the circumscribed rectangle, and the position of the virtual eye may be determined based on the position information of the centroid of the circumscribed rectangle, and the virtual eye may be added to the position of the virtual eye.

[0136] Among them, the centroid of the circumscribed rectangle can be set in advance. In this embodiment, the centroid is the center of the circumscribed rectangle. In other embodiments, it can also be other positions, such as the top of the circumscribed rectangle, etc., which is not limited here.

[0137] Specifically, the center position of the virtual eye can be adjusted to the centroid position, so that the virtual eye is added to the corresponding position of the target object layer. Here, the center position of the virtual eye refers to the center position between the two eyes.

[0138] For the virtual mouth, the position of the virtual mouth can be determined based on the position of the virtual eyes and the pupil distance between the virtual eyes, and the virtual mouth is added to the position of the virtual mouth to obtain a target object layer.

[0139] In other implementations, the virtual mouth may also be added to other positions of the target object layer, which is not limited thereto.

[0140] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0141] Based on the same inventive concept, a video generating device corresponding to the video generating method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned video generating method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0142] Reference Figure 8 FIG. 8 is a schematic diagram of a video generating device provided in some embodiments of the present disclosure, wherein the video generating device 800 includes:

[0143] An image acquisition module 810 is used to acquire and display a target image, wherein the target image includes at least one object;

[0144] an object determination module 820, configured to determine, in response to a trigger operation on the target image, a trigger position corresponding to the trigger operation, and determine a target object corresponding to the trigger position from the target image;

[0145] Animation generation module 830, used to separate the target object from the target image to obtain a target object layer, and generate an animation effect associated with the target object based on the target object layer;

[0146] The video generation module 840 is used to input the target object layer into a preset model to obtain description information matching the target object, and generate an encyclopedia video associated with the target object based on the animation effects associated with the target object and the description information.

[0147] In a possible implementation, the animation generation module 830 is specifically used to:

[0148] The target object is separated from the target image to obtain a background layer and the target object layer, and an animation effect associated with the target object is generated based on the background layer, the target object layer and preset virtual materials.

[0149] In a possible implementation, the preset virtual material includes a virtual facial material, and the virtual facial material includes at least one of a virtual eye and a virtual mouth.

[0150] In a possible implementation, the animation generation module 830 is specifically used to:

[0151] Inputting the background layer into a preset image restoration model to obtain a restored background layer;

[0152] Based on the restored background layer, the target object layer and the preset virtual material, an animation effect associated with the target object is generated.

[0153] In a possible implementation, the video generation module 840 is specifically used for:

[0154] Inputting the target object layer into a preset text information generation model to obtain text information matching the target object layer;

[0155] The text information is input into a preset description information generation model to obtain description information matching the target object.

[0156] In a possible implementation, the video generation module 840 is specifically used for:

[0157] Based on speech synthesis technology, the description information is converted into audio information, and based on the animation special effects and the audio information, an encyclopedia video associated with the target object is generated.

[0158] In a possible implementation manner, the object determination module 820 is specifically configured to:

[0159] Performing image segmentation processing on the target image to obtain multiple semantic regions; each semantic region corresponds to a target object;

[0160] Pixel information of the trigger position is determined, and based on the pixel information of the trigger position and the multiple semantic areas, a target object corresponding to the trigger position is determined.

[0161] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0162] Based on the same technical concept, the embodiment of the present disclosure also provides an electronic device. Fig. 9 , which is a schematic diagram of the structure of an electronic device 900 provided in an embodiment of the present disclosure, including a processor 901, a memory 902, and a bus 903. The memory 902 is used to store execution instructions, including a memory 9021 and an external memory 9022; the memory 9021 is also called an internal memory, which is used to temporarily store the operation data in the processor 901 and the data exchanged with the external memory 9022 such as a hard disk. The processor 901 exchanges data with the external memory 9022 through the memory 9021.

[0163] In the embodiment of the present application, the memory 902 is specifically used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 901. That is, when the electronic device 900 is running, the processor 901 communicates with the memory 902 through the bus 903, so that the processor 901 executes the application code stored in the memory 902, and then executes the method described in any of the above embodiments.

[0164] Among them, the memory 902 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0165] Processor 901 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0166] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 900. In other embodiments of the present application, the electronic device 900 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0167] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the video generation step in the above method embodiment is performed. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0168] The present disclosure also provides a computer program product that carries a program code. The program code includes instructions that can be used to execute the video generation steps in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0169] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0170] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0171] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0173] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0174] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A video generation method, characterized in that: include: Acquire and display a target image, wherein the target image includes at least one object; In response to a trigger operation on the target image, determining a trigger position corresponding to the trigger operation, and determining a target object corresponding to the trigger position from the target image; Separating the target object from the target image to obtain a target object layer, and generating an animation effect associated with the target object based on the target object layer; The target object layer is input into a preset model to obtain description information matching the target object, and based on the animation effects associated with the target object and the description information, an encyclopedia video associated with the target object is generated.

2. The method according to claim 1, characterized in that The step of separating the target object from the target image to obtain a target object layer, and generating an animation effect associated with the target object based on the target object layer includes: The target object is separated from the target image to obtain a background layer and the target object layer, and an animation effect associated with the target object is generated based on the background layer, the target object layer and preset virtual materials.

3. The method according to claim 2, characterized in that The preset virtual material includes a virtual facial material, and the virtual facial material includes at least one of a virtual eye and a virtual mouth.

4. The method according to claim 2 or 3, characterized in that: The step of generating an animation effect associated with the target object based on the background layer, the target object layer and preset virtual materials includes: Inputting the background layer into a preset image restoration model to obtain a restored background layer; Based on the restored background layer, the target object layer and the preset virtual material, an animation effect associated with the target object is generated.

5. The method according to claim 1, characterized in that The step of inputting the target object layer into a preset model to obtain description information matching the target object includes: Inputting the target object layer into a preset text information generation model to obtain text information matching the target object layer; The text information is input into a preset description information generation model to obtain description information matching the target object.

6. The method according to any one of claims 1 to 5, characterized in that: The step of generating an encyclopedia video associated with the target object based on the animation special effects associated with the target object and the description information includes: Based on speech synthesis technology, the description information is converted into audio information, and based on the animation special effects and the audio information, an encyclopedia video associated with the target object is generated.

7. The method according to claim 1, characterized in that The step of determining the target object corresponding to the trigger position from the target image includes: Performing image segmentation processing on the target image to obtain multiple semantic regions; each semantic region corresponds to a target object; Pixel information of the trigger position is determined, and based on the pixel information of the trigger position and the multiple semantic areas, a target object corresponding to the trigger position is determined.

8. A video generating device, characterized in that: include: An image acquisition module, used to acquire and display a target image, wherein the target image includes at least one object; an object determination module, configured to determine, in response to a trigger operation on the target image, a trigger position corresponding to the trigger operation, and determine a target object corresponding to the trigger position from the target image; An animation generation module, used for separating the target object from the target image to obtain a target object layer, and generating an animation effect associated with the target object based on the target object layer; The video generation module is used to input the target object layer into a preset model to obtain description information matching the target object, and generate an encyclopedia video associated with the target object based on the animation effects associated with the target object and the description information.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable requests executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable request is executed by the processor, the steps of the video generation method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video generation method according to any one of claims 1 to 7 are executed.

Images