Video generation method and electronic device

By collecting the second object video image that matches the first object action in the first video, and fusing the image except the first object area in the original video with the second object video image, video object replacement learning in the prior art is solved The problems of high cost and low processing efficiency achieve fast and low-cost video object replacement.

CN114449323BActive Publication Date: 2024-11-26VIVO MOBILE COMM CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210109411.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-28
Publication Date
2024-11-26
Estimated Expiration
2042-01-28

AI Technical Summary

Technical Problem

In the prior art, video object replacement requires professional software, which has high learning cost and low processing efficiency.

Method used

A new video is generated by collecting a video image of a second object matching the first object action in the first video, and fusing the image except the first object area in the original video with the second object video image.

Benefits of technology

It realizes the rapid completion of video object replacement without the need for user learning professional tools, reduces the learning cost of video processing and improves processing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114449323B_ABST
    Figure CN114449323B_ABST
Patent Text Reader

Abstract

The present application discloses a video generation method and an electronic device. It belongs to the field of communication technology. An embodiment of the method includes: based on the action of the first object in the first video, collecting at least one frame of object video image of the second object, wherein the action of the second object matches the action of the first object; performing image fusion on the image of the first area in the original video image of the first video and the image of the second area in at least one frame of the object video image to generate a second video; wherein the first area is the image area in the original video image except the area where the first object is located, and the second area is the area where the second object is located in the object video image. This implementation reduces the learning cost of video processing and improves the efficiency of video processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video technology, and more particularly to a video generation method and an electronic device. Background Art

[0002] Currently, users can shoot various videos through electronic devices. In some cases, users have a need to replace objects (such as faces, clothes, etc.) in the shot videos.

[0003] In the prior art, it is usually necessary to use professional software to replace objects in a video, which not only has a high learning cost but also has low video processing efficiency. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a video generation method, device and electronic device, which reduce the learning cost of video processing and improve the efficiency of video processing.

[0005] In a first aspect, an embodiment of the present application provides a video generation method, the method comprising: based on the action of a first object in a first video, acquiring at least one frame of object video image of a second object, wherein the action of the second object matches the action of the first object; performing image fusion on an image of a first area in the original video image of the first video and an image of a second area in the at least one frame of object video image to generate a second video; wherein the first area is an image area in the original video image excluding an area where the first object is located, and the second area is an area in the object video image where the second object is located.

[0006] In a second aspect, an embodiment of the present application provides a video generating device, comprising: an acquisition unit, for acquiring at least one frame of object video image of a second object based on the action of a first object in a first video, wherein the action of the second object matches the action of the first object; a fusion unit, for performing image fusion on an image of a first area in the original video image of the first video with an image of a second area in the at least one frame of object video image to generate a second video; wherein the first area is an image area in the original video image excluding an area where the first object is located, and the second area is an area in the object video image where the second object is located.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0011] In an embodiment of the present application, at least one frame of object video image of a second object is captured based on the action of a first object in a first video, wherein the action of the second object matches the action of the first object; then, an image of a first area in the original video image of the first video is fused with an image of a second area in at least one frame of the object video image, thereby obtaining a second video in which the first object is replaced by the second object, thereby achieving rapid object replacement. This process does not require users to learn professional video tools, thereby reducing the learning cost of video processing and improving video processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flow chart of a video generation method provided in an embodiment of the present application;

[0013] Figure 2 is a schematic diagram of a process of receiving a first input in a video generating method provided in an embodiment of the present application;

[0014] Figure 3 is a schematic diagram of each frame in the first video of the video generation method provided in an embodiment of the present application;

[0015] Figure 4 is a schematic diagram of a process of receiving a second input in a video generating method provided in an embodiment of the present application;

[0016] Figure 5 This is one of the interface schematic diagrams of the object video image acquisition process of the video generation method provided in the embodiment of the present application;

[0017] Figure 6 This is the second interface schematic diagram of the object video image acquisition process of the video generation method provided in the embodiment of the present application;

[0018] Figure 7 is an effect diagram of the second video of the video generation method provided in an embodiment of the present application;

[0019] Figure 8 is a schematic diagram of a video fusion process of a video generation method provided in an embodiment of the present application;

[0020] Fig. 9 is a schematic diagram of a process of receiving a third input in the video generating method provided in an embodiment of the present application;

[0021] Fig.10 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0022] Fig.11 It is a schematic diagram of the hardware structure of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0024] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0025] The video generation method, device and electronic device provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0026] Please refer to Figure 1 , which shows one of the flow charts of the video generation method provided in the embodiment of the present application. The video generation method provided in the embodiment of the present application can be applied to electronic devices. In practice, the above electronic devices can be electronic devices such as smart phones, tablet computers, laptop portable computers, etc.

[0027] The process of the video generation method provided in the embodiment of the present application includes the following steps:

[0028] Step 101: based on the action of a first object in a first video, capture at least one frame of object video image of a second object.

[0029] In this embodiment, the first video may be a video in any format stored locally, such as avi, mp4, mkv, flv, rm, rmvb and other video formats. The first object may be any object in the picture of the first video, such as a face, a body, an animal, etc., which is not specifically limited here. The second object may be any object different from the first object. The second object may be a real object or a 3D object, which is not specifically limited here. The object video image may be a video image captured with the second object as the shooting image, and each video image is a frame. The action of the second object in the captured object video image matches the action of the first object in the first video. As an example, the recording scene of the first video may be a scene in which a teacher welcomes new students to school, see Figure 2 The preview interface of the first video with label 201. The first video may include several shooting subjects such as a teacher, a boy, and a girl. The first subject may be any of the above shooting subjects, such as a boy. In the first video, the boy may bend down and bow. The second subject may be a shooting subject in the current shooting scene. The current shooting scene may be any scene, and the shooting subject may be any person in the current shooting scene. The second subject may imitate the boy's bending down and bowing action, and in this process, at least one frame of object video image of the second subject may be collected.

[0030] Optionally, the above-mentioned execution subject can record a video of the second object based on the action of the first object in the first video. In the process of recording the video, the action of the second object in the recorded i-th video frame can be obtained. When the matching degree between the action of the second object in the i-th video frame and the action of the first object in the i-th original video image in the first video is greater than a preset threshold, the recorded i-th video frame can be determined as a frame of object video image. Among them, the action can be determined by key point detection and can be characterized by the coordinates of the key points. The matching degree can be determined by a similarity calculation method such as Euclidean distance. Continuing with the above example, the second object can imitate the bending action of the boy in the first video. In the process of recording the video of the second object, for a certain frame, the action of the second object in the frame can be obtained. When the matching degree between the action and the action of the boy in a certain original video image in the first video is greater than 80%, the recorded video frame can be determined as a frame of object video image.

[0031] Optionally, before capturing at least one frame of object video image of the second object, the execution subject may also receive a first input from the user to the first video. The first input may be used to select the first object in the first video. Exemplarily, the first input may include: a touch input from the user to the area where the first object is located, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, which may be determined according to actual usage requirements and are not limited in the embodiments of the present application. The specific gesture in the embodiments of the present application may be any one of a single-click gesture, a sliding gesture, a dragging gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture. The click input in the embodiments of the present application may be a single-click input, a double-click input, or any number of click inputs, etc., and may also be a long press input or a short press input. For example, the first input may be: a circle selection input or a box selection input from the user to the first object in the first video, etc. Figure 2 For example, the recording scene of the first video may be a scene of a teacher welcoming new students. Referring to the preview interface of the first video shown by reference numeral 201, the first video may include several shooting subjects such as a teacher, a boy, and a girl. If the user wants to take the boy as the first subject, the outline of the boy can be circled in any video frame of the first video, thereby realizing the first input.

[0032] In response to the first input, the execution entity may identify the first object in the first video and display the first outline of the first object. In practice, the execution entity may use a pre-trained object recognition model to identify the first object in each video frame in the first video to determine the position of the first object in each video frame. The object recognition model may be pre-trained using a machine learning method (such as a supervised learning method). Specifically, it may be obtained by training a network structure such as a convolutional neural network. As an example, if the first object is a human face, the object recognition model may be a face recognition model. If the first object is a human body, the object recognition model may be a human body recognition model. See more Figure 2 , the first video may be a recorded scene of the first video or a scene of a teacher welcoming new students. Referring to the preview interface of the first video shown by reference numeral 202, when the user circles the area where the boy is located in any video frame of the first video, the above-mentioned execution subject may take the boy as the first object, perform feature recognition on the boy in the circled area, and perform feature recognition on each object in each video frame of the first video, so as to determine whether each video frame contains the boy based on feature comparison. If contained, the outline of the boy may be displayed, and the outline is the first outline. As an example, see Figure 3, the first video may include multiple video frames, wherein the first video frame 301 and the second video frame 302 may include a teacher, a black-haired boy, and a black-haired girl. The third video frame 303 and the fourth video frame 304 may include a black-haired boy and a black-haired girl. The fifth video frame 305 may include a black-haired boy, a black-haired girl, a white-haired boy, and a white-haired girl. The sixth video frame 306 may include a white-haired boy and a white-haired girl. If the black-haired boy is the first object, since the first five frames all include the black-haired boy, their first contour lines should all be displayed.

[0033] After displaying the first contour of the first object, the execution subject may receive a second input from the user on the first contour. The second input may be used to adjust the first contour. For example, the second input may include a sliding gesture or a dragging gesture from the user on the first contour of the first object in any video frame of the first video. Figure 4 As shown in the schematic diagram of the process of receiving the second input, in this example, the boy in the first video is the first object, and the first contour line 401 is the automatically identified contour line of the boy. The user can adjust the first contour line 401 through gestures (such as dragging, sliding, clicking, etc.) to make the contour line more accurate, and the adjusted contour line can be used as the second contour line. In this way, the first contour line of the first object can be manually corrected to improve the accuracy of the first contour line, thereby making the recognition of the action features of the first object more accurate and preventing the second object from imitating the wrong action.

[0034] Optionally, at least one frame of object video image of the second object may be acquired through the following steps: First, the first video may be divided into at least two original video segments based on the target information. Then, at least one frame of object video image of the second object may be acquired based on the action of the first object in each original video segment. The target information includes but is not limited to at least one of the following: the field of view of the lens, the shooting scene, the action of the first object, and the position of the first object. As an example, the recording scene of the first video may be a scene of a teacher welcoming new students to school, and the shooting objects may include a teacher, a black-haired boy, a black-haired girl, a white-haired boy, and a white-haired girl. The first object may be a black-haired boy. See Figure 3, the first video frame 301 and the second video frame 302 in the first video include a teacher, a black-haired boy and a black-haired girl, and the position of the first object (i.e., the black-haired boy) is the same. The third video frame 303 and the fourth video frame 304 include a black-haired boy and a black-haired girl, and the position of the first object (i.e., the black-haired boy) is the same. The fifth video frame 305 includes a black-haired boy, a black-haired girl, a white-haired boy and a white-haired girl. The sixth video frame 306 includes a white-haired boy and a white-haired girl. Thus, based on the position of the first object, the first video can be segmented into four original video clips. The first original video clip includes the first video frame 301 and the second video frame 302. The second original video clip includes the third video frame 303 and the fourth video frame 304. The third original video clip includes the fifth video frame 305. The fourth original video clip includes the sixth video frame 306. Then, the object video image of the second object can be collected for each original video clip respectively. Dividing the first video into at least two original video segments and collecting the object video image of the second object for each original video segment can reduce the difficulty of the second object imitating the actions of the first object, thereby reducing the difficulty of video recording and improving the efficiency of collecting object video images, which in turn helps to improve the efficiency of generating the second video.

[0035] Optionally, after the first video is divided into at least two original video segments, the execution subject can start recording the video of the second object. In the process of recording the video, the second action of the second object in the recorded i-th video frame can be obtained. In the case where the matching degree of the second action of the second object in the i-th video frame and the first action of the first object in the i-th frame original video image in the j-th original video segment is greater than a preset threshold, the recorded i-th video frame is determined as a frame of object video image corresponding to the j-th original video segment. Wherein, i and j are both positive integers. The i-th video frame can be any recorded video frame, and the j-th original video segment can be any original video segment in the original video segments obtained after the first video is divided. As an example, in a certain original video segment of the first video shot in the scene where the teacher welcomes the new students to school, the boy bows at the waist. Then the second object can imitate the bowing action of the boy in the original video segment. In the process of recording the video of the second object, for a certain frame, the action of the second object in the frame can be obtained. When the matching degree between the action and the action of the boy in a certain original video image in the first video is greater than 80%, the recorded video frame can be determined as a frame of the object video image. After the corresponding object video image is captured for each video frame in the original video segment, a video segment corresponding to the original video segment can be obtained, in which the second object performs a bowing action. Therefore, it is not necessary to record the second object in the order of the original video segments, which improves the flexibility of video recording.

[0036] Optionally, during the process of recording a video, the above-mentioned execution entity may also display the video playback interface and video recording preview interface of the jth original video clip. The video playback interface may include a first object, and the video recording preview interface may include a second object. Among them, the first object in the first video may be displayed with original transparency, and other areas may be displayed with reduced transparency to fade other objects. When the degree of match between the second action of the second object in the i-th video frame and the first action of the first object in the i-th original video image in the j-th original video clip is less than or equal to a preset threshold, a prompt message may be displayed, and the prompt message may be used to prompt the user to re-record the i-th video frame. As an example, Figure 5 This is one of the interface diagrams of the object video image acquisition process. Figure 5 As shown, the posture of the second object in the video recording preview interface 502 does not match the posture of the first object in the first video played in the video playback interface 501. At this time, a prompt message 503 of "Material posture does not match" may be prompted to prompt the user to adjust the posture. As an example, Figure 6 This is the second interface diagram of the object video image acquisition process. Figure 6As shown, the matching degree of the posture of the second object in the video recording preview interface 602 and the first object in the first video played in the video playback interface 601 is 98%, and a prompt message 603 of "Material posture matches 98%" may be displayed. At this time, if the matching degree is greater than the preset threshold, the frame of video can be saved. By identifying the difference between the action of the first object in the video playback interface and the action of the second object in the video recording preview interface and prompting the user, it is helpful for the user to quickly understand the matching results, so that the user can quickly guide the second object to adjust the action, which can not only improve the shooting efficiency, but also improve the matching degree of the action, thereby improving the effect of image fusion.

[0037] Step 102: perform image fusion on an image of a first area in an original video image of a first video and an image of a second area in at least one frame of an object video image to generate a second video.

[0038] In this embodiment, the execution subject may fuse the image of the first area in the original video image of the first video with the image of the second area in at least one frame of the object video image to generate a second video. The first area is the image area in the original video image except the area where the first object is located, and the second area is the area where the second object is located in the object video image. As an example, the recording scene of the first video may be a scene where a teacher welcomes new students to school, such as Figure 2 As shown, the first video may include several shooting subjects such as a teacher, a boy, and a girl. The first subject may be any of the above shooting subjects, such as a boy. In the first video, the boy may bend and bow. When shooting the second subject, the second subject may imitate the boy's bending and bowing action, such as Figure 6 As shown. The object video images obtained by shooting can correspond to the original video images in the first video one by one. When performing image fusion, the image area outside the area where the boy is located in each original video image in the first video can be fused with the image of the area where the second object is located in the corresponding object video image to generate the second video. Figure 7 As shown, the second video may include the second object 701 and other objects such as the teacher and the girl in the first video. In the second video, the second object performs a bowing action.

[0039] Optionally, firstly, the image of the first area in each frame of the original video image in the first video can be fused with the object video image corresponding to the frame of the original video image to obtain at least two intermediate video segments; then, the at least two intermediate video segments can be spliced ​​to generate the second video. In this way, the image area can be replaced accurately, replacement errors can be avoided, and the accuracy of video fusion is improved. As an example, see Figure 8As shown in the schematic diagram of the video fusion process, the execution subject can crop each frame of the original video image in the first video, and retain the image area except the area where the first object is located in the original video image. In addition, each frame of the object video image can be cropped to retain the area where the second object is located in the object video image. Afterwards, the frames are aligned one by one to ensure that the order of the object video image is consistent with the order of the original video image. Finally, the corresponding groups of video frames are synthesized to obtain the second video.

[0040] See also Figure 3 In the example in, the first video may include multiple video frames. The first video frame 301 and the second video frame 302 may include a teacher, a black-haired boy, and a black-haired girl. The third video frame 303 and the fourth video frame 304 may include a black-haired boy and a black-haired girl. The fifth video frame 305 may include a black-haired boy, a black-haired girl, a white-haired boy, and a white-haired girl. The sixth video frame 306 may include a white-haired boy and a white-haired girl.

[0041] Correspondingly, after the video is recorded, a first object video image corresponding to the first video frame 301, a second object video image corresponding to the second video frame 302, a third object video image corresponding to the third video frame 303, a fourth object video image corresponding to the fourth video frame 304, and a fifth object video image corresponding to the fifth video frame 305 can be obtained. At this time, the area image outside the area where the black-haired boy is located in the first video frame 301 can be fused with the area image where the second object is located in the first object video image, the area image outside the area where the black-haired boy is located in the second video frame 302 can be fused with the area image where the second object is located in the second object video image, the area image outside the area where the black-haired boy is located in the third video frame 303 can be fused with the area image where the second object is located in the third object video image, the area image outside the area where the black-haired boy is located in the fourth video frame 304 can be fused with the area image where the second object is located in the fourth object video image, and the area image outside the area where the black-haired boy is located in the fifth video frame 305 can be fused with the area image where the second object is located in the fifth object video image, to obtain five intermediate video images. Since the sixth video frame 306 does not include the black-haired boy as the first object, it can be directly used as the sixth intermediate video image. Then, the six intermediate video images can be spliced ​​to obtain the second video. In the first two frames of the second video, the teacher, the second object and the black-haired girl can be included. In the third and fourth frames of the second video, the second object and the black-haired girl can be included. In the fifth frame of the second video, the teacher, the second object, the black-haired girl, the white-haired girl and the white-haired boy can be included. In the sixth frame of the second video, the white-haired girl and the white-haired boy can be included. Among them, the action of the second object in the second video is basically the same as the action of the black-haired boy in the first video, both of which are bending and bowing.

[0042] Optionally, if the first video is divided into at least two original video segments, when performing image fusion of the image of the first area in each frame of the original video image in each original video segment with the object video image corresponding to each frame of the original video image, the image of the first area in each frame of the original video image in each original video segment can be firstly extracted to obtain at least one background image sequence. Then, the image of the second area in the object video image corresponding to each original video segment is extracted to obtain at least one object image sequence. Finally, the background image in at least one background image sequence and the object video image in at least one object image sequence are image fused frame by frame to obtain at least two intermediate video segments. Performing image fusion in the form of an image sequence can further ensure that the corresponding image sequence is consistent, thereby improving the accuracy of video fusion.

[0043] Optionally, the execution entity can also display a three-dimensional model of the object, in which case the second object is the three-dimensional model of the object. Then, a third input from the user to the three-dimensional model of the object can be received. The third input can be used to adjust the action, posture, angle, etc. of the three-dimensional model of the object. Exemplarily, the third input specifically includes: a touch input from the user to at least one area of ​​the three-dimensional model of the object. The area may include but is not limited to at least one of the following: head, torso, upper limbs, lower limbs. The touch input may include but is not limited to at least one of the following: sliding input, dragging input, clicking input. The user can make the third input based on the action of the first object in the first video. In response to the third input, the execution entity can update the action of the three-dimensional model of the object and capture at least one frame of the object video image of the three-dimensional model of the object. As an example, Fig. 9 As shown in the third interface diagram of the object video image acquisition process, when the above-mentioned execution subject acquires the object video image, the object three-dimensional model can be displayed in the video recording preview interface 902, and the first object in the first video can be displayed in the video playback interface 901. While playing the first video, the user can adjust the direction, posture, action, etc. of the object three-dimensional model through sliding input, dragging input, etc., so as to adjust it to an action that matches the first object. The second video obtained based on this may include a three-dimensional model and other photographed objects in the first video except the first object. Among them, the action of the three-dimensional model in the second video is basically the same as the action of the first object in the first video, such as bending and bowing. Since the second object can be a three-dimensional model of an object in addition to a real object, the applicable scenarios of video fusion are expanded to meet the usage needs of more users.

[0044] Furthermore, when performing video fusion, the image of the first area in the original video image of the first video may first be subjected to image preprocessing. The image preprocessing includes at least one of the following: cartooning and blurring. Then, the image of the first area in the original video image of the first video after image preprocessing is fused with the image of the second area in at least one frame of the object video image to generate a second video. Performing image preprocessing such as cartooning and blurring on the image of the first area in the original video image of the first video can provide richer video styles and meet the usage needs of more users.

[0045] The video generation method provided in the embodiment of the present application can be executed by a video generation device. In the embodiment of the present application, the video generation device provided in the embodiment of the present application is described by taking the video generation method executed by the video generation device as an example.

[0046] The video generating device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0047] The video generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0048] The video generation device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0049] Alternatively, if Fig.10 As shown, an embodiment of the present application also provides an electronic device 1000, including a processor 1001 and a memory 1002, wherein the memory 1002 stores a program or instruction that can be executed on the processor 1001, and when the program or instruction is executed by the processor 1001, each step of the above-mentioned video generation method embodiment is implemented, and the same technical effect can be achieved.

[0050] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices described above.

[0051] In this embodiment, when the program or instruction is executed by the processor 1001, at least one frame of object video image of the second object can be captured based on the action of the first object in the first video, wherein the action of the second object matches the action of the first object; the image of the first area in the original video image of the first video is fused with the image of the second area in the at least one frame of object video image to generate a second video; wherein the first area is the image area in the original video image excluding the area where the first object is located, and the second area is the area where the second object is located in the object video image.

[0052] Optionally, when the program or instruction is executed by the processor 1001, the first video can also be divided into at least two original video segments based on the target information; at least one frame of object video image of the second object is captured based on the action of the first object in each original video segment; wherein the target information includes at least one of the following: the lens field of view, the shooting scene, the action of the first object, and the position of the first object.

[0053] Optionally, when the program or instruction is executed by the processor 1001, it can also start recording the video of the second object; during the video recording process, when the degree of match between the second action of the second object in the i-th video frame and the first action of the first object in the i-th frame original video image in the j-th original video clip is greater than a preset threshold, the recorded i-th video frame is determined as a frame of object video image corresponding to the j-th original video clip; wherein i and j are both positive integers.

[0054] Optionally, when the program or instruction is executed by the processor 1001, a video playback interface and a video recording preview interface of the j-th original video clip may also be displayed, wherein the video playback interface includes a first object, and the video recording preview interface includes a second object; when the degree of match between the second action of the second object in the i-th video frame and the first action of the first object in the i-th original video image in the j-th original video clip is less than or equal to a preset threshold, a prompt message is displayed, wherein the prompt message is used to prompt the user to re-record the i-th video frame.

[0055] Optionally, when the program or instruction is executed by the processor 1001, the image of the first area in each frame of the original video image in each original video segment can be fused with the object video image corresponding to each frame of the original video image to obtain at least two intermediate video segments; and the at least two intermediate video segments can be spliced ​​to generate a second video.

[0056] Optionally, when the program or instruction is executed by the processor 1001, it can also extract the image of the first area in each frame of the original video image of each original video clip to obtain at least one background image sequence; extract the image of the second area in the object video image corresponding to each original video clip to obtain at least one object image sequence; and fuse the background image in the at least one background image sequence and the object video image in the at least one object image sequence frame by frame to obtain at least two intermediate video clips.

[0057] Optionally, when the program or instruction is executed by processor 1001, it can also receive a first input from a user for a first video; in response to the first input, identify a first object in the first video and display a first contour of the first object; receive a second input from the user for the first contour; and in response to the second input, update the first contour to a second contour.

[0058] Optionally, when the program or instruction is executed by the processor 1001, it can also display the object three-dimensional model, where the second object is the object three-dimensional model; receive a third input from the user to the object three-dimensional model; in response to the third input, update the action of the object three-dimensional model, and capture at least one frame of object video image of the object three-dimensional model.

[0059] Optionally, when the program or instruction is executed by the processor 1001, image preprocessing may be performed on the image of the first area in the original video image of the first video; the image of the first area in the original video image of the first video after image preprocessing is fused with the image of the second area in the at least one frame of object video image to generate a second video; wherein the image preprocessing includes at least one of the following: cartoon processing and blur processing.

[0060] Fig.11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0061] The electronic device 1100 includes but is not limited to: a radio frequency unit 1101, a network module 1102, an audio output unit 1103, an input unit 1104, a sensor 1105, a display unit 1106, a user input unit 1107, an interface unit 1108, a memory 1109, and a processor 1110 and other components.

[0062] Those skilled in the art will appreciate that the electronic device 1100 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.11The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0063] Among them, the processor 1110 is used to capture at least one frame of object video image of a second object based on the action of a first object in a first video, wherein the action of the second object matches the action of the first object; perform image fusion on an image of a first area in an original video image of the first video and an image of a second area in the at least one frame of object video image to generate a second video; wherein the first area is an image area in the original video image excluding an area where the first object is located, and the second area is an area in the object video image where the second object is located.

[0064] In an embodiment of the present application, at least one frame of object video image of a second object is captured based on the action of a first object in a first video, wherein the action of the second object matches the action of the first object; then, an image of a first area in the original video image of the first video is fused with an image of a second area in at least one frame of the object video image, thereby obtaining a second video in which the first object is replaced by the second object, thereby achieving rapid object replacement. This process does not require users to learn professional video tools, thereby reducing the learning cost of video processing and improving video processing efficiency.

[0065] Optionally, the processor 1110 is further used to divide the first video into at least two original video segments based on target information; and to capture at least one frame of object video image of the second object based on the action of the first object in each original video segment; wherein the target information includes at least one of the following: the lens field of view, the shooting scene, the action of the first object, and the position of the first object.

[0066] Optionally, the processor 1110 is also used to control the sensor 1105 to start recording video of the second object; during the video recording process, when the degree of match between the second action of the second object in the i-th video frame and the first action of the first object in the i-th frame original video image in the j-th original video clip is greater than a preset threshold, the recorded i-th video frame is determined as a frame of object video image corresponding to the j-th original video clip; wherein i and j are both positive integers.

[0067] Optionally, the processor 1110 is also used to control the display unit 1106 to display the video playback interface and the video recording preview interface of the j-th original video clip, the video playback interface including a first object, and the video recording preview interface including a second object; when the degree of match between the second action of the second object in the i-th video frame and the first action of the first object in the i-th frame original video image in the j-th original video clip is less than or equal to a preset threshold, a prompt message is displayed, and the prompt message is used to prompt the user to re-record the i-th video frame.

[0068] Optionally, the processor 1110 is further used to perform image fusion on the image of the first area in each frame of the original video image in each original video segment with the object video image corresponding to each frame of the original video image to obtain at least two intermediate video segments; and perform video splicing on the at least two intermediate video segments to generate a second video.

[0069] Optionally, the processor 1110 is further used to extract the image of the first area in each frame of the original video image of each original video clip to obtain at least one background image sequence; extract the image of the second area in the object video image corresponding to each original video clip to obtain at least one object image sequence; and perform image fusion on the background image in the at least one background image sequence and the object video image in the at least one object image sequence frame by frame to obtain at least two intermediate video clips.

[0070] Optionally, the processor 1110 is also used to control the user input unit 1107 to receive a first input from a user to a first video; in response to the first input, identify a first object in the first video and display a first contour line of the first object; control the user input unit 1107 to receive a second input from a user to the first contour line; and in response to the second input, update the first contour line to a second contour line.

[0071] Optionally, the processor 1110 is further used to control the display unit 1106 to display the object three-dimensional model, where the second object is the object three-dimensional model; control the user input unit 1107 to receive a third input from the user to the object three-dimensional model; update the action of the object three-dimensional model in response to the third input, and capture at least one frame of object video image of the object three-dimensional model.

[0072] Optionally, the processor 1110 is further used to perform image preprocessing on the image of the first area in the original video image of the first video; perform image fusion on the image of the first area in the original video image of the first video after image preprocessing with the image of the second area in the at least one frame of object video image to generate a second video; wherein the image preprocessing includes at least one of the following: cartoon processing, blur processing.

[0073] It should be understood that in the embodiment of the present application, the input unit 1104 may include a graphics processor (Graphics Processing Unit, GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0074] The memory 1109 can be used to store software programs and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1109 may include a volatile memory or a non-volatile memory, or the memory 1109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0075] The processor 1110 may include one or more processing units; optionally, the processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1110.

[0076] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned video generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0077] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0078] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned video generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0079] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0080] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned video generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0081] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and electronic devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0083] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A video generation method, characterized in that: The method comprises: Based on the target information, the first video is divided into at least two original video segments; the video of the second object is recorded; during the video recording process, when the matching degree between the second action of the second object in the i-th video frame and the first action of the first object in the i-th frame original video image in the j-th original video segment is greater than a preset threshold, the recorded i-th video frame is determined as a frame of object video image corresponding to the j-th original video segment; wherein i and j are both positive integers; wherein the action of the second object matches the action of the first object; Performing image fusion on an image of a first area in an original video image of the first video and an image of a second area in the at least one frame of object video image to generate a second video, wherein the object video image is shot with respect to the original video image in the first video, and the object video image obtained by shooting corresponds to the original video image in the first video; The first area is an image area in the original video image except an area where the first object is located, and the second area is an area in the object video image where the second object is located.

2. The method according to claim 1, characterized in that The target information includes at least one of the following: a lens field of view, a shooting scene, an action of the first object, and a position of the first object.

3. The method according to claim 1, characterized in that During the video recording process, the method further includes: Displaying a video playback interface and a video recording preview interface of the j-th original video clip, wherein the video playback interface includes a first object, and the video recording preview interface includes a second object; When the degree of match between the second action of the second object in the i-th video frame and the first action of the first object in the i-th original video image in the j-th original video clip is less than or equal to a preset threshold, a prompt message is displayed, wherein the prompt message is used to prompt the user to re-record the i-th video frame.

4. The method according to claim 2, characterized in that: The step of fusing an image of a first area in an original video image of the first video with an image of a second area in the at least one frame of object video image to generate a second video includes: Performing image fusion on an image of a first region in each frame of an original video image in each original video segment and an object video image corresponding to each frame of the original video image to obtain at least two intermediate video segments; The at least two intermediate video clips are spliced ​​to generate a second video.

5. The method according to claim 4, characterized in that The step of fusing the image of the first area in each frame of the original video image in each original video segment with the object video image corresponding to each frame of the original video image to obtain at least two intermediate video segments includes: Extracting an image of a first area in each frame of an original video image of each original video segment to obtain at least one background image sequence; Extracting an image of a second region in the object video image corresponding to each original video clip to obtain at least one object image sequence; The background images in the at least one background image sequence and the object video images in the at least one object image sequence are fused frame by frame to obtain at least two intermediate video segments.

6. The method according to claim 1, characterized in that Before acquiring at least one frame of object video image of the second object based on the action of the first object in the first video, the method further includes: Receiving a first input of a first video from a user; In response to the first input, identifying a first object in the first video and displaying a first outline of the first object; receiving a second input of the first contour line by a user; In response to the second input, the first contour is updated to a second contour.

7. The method according to claim 1, characterized in that Based on the action of the first object in the first video, collecting at least one frame of object video image of the second object, including: Displaying a three-dimensional model of an object, wherein the second object is the three-dimensional model of the object; receiving a third input from a user regarding the three-dimensional model of the object; In response to the third input, the object three-dimensional model is updated, and at least one frame of object video image of the object three-dimensional model is captured.

8. The method according to claim 7, characterized in that The step of fusing an image of a first area in an original video image of the first video with an image of a second area in the at least one frame of object video image to generate a second video includes: Performing image preprocessing on an image in a first area of ​​an original video image of the first video; Performing image fusion on an image of a first area in an original video image of the first video after image preprocessing and an image of a second area in the at least one frame of object video image to generate a second video; The image preprocessing includes at least one of the following: cartoon processing and blur processing.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the video generation method according to any one of claims 1 to 8.