Plane tracking replacement method, device, equipment and medium based on color harmony

By calculating the homogenous processing of the color of the substitute, the position and color of the substitute are adjusted frame by frame, and the abrupt problem caused by the fixed color of the substitute in the existing technology is solved, and the realism and effect of plane tracking is improved.

CN114639053BActive Publication Date: 2024-11-12SHENZHEN WONDERSHARE SOFTWARE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210320464.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-29
Publication Date
2024-11-12
Estimated Expiration
2042-03-29

AI Technical Summary

Technical Problem

In the existing plane tracking replacement methods, the color of the substitute is fixed, which makes it appear abrupt, lacking realism in the video, and the effect is not good.

Method used

By obtaining the endpoint coordinates of the marked substitute in the alternative plan, calculating the homography matrix, performing homography conversion and color harmony processing, combining the preset color harmony model, adjusting the position and color of the substitute frame by frame to enhance the reality of the video feel.

Benefits of technology

Reduces the abruptness of the video after the replacement, increases the sense of reality, and improves the effect of plane tracking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114639053B_ABST
    Figure CN114639053B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses a plane tracking replacement method, device, equipment and medium based on color harmony. The present application is applied to the field of plane tracking replacement technology, which includes: obtaining the endpoint coordinates of the replacement object in the replacement plane map; determining the endpoint coordinate set according to the initial endpoint coordinates of the object to be replaced on the first frame of the target video frame and the preset plane tracking algorithm, and performing homography transformation on the replacement object according to the endpoint coordinate set, the initial endpoint coordinates and the endpoint coordinates of the replacement object to obtain an input atlas; cutting the object to be replaced on the target video frame picture according to the endpoint coordinate set and the initial endpoint coordinates to obtain a color reference atlas, and obtaining a color harmony atlas according to the color reference atlas, the input atlas and the preset color harmony model; replacing the object to be replaced according to the color harmony atlas, the endpoint coordinate set and the initial endpoint coordinates. The embodiment of the present application can improve the effect of plane tracking.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of plane tracking replacement, and in particular to a plane tracking replacement method, device, equipment and medium based on color harmony. Background Art

[0002] Most of the existing planar tracking replacement methods are to give the center position or the endpoint positions of the four corners of the tracking target. Each subsequent frame in the video frame can use the tracking replacement algorithm to continuously track the position of the center position or the endpoint positions of the four corners in the video screen, so as to use the replacement object to replace the target object in the screen. However, since the replacement object usually comes from a single picture and has a fixed color, it will appear abrupt in the video after replacement and lacks realism, making the planar tracking replacement effect poor. Summary of the invention

[0003] The embodiments of the present invention provide a color harmony-based plane tracking replacement method, device, equipment and medium, aiming to solve the problem of poor effect of existing plane tracking replacement.

[0004] In a first aspect, an embodiment of the present invention provides a plane tracking replacement method based on color harmony, which includes:

[0005] Get the endpoint coordinates of the marked replacement in the replacement plan;

[0006] Determine an endpoint coordinate set according to the initial endpoint coordinates of the object to be replaced on the first frame of the target video frame and a preset plane tracking algorithm, and calculate a homography matrix set according to the endpoint coordinate set, the initial endpoint coordinates and the endpoint coordinates of the replacement object;

[0007] Performing homography transformation on the replacement objects on the replacement plane graph according to the homography matrix set to obtain an input graph set;

[0008] Cutting the object to be replaced on the target video frame picture according to the endpoint coordinate set and the initial endpoint coordinates to obtain a color reference atlas, and obtaining a color harmony atlas according to the color reference atlas, the input atlas and a preset color harmony model;

[0009] The object to be replaced on the target video frame picture is replaced according to the color harmony atlas, the endpoint coordinate set and the initial endpoint coordinates.

[0010] Furthermore, the initial endpoint coordinates of the object to be replaced on the first frame image corresponding to the target video frame are taken as the current endpoint coordinates; the current endpoint coordinates are input into a preset plane tracking algorithm to obtain the endpoint coordinates of the object to be replaced on the next frame image corresponding to the target video frame; the endpoint coordinates of the object to be replaced are taken as the current endpoint coordinates, and the step of inputting the current endpoint coordinates into a preset plane tracking algorithm to obtain the endpoint coordinates of the object to be replaced on the next frame image corresponding to the target video frame is returned to execute until the endpoint coordinates of the object to be replaced on all frame images corresponding to the target video frame are obtained; the endpoint coordinates of all the objects to be replaced except the initial endpoint coordinates are taken as an endpoint coordinate set.

[0011] Further, a homography matrix is ​​calculated based on the initial endpoint coordinates and the endpoint coordinates of the replacement object; a homography matrix subset is calculated based on the endpoint coordinate set and the endpoint coordinates of the replacement object, wherein the homography matrix subset includes multiple homography matrices corresponding to multiple endpoint coordinates in the endpoint coordinate set; the homography matrix and the homography matrix subset are used as a homography matrix set.

[0012] Furthermore, a homography transformation is performed on the replacement object on the replacement plane image according to the homography matrix to obtain an input image; a homography transformation is performed on the replacement object on the replacement plane image according to a subset of the homography matrix to obtain an input image subset, and the input image and the input image subset are used as an input image set.

[0013] Furthermore, the objects to be replaced on the first frame image corresponding to the target video frame are cut according to the initial endpoint coordinates to obtain a color reference map; the replacements on the remaining frame images excluding the first frame image corresponding to the target video frame are cut according to the endpoint coordinate set to obtain a color reference map subset, and the color reference map subset and the color reference map are used as a color reference map set.

[0014] Furthermore, the color feature set and the substitute feature set in the color reference atlas and the input atlas are respectively extracted through a preset color harmony model; the color feature set and the substitute feature set are color-fused to obtain a color harmony atlas.

[0015] Furthermore, a mask atlas corresponding to the color harmony atlas is obtained according to any endpoint coordinate in the endpoint coordinate set; a replacement area in the color harmony atlas is obtained according to a first area of ​​the mask atlas; a retained area in the target video frame after removing the object to be replaced is obtained according to a second area of ​​the mask atlas; and the replacement area and the retained area are superimposed to replace the object to be replaced on the target video frame image.

[0016] In a second aspect, an embodiment of the present invention further provides a plane tracking replacement device based on color harmony, which includes:

[0017] An acquisition unit, used for acquiring the endpoint coordinates of the marked replacement object in the replacement plan;

[0018] A calculation unit, used to determine an endpoint coordinate set according to the initial endpoint coordinates of the object to be replaced on the first frame of the target video frame and a preset plane tracking algorithm, and calculate a homography matrix set according to the endpoint coordinate set, the initial endpoint coordinates and the endpoint coordinates of the replacement object;

[0019] A conversion unit, configured to perform homography conversion on the replacement objects on the replacement plane graph according to the homography matrix set to obtain an input graph set;

[0020] A cutting unit, used for cutting the object to be replaced on the target video frame picture according to the endpoint coordinate set and the initial endpoint coordinates to obtain a color reference atlas, and obtaining a color harmony atlas according to the color reference atlas, the input atlas and a preset color harmony model;

[0021] A replacement unit is used to replace the object to be replaced on the target video frame picture according to the color harmony atlas, the endpoint coordinate set and the initial endpoint coordinates.

[0022] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0023] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can implement the above method when executed by a processor.

[0024] The embodiment of the present invention provides a method, device, equipment and medium for plane tracking replacement based on color harmony. The method includes: obtaining the endpoint coordinates of the marked replacement object in the replacement plane map; determining the endpoint coordinate set according to the initial endpoint coordinates of the object to be replaced on the first frame of the target video frame and the preset plane tracking algorithm, and calculating the homography matrix set according to the endpoint coordinate set, the initial endpoint coordinates and the endpoint coordinates of the replacement object; performing homography transformation on the replacement object on the replacement plane map according to the homography matrix set to obtain an input atlas; cutting the object to be replaced on the target video frame picture according to the endpoint coordinate set and the initial endpoint coordinates to obtain a color reference atlas, and obtaining a color harmony atlas according to the color reference atlas, the input atlas and the preset color harmony model; replacing the object to be replaced on the target video frame picture according to the color harmony atlas, the endpoint coordinate set and the initial endpoint coordinates. The technical solution of the embodiment of the present invention first performs homography transformation on the replacement objects on the replacement plane according to the homography matrix set to obtain an input atlas, then obtains a color harmony atlas according to the color reference atlas, the input atlas and a preset color harmony model, and finally replaces the objects to be replaced on the target video frame image according to the color harmony atlas, the endpoint coordinate set and the initial endpoint coordinates, thereby reducing the abruptness of the replaced video, increasing the sense of reality, and improving the effect of plane tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0026] Figure 1 A schematic flow chart of a color harmony-based plane tracking replacement method provided by an embodiment of the present invention;

[0027] Figure 2 A schematic block diagram of a color harmony-based plane tracking replacement device provided by an embodiment of the present invention; and

[0028] Figure 3 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0031] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0032] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0033] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0034] See also Figure 1 , Figure 1 It is a flow chart of the plane tracking replacement method based on color harmony provided by an embodiment of the present invention. The plane tracking replacement method based on color harmony of the embodiment of the present invention can be applied to terminals, such as smart phones, laptops, desktop computers, notebook computers and other intelligent terminal devices, and the plane tracking replacement method based on color harmony is implemented by software installed on the terminal, such as an application named video editing software, thereby reducing the abruptness of the replaced video, increasing the sense of reality, and improving the effect of plane tracking. The plane tracking replacement method based on color harmony is described in detail below. Figure 1 As shown, the method includes the following steps S100-S140.

[0035] S100, obtaining the endpoint coordinates of the marked replacement object in the replacement plan.

[0036] In an embodiment of the present invention, before the video editing software replaces the object to be replaced in the target video frame, the relevant personnel are first required to manually mark the endpoint coordinates of the replacement object in the replacement plan, wherein the endpoint coordinates include the upper left endpoint coordinate, the lower left endpoint coordinate, the upper right endpoint coordinate and the lower right endpoint coordinate. It can be understood that the upper left endpoint coordinate, the lower left endpoint coordinate, the upper right endpoint coordinate and the lower right endpoint coordinate are the endpoint coordinates of the four corners of the replacement object; then the endpoint coordinates of the marked replacement object in the replacement plan are obtained for subsequent use.

[0037] S110, determining an endpoint coordinate set according to the initial endpoint coordinates of the object to be replaced on the first frame of the target video frame and a preset plane tracking algorithm, and calculating a homography matrix set according to the endpoint coordinate set, the initial endpoint coordinates and the endpoint coordinates of the replacement object.

[0038] In an embodiment of the present invention, after obtaining the endpoint coordinates of the marked replacement object in the replacement plane diagram, the initial endpoint coordinates of the object to be replaced on the first frame image corresponding to the target video frame are used as the current endpoint coordinates. It can be understood that the target video frame includes multiple frames, wherein the initial endpoint coordinates of the object to be replaced on the first frame image are manually marked by relevant personnel for use by video editing software; the current endpoint coordinates are input into a preset plane tracking algorithm to obtain the endpoint coordinates of the object to be replaced on the next frame image corresponding to the target video frame, wherein the preset plane tracking algorithm is the plane target tracking algorithm proposed in Li Yang's paper; the endpoint coordinates of the object to be replaced are used as the current endpoint coordinates, and the step of inputting the current endpoint coordinates into the preset plane tracking algorithm to obtain the endpoint coordinates of the object to be replaced on the next frame image corresponding to the target video frame is returned to execute until the endpoint coordinates of the object to be replaced on all frame images corresponding to the target video frame are obtained; finally, the endpoint coordinates of all the objects to be replaced except the initial endpoint coordinates are used as the endpoint coordinate set. It is understandable that the initial endpoint coordinates include the coordinates of four points, namely, the initial upper left endpoint coordinates, the initial lower left endpoint coordinates, the initial upper right endpoint coordinates, and the initial lower right endpoint coordinates. The endpoint coordinate set includes multiple upper left endpoint coordinates, multiple lower left endpoint coordinates, multiple upper right endpoint coordinates, and multiple lower right endpoint coordinates. In practical applications, after determining the endpoint coordinate set, the homography matrix will be calculated based on the initial endpoint coordinates and the endpoint coordinates of the substitute. It is understandable that the homography matrix is ​​calculated based on the four initial endpoint coordinates on the first frame of the target video and the four endpoint coordinates of the substitute corresponding thereto; after calculating the homography matrix, the homography matrix subset is calculated based on the endpoint coordinate set and the endpoint coordinates of the substitute, wherein the homography matrix subset includes multiple homography matrices, and the multiple homography matrices correspond to the multiple endpoint coordinates in the endpoint coordinate set. It is understandable that the homography matrix subset is calculated based on the four endpoint coordinates on each frame of the target video frame except the first frame of the target video frame and the four endpoint coordinates of the substitute corresponding thereto; the homography matrix and the homography matrix subset are used as the homography matrix set. It should be noted that the homography matrix is ​​the Homography Matrix. In the embodiment of the present invention, the Homography Matrix can be used to correct the posture of the replacement object on the replacement plane, so that the target video after replacement is more realistic.

[0039] S120, performing homography transformation on the replacement objects on the replacement plane graph according to the homography matrix set to obtain an input graph set.

[0040] In an embodiment of the present invention, after calculating the homography matrix set according to the endpoint coordinate set, the initial endpoint coordinates and the endpoint coordinates of the substitute, the substitute on the substitute plane is firstly subjected to homography transformation according to the homography matrix to obtain an input map; then, the substitute on the substitute plane is subjected to homography transformation according to the homography matrix subset to obtain an input map subset, and the input map and the input map subset are used as an input map set, wherein the input map subset includes a plurality of input maps, and the plurality of input maps correspond to the plurality of homography matrices. Specifically, the coordinates of all points between the four endpoint coordinates of the substitute on the substitute plane are multiplied by the homography matrix and the homography matrix subset respectively to obtain the position of the substitute on the substitute plane in each frame of the target video, thereby obtaining the input map set. It should be noted that, in an embodiment of the present invention, a better color harmony map set can be obtained by inputting the input map set after homography transformation into a preset color harmony model.

[0041] S130, cutting the object to be replaced on the target video frame picture according to the endpoint coordinate set and the initial endpoint coordinates to obtain a color reference atlas, and obtaining a color harmony atlas according to the color reference atlas, the input atlas and a preset color harmony model.

[0042] In an embodiment of the present invention, after homography transformation is performed on the replacement object on the replacement plane to obtain an input atlas, the object to be replaced on the first frame corresponding to the target video frame is cut according to the initial endpoint coordinates to obtain a color reference map; the replacement object on the remaining frame images excluding the first frame corresponding to the target video frame is cut according to the endpoint coordinate set to obtain a color reference map subset, and the color reference map subset and the color reference map are used as a color reference atlas. It can be understood that the color reference map subset includes multiple color reference maps. After the object to be replaced on the target video frame image is cut to obtain a color reference atlas, the color feature set and the replacement feature set in the color reference atlas and the input atlas are respectively extracted by a preset color harmony model, wherein the preset color harmony model is an SSH color harmony model, the color feature set is the background color set in the color reference image, and the replacement feature set is the structural feature set of the replacement in the input atlas; the color feature set and the replacement feature set are color-fused to obtain a color harmony atlas. It should be noted that, in the embodiment of the present invention, the color of the input atlas is changed by the SSH color harmony model to generate the color harmony atlas, so that the color of the object to be replaced on the target video frame image after the replacement object replaces the object to be replaced is more realistic and natural.

[0043] S140: Replace the object to be replaced on the target video frame picture according to the color harmony atlas, the endpoint coordinate set, and the initial endpoint coordinates.

[0044] In an embodiment of the present invention, a mask atlas corresponding to the color harmony atlas is obtained according to the endpoint coordinate set, wherein the mask atlas is a mask atlas including a plurality of mask images; the object to be replaced on the target video frame image is replaced according to the mask atlas and the color harmony atlas. Specifically, a replacement area of ​​the color harmony image in the color harmony atlas is obtained according to a first area corresponding to the mask image in the mask atlas, wherein the first area is a white area; a retained area in the target video frame after removing the object to be replaced is obtained according to a second area corresponding to the mask image in the mask atlas, wherein the second area is a black area; the replacement area and the retained area are superimposed to replace the object to be replaced on the target video frame image.

[0045] Figure 2 is a schematic block diagram of a color harmony-based plane tracking replacement device 200 provided by an embodiment of the present invention. Figure 2 As shown, corresponding to the above color harmony-based plane tracking replacement method, the present invention also provides a color harmony-based plane tracking replacement device 200. The color harmony-based plane tracking replacement device 200 includes a unit for executing the above color harmony-based plane tracking replacement method, and the device can be configured in a terminal. Specifically, please refer to Figure 2 The color harmony-based plane tracking replacement device 200 includes an acquisition unit 201 , a calculation unit 202 , a conversion unit 203 , a cutting unit 204 and a replacement unit 205 .

[0046] Among them, the acquisition unit 201 is used to obtain the endpoint coordinates of the marked replacement object in the replacement plane map; the calculation unit 202 is used to determine the endpoint coordinate set according to the initial endpoint coordinates of the object to be replaced on the first frame of the target video frame and the preset plane tracking algorithm, and calculate the homography matrix set according to the endpoint coordinate set, the initial endpoint coordinates and the endpoint coordinates of the replacement object; the conversion unit 203 is used to perform homography transformation on the replacement object on the replacement plane map according to the homography matrix set to obtain an input atlas; the cutting unit 204 is used to cut the object to be replaced on the target video frame picture according to the endpoint coordinate set and the initial endpoint coordinates to obtain a color reference atlas, and obtain a color harmony atlas according to the color reference atlas, the input atlas and a preset color harmony model; the replacement unit 205 is used to replace the object to be replaced on the target video frame picture according to the color harmony atlas, the endpoint coordinate set and the initial endpoint coordinates.

[0047] In some embodiments, such as the present embodiment, the computing unit 202 includes a first operating unit 2021 , an input unit 2022 , a return execution unit 2023 , a second operating unit 2024 , a first computing subunit 2025 , a second computing subunit 2026 , and a third operating unit 2027 .

[0048] Among them, the first taking unit 2021 is used to use the initial endpoint coordinates of the object to be replaced on the first frame picture corresponding to the target video frame as the current endpoint coordinates; the input unit 2022 is used to input the current endpoint coordinates into a preset plane tracking algorithm to obtain the endpoint coordinates of the object to be replaced on the next frame picture corresponding to the target video frame; the return execution unit 2023 is used to use the endpoint coordinates of the object to be replaced as the current endpoint coordinates, and return to execute the step of inputting the current endpoint coordinates into the preset plane tracking algorithm to obtain the endpoint coordinates of the object to be replaced on the next frame picture corresponding to the target video frame, until all frame pictures corresponding to the target video frame are obtained The second as unit 2024 is used to take the endpoint coordinates of all the objects to be replaced except the initial endpoint coordinates as the endpoint coordinate set; the first calculation subunit 2025 is used to calculate the homography matrix according to the initial endpoint coordinates and the endpoint coordinates of the replacement object; the second calculation subunit 2026 is used to calculate the homography matrix subset according to the endpoint coordinate set and the endpoint coordinates of the replacement object, wherein the homography matrix subset includes multiple homography matrices, and the multiple homography matrices correspond to the multiple endpoint coordinates in the endpoint coordinate set; the third as unit 2027 is used to take the homography matrix and the homography matrix subset as the homography matrix set.

[0049] In some embodiments, such as this embodiment, the conversion unit 203 includes a first conversion subunit 2031 and a second conversion subunit 2032 .

[0050] Among them, the first conversion subunit 2031 is used to perform homography transformation on the replacement object on the replacement plane map according to the homography matrix to obtain an input image; the second conversion subunit 2032 is used to perform homography transformation on the replacement object on the replacement plane map according to the homography matrix subset to obtain an input image subset, and use the input image and the input image subset as the input image set.

[0051] In some embodiments, such as the present embodiment, the cutting unit 204 includes a first cutting subunit 2041 , a second cutting subunit 2042 , an extracting unit 2043 , and a fusion unit 2044 .

[0052] Among them, the first cutting subunit 2041 is used to cut the objects to be replaced on the first frame picture corresponding to the target video frame according to the initial endpoint coordinates to obtain a color reference map; the second cutting subunit 2042 is used to cut the replacements on the remaining frame pictures corresponding to the target video frame except the first frame picture according to the endpoint coordinate set to obtain a color reference map subset, and use the color reference map subset and the color reference map as a color reference atlas; the extraction unit 2043 is used to extract the color feature set and the replacement feature set in the color reference atlas and the input atlas respectively through a preset color harmony model; the fusion unit 2044 is used to perform color fusion on the color feature set and the replacement feature set to obtain a color harmony atlas.

[0053] The specific implementation of the color harmony-based plane tracking replacement device 200 of the embodiment of the present invention corresponds to the above-mentioned color harmony-based plane tracking replacement method, which will not be described in detail herein.

[0054] The above-mentioned plane tracking replacement device based on color harmony can be implemented in the form of a computer program, which can be used in a computer program such as Figure 3 Runs on the computer device shown.

[0055] See also Figure 3 , Figure 3 300 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 300 is a terminal, which can be an electronic device with communication function such as a smart phone, a desktop computer, a laptop, a tablet computer, etc.

[0056] See also Figure 3 The computer device 300 includes a processor 302 , a memory and a network interface 305 connected via a system bus 301 , wherein the memory may include a storage medium 303 and an internal memory 304 .

[0057] The storage medium 303 may store an operating system 3031 and a computer program 3032. When the computer program 3032 is executed, the processor 302 may execute a color harmony-based plane tracking replacement method.

[0058] The processor 302 is used to provide computing and control capabilities to support the operation of the entire computer device 300 .

[0059] The internal memory 304 provides an environment for the operation of the computer program 3032 in the storage medium 303. When the computer program 3032 is executed by the processor 302, the processor 302 can execute a plane tracking replacement method based on color harmony.

[0060] The network interface 305 is used to communicate with other devices over the network. Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 300 to which the solution of the present application is applied. The specific computer device 300 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0061] The processor 302 is used to run a computer program 3032 stored in a memory to implement the following steps: obtaining the endpoint coordinates of the marked replacement object in the replacement plane map; determining an endpoint coordinate set based on the initial endpoint coordinates of the object to be replaced on the first frame of the target video frame and a preset plane tracking algorithm, and calculating a homography matrix set based on the endpoint coordinate set, the initial endpoint coordinates and the endpoint coordinates of the replacement object; performing a homography transformation on the replacement object on the replacement plane map according to the homography matrix set to obtain an input atlas; cutting the object to be replaced on the target video frame picture according to the endpoint coordinate set and the initial endpoint coordinates to obtain a color reference atlas, and obtaining a color harmony atlas based on the color reference atlas, the input atlas and a preset color harmony model; replacing the object to be replaced on the target video frame picture according to the color harmony atlas, the endpoint coordinate set and the initial endpoint coordinates.

[0062] In certain embodiments, such as the present embodiment, the processor 302, when implementing the step of determining the endpoint coordinate set based on the initial endpoint coordinates of the object to be replaced on the first frame of the target video frame and the preset plane tracking algorithm, specifically implements the following steps: using the initial endpoint coordinates of the object to be replaced on the first frame corresponding to the target video frame as the current endpoint coordinates; inputting the current endpoint coordinates into the preset plane tracking algorithm to obtain the endpoint coordinates of the object to be replaced on the next frame corresponding to the target video frame; using the endpoint coordinates of the object to be replaced as the current endpoint coordinates, and returning to execute the step of inputting the current endpoint coordinates into the preset plane tracking algorithm to obtain the endpoint coordinates of the object to be replaced on the next frame corresponding to the target video frame, until the endpoint coordinates of the object to be replaced on all frame pictures corresponding to the target video frame are obtained; using the endpoint coordinates of all the objects to be replaced except the initial endpoint coordinates as the endpoint coordinate set.

[0063] In certain embodiments, such as the present embodiment, the processor 302, when implementing the step of calculating the homography matrix set according to the endpoint coordinate set, the initial endpoint coordinates and the endpoint coordinates of the replacement object, specifically implements the following steps: calculating the homography matrix according to the initial endpoint coordinates and the endpoint coordinates of the replacement object; calculating a homography matrix subset according to the endpoint coordinate set and the endpoint coordinates of the replacement object, wherein the homography matrix subset includes multiple homography matrices corresponding to multiple endpoint coordinates in the endpoint coordinate set; and using the homography matrix and the homography matrix subset as the homography matrix set.

[0064] In certain embodiments, such as the present embodiment, when the processor 302 implements the step of performing a homographic transformation on the replacement objects on the replacement plane image according to the homography matrix set to obtain an input atlas, the processor 302 specifically implements the following steps: performing a homographic transformation on the replacement objects on the replacement plane image according to the homography matrix to obtain an input image; performing a homographic transformation on the replacement objects on the replacement plane image according to a subset of the homography matrix to obtain a subset of the input image, and using the input image and the subset of the input image as the input atlas.

[0065] In certain embodiments, such as the present embodiment, when the processor 302 implements the step of cutting the objects to be replaced on the target video frame picture according to the endpoint coordinate set and the initial endpoint coordinates to obtain a color reference atlas, it specifically implements the following steps: cutting the objects to be replaced on the first frame picture corresponding to the target video frame according to the initial endpoint coordinates to obtain a color reference map; cutting the replacements on the remaining frame pictures corresponding to the target video frame except the first frame picture according to the endpoint coordinate set to obtain a color reference map subset, and using the color reference map subset and the color reference map as the color reference atlas.

[0066] In certain embodiments, such as the present embodiment, when the processor 302 implements the step of obtaining a color harmony atlas based on the color reference atlas, the input atlas and a preset color harmony model, the processor 302 specifically implements the following steps: respectively extracting the color feature set and the substitute feature set in the color reference atlas and the input atlas through a preset color harmony model; and color-fusion the color feature set and the substitute feature set to obtain a color harmony atlas.

[0067] In certain embodiments, such as the present embodiment, when the processor 302 implements the step of replacing the object to be replaced on the target video frame picture according to the color harmony atlas, the endpoint coordinate set and the initial endpoint coordinates, it specifically implements the following steps: obtaining a mask atlas corresponding to the color harmony atlas according to the endpoint coordinate set; replacing the object to be replaced on the target video frame picture according to the mask atlas and the color harmony atlas.

[0068] It should be understood that in the embodiment of the present application, the processor 302 may be a central processing unit (CPU), and the processor 302 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0069] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0070] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above-mentioned color harmony-based plane tracking replacement method.

[0071] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0072] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0073] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0074] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0075] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0076] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0077] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0078] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A color harmony based plane tracking replacement method, characterized in that: include: Get the endpoint coordinates of the marked replacement in the replacement plan; Determine an endpoint coordinate set according to the initial endpoint coordinates of the object to be replaced on the first frame of the target video frame and a preset plane tracking algorithm, and calculate a homography matrix set according to the endpoint coordinate set, the initial endpoint coordinates and the endpoint coordinates of the replacement object, wherein the target video frame includes multiple frames of pictures; Performing homography transformation on the replacement objects on the replacement plane graph according to the homography matrix set to obtain an input graph set; Cutting the object to be replaced on the target video frame according to the endpoint coordinate set and the initial endpoint coordinates to obtain a color reference atlas, and obtaining a color harmony atlas according to the color reference atlas, the input atlas and a preset color harmony model, wherein the preset color harmony model is an SSH color harmony model; Replacing the object to be replaced on the target video frame picture according to the color harmony atlas, the endpoint coordinate set and the initial endpoint coordinates; The step of determining the endpoint coordinate set according to the initial endpoint coordinates of the object to be replaced on the first frame of the target video frame and a preset plane tracking algorithm includes: Taking the initial endpoint coordinates of the object to be replaced on the first frame of the picture corresponding to the target video frame as the current endpoint coordinates; Inputting the current endpoint coordinates into a preset plane tracking algorithm to obtain the endpoint coordinates of the object to be replaced on the next frame of the picture corresponding to the target video frame; The endpoint coordinates of the object to be replaced are used as the current endpoint coordinates, and the step of inputting the current endpoint coordinates into a preset plane tracking algorithm to obtain the endpoint coordinates of the object to be replaced on the next frame image corresponding to the target video frame is returned until the endpoint coordinates of the object to be replaced on all frame images corresponding to the target video frame are obtained; Taking the endpoint coordinates of all the objects to be replaced except the initial endpoint coordinates as an endpoint coordinate set; The step of calculating a homography matrix set according to the endpoint coordinate set, the initial endpoint coordinates, and the endpoint coordinates of the replacement object comprises: Calculate a homography matrix according to the initial endpoint coordinates and the endpoint coordinates of the substitute; Calculating a homography matrix subset according to the endpoint coordinate set and the endpoint coordinates of the substitute, wherein the homography matrix subset includes a plurality of homography matrices, and the plurality of homography matrices correspond to a plurality of endpoint coordinates in the endpoint coordinate set; Taking the homography matrix and the homography matrix subset as a homography matrix set; The step of performing homography transformation on the replacement objects on the replacement plane graph according to the homography matrix set to obtain an input graph set includes: Performing homography transformation on the replacement object on the replacement plane graph according to the homography matrix to obtain an input graph; Performing homography transformation on the substitute on the replacement plane graph according to the homography matrix subset to obtain an input graph subset, and using the input graph and the input graph subset as an input graph set; The replacing the object to be replaced on the target video frame picture according to the color harmony atlas, the endpoint coordinate set and the initial endpoint coordinates includes: Acquire a mask atlas corresponding to the color harmony atlas according to the endpoint coordinate set; The object to be replaced on the target video frame picture is replaced according to the mask atlas and the color harmony atlas.

2. The color harmony-based plane tracking replacement method according to claim 1, characterized in that: The step of cutting the object to be replaced on the target video frame picture according to the endpoint coordinate set and the initial endpoint coordinates to obtain a color reference atlas includes: Cutting the object to be replaced on the first frame of the picture corresponding to the target video frame according to the initial endpoint coordinates to obtain a color reference picture; According to the endpoint coordinate set, the substitutes on the remaining frame pictures corresponding to the target video frame except the first frame picture are cut to obtain a color reference map subset, and the color reference map subset and the color reference map are used as a color reference map set.

3. The color harmony-based plane tracking replacement method according to claim 2, characterized in that: The step of obtaining a color harmony atlas according to the color reference atlas, the input atlas and a preset color harmony model comprises: Extracting the color feature set and the substitute feature set in the color reference atlas and the input atlas respectively through a preset color harmony model; The color feature set and the substitute feature set are color-fused to obtain a color harmony atlas.

4. A planar tracking replacement device based on color harmony, characterized in that: include: An acquisition unit, used for acquiring the endpoint coordinates of the marked replacement object in the replacement plan; a calculation unit, configured to determine an endpoint coordinate set according to initial endpoint coordinates of an object to be replaced on a first frame of a target video frame and a preset plane tracking algorithm, and calculate a homography matrix set according to the endpoint coordinate set, the initial endpoint coordinates, and the endpoint coordinates of the replacement object, wherein the target video frame includes multiple frames of pictures; A conversion unit, configured to perform homography conversion on the replacement objects on the replacement plane graph according to the homography matrix set to obtain an input graph set; a cutting unit, configured to cut the object to be replaced on the target video frame according to the endpoint coordinate set and the initial endpoint coordinates to obtain a color reference atlas, and to obtain a color harmony atlas according to the color reference atlas, the input atlas and a preset color harmony model, wherein the preset color harmony model is an SSH color harmony model; A replacement unit, used for replacing the object to be replaced on the target video frame picture according to the color harmony atlas, the endpoint coordinate set and the initial endpoint coordinates; Wherein, the computing unit comprises: The first is a unit for taking the initial endpoint coordinates of the object to be replaced on the first frame of the picture corresponding to the target video frame as the current endpoint coordinates; An input unit, used for inputting the current endpoint coordinates into a preset plane tracking algorithm to obtain the endpoint coordinates of the object to be replaced on the next frame of the picture corresponding to the target video frame; A return execution unit is used to use the endpoint coordinates of the object to be replaced as the current endpoint coordinates, and return to the step of inputting the current endpoint coordinates into a preset plane tracking algorithm to obtain the endpoint coordinates of the object to be replaced on the next frame picture corresponding to the target video frame, until the endpoint coordinates of the object to be replaced on all frame pictures corresponding to the target video frame are obtained; The second is used as a unit, for taking the endpoint coordinates of all the objects to be replaced except the initial endpoint coordinates as an endpoint coordinate set; A first calculation subunit, used for calculating a homography matrix according to the initial endpoint coordinates and the endpoint coordinates of the replacement object; A second calculation subunit is used to calculate a homography matrix subset according to the endpoint coordinate set and the endpoint coordinates of the replacement object, wherein the homography matrix subset includes a plurality of homography matrices, and the plurality of homography matrices correspond to a plurality of endpoint coordinates in the endpoint coordinate set; A third unit is used to use the homography matrix and the homography matrix subset as a homography matrix set; The conversion unit comprises: A first conversion subunit, configured to perform homography conversion on the replacement object on the replacement plane graph according to the homography matrix to obtain an input graph; A second conversion subunit is used for performing homography conversion on the replacement on the replacement plane graph according to the homography matrix subset to obtain an input graph subset, and using the input graph and the input graph subset as an input graph set; The replacement unit is specifically used to obtain a mask atlas corresponding to the color harmony atlas according to the endpoint coordinate set; and replace the object to be replaced on the target video frame picture according to the mask atlas and the color harmony atlas.

5. A computer device, characterized in that: The computer device includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the color harmony-based plane tracking replacement method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the computer program can implement the color harmony-based plane tracking replacement method as described in any one of claims 1 to 3.