Image processing method, device, electronic device and readable storage medium

Abstract

The present application discloses an image processing method, device, electronic device, and readable storage medium, which belongs to the field of electronic technology. The method includes: determining N key points in a first frame image of a first video, where N is a positive integer, in an area where a first object is located; obtaining first position information of the first key point in the first frame image; and obtaining first feature information of the first key point in the first frame image; the first key point is one of the N key points; determining second position information in a second frame image of the first video, where the first position information matches the second position information; determining third position information in the second frame image based on the second position information, where the second feature information of the second key point corresponding to the third position information matches the first feature information; and determining a movement trajectory of the first key point based on the first position information and the third position information.

Description

Technical Field

[0001] The present application belongs to the field of electronic technology, and specifically relates to an image processing method, device, electronic device and readable storage medium. Background Art

[0002] At present, various short video platforms are emerging one after another, and the number of short video users is increasing. Therefore, the application scenarios of video creation are gradually becoming common.

[0003] Typically, after shooting a video or downloading it online, users need to process it before publishing it as a short video. For example, they might block an object in the video or replace a person in the video. With some post-processing, the video can be published on the platform as a user-created video. However, in existing technologies, if a user wants to block or replace an object, they need to process each frame of the video to overlay the selected element at the corresponding preset position in each frame.

[0004] It can be seen that in the prior art, if a user wants to block or replace an object in a video, the user needs to manually process a large number of frames in the video, which makes the user operation cumbersome. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method that can solve the problem in the prior art that if a user wants to occlude or replace an object in a video, he or she needs to manually process a large number of frames in the video, which leads to cumbersome user operations.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising: determining N key points in an area where a first object is located in a first frame image of a first video, where N is a positive integer; obtaining first position information of the first key point in the first frame image; and obtaining first feature information of the first key point in the first frame image; the first key point is one of the N key points; in a second frame image of the first video, determining second position information, the first position information matches the second position information; based on the second position information, determining third position information in the second frame image, the second feature information of the second key point corresponding to the third position information matches the first feature information; and determining a movement trajectory of the first key point based on the first position information and the third position information.

[0007] In the second aspect, an embodiment of the present application provides an image processing device, which includes: a first determination module for determining N key points in the area where a first object is located in a first frame image of a first video, where N is a positive integer; an acquisition module for acquiring first position information of the first key point in the first frame image; and acquiring first feature information of the first key point in the first frame image; the first key point is one of the N key points; a second determination module for determining second position information in the second frame image of the first video, where the first position information matches the second position information; a third determination module for determining third position information in the second frame image based on the second position information, where the second feature information of the second key point corresponding to the third position information matches the first feature information; and a fourth determination module for determining the movement trajectory of the first key point based on the first position information and the third position information.

[0008] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the method described in the first aspect.

[0012] Thus, in an embodiment of the present application, for the first video, the user can select the first object as the object to be processed in any frame image through a preset input. Thus, based on the user input, the first object is obtained in the first frame image, and N key points of the area where the first object is located are determined in the first frame image to represent the shape, area and other features of the first object. Furthermore, for each key point (taking the first key point as an example), the first position information and first feature information of the first key point in the first frame image are first obtained. Then, in the second frame image of the first video, the second position information matching the first position information is determined, and the third position information is determined with the second position information as the reference position. In the process of determining the third position information, the feature information of the key point on any position information can be matched based on the first feature information until the feature information of the key point on a certain position information matches the first feature information, thereby determining the position information as the third position information, that is, it is considered to be the position information of the first key point in the second frame image. Finally, based on the first position information and the third position information, the movement trajectory of the first key point is determined. It can be seen that in an embodiment of the present application, a method for obtaining the movement trajectory of a first object is provided. When the user replaces or occludes the first object, the background can directly obtain the movement trajectory of the first object and automatically process it in multiple frames of images in the first video without the need for manual operation by the user, thereby simplifying the user operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flowchart of an image processing method according to an embodiment of the present application;

[0014] Figure 2 is a schematic diagram of an interface of an electronic device according to an embodiment of the present application;

[0015] Figure 3 and Figure 4 is a schematic diagram illustrating an image processing method according to an embodiment of the present application;

[0016] Figure 5 is a block diagram of an image processing device according to an embodiment of the present application;

[0017] Figure 6 This is one of the hardware structure diagrams of the electronic device according to the embodiment of the present application;

[0018] Figure 7 This is the second hardware structure diagram of the electronic device according to the embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the accompanying drawings of the embodiments of the present application to clearly describe the technical solutions of the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0020] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0021] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0022] Figure 1 A flowchart of an image processing method according to an embodiment of the present application is shown. The method is applied to an electronic device and includes:

[0023] Step 110: Determine N key points in the area where the first object is located in the first frame image of the first video, where N is a positive integer.

[0024] In this step, the first object may be a first object selected by a user in the first video through a first input.

[0025] The first input includes touch input performed by the user on the screen, including but not limited to clicks, slides, and drags. The first input may also be airborne input, gestures, facial movements, etc. The first input also includes input performed by the user on physical buttons on the device, including but not limited to presses. Furthermore, the first input may include one or more inputs, wherein the multiple inputs may be continuous or intermittent.

[0026] Application scenarios such as the one provided in this embodiment Figure 2 In the user interaction interface shown, the user clicks the "video loading" control to display the album or other custom directory, so that the user can select any video in the album or other custom directory as the first video. Figure 2 The left area of ​​the interface shows the first video, wherein the first frame of the first video is first displayed, so that the user can select the first object in the first frame.

[0027] Optionally, the first frame image is the first frame image of the first video. Optionally, the first frame image may also be another frame image of the first video.

[0028] For reference, the user selects the first object through the “region selection” tool provided in the interactive interface of this embodiment, so that the first object is located within the region defined by the “region selection” tool.

[0029] For reference, the user clicks on the first object, thereby using image recognition technology to identify the first object selected by the user.

[0030] For reference, the user selects several pixel points on the outline of the first object, or within the area enclosed by the outline.

[0031] For example, if the first object is a plane, such as the surface of an object in the environment, the user can select pixels based on the shape of the first object. Specifically, if the first object is a rectangle, four pixels can be selected, one at each corner of the rectangle. Specifically, if the first object is an arbitrary polygon, the number of pixels can be selected based on the polygon's shape, with the selected pixels corresponding to the polygon's vertices.

[0032] For another example, if the first object is a curved surface, such as a curved surface of an object in the environment, the user can select pixels in the area where the first object is located. Specifically, if the first object is a sphere, the user can select multiple pixels within the contour of the sphere.

[0033] Thus, based on the first object selected by the user, the first object is determined in the first frame image, and N key points are further determined in the first frame image, where the N key points are used to describe features such as the shape and area of ​​the first object.

[0034] Optionally, the N key points are evenly distributed in the area where the first object is located.

[0035] For reference, when the first object is of a plane type, since the plane graphic structure is simple, a key point is determined at each corner of the area where the first object is located.

[0036] For reference, in the case where the first object is of a curved surface type, due to the complex structure of the curved surface graphic, a plurality of key points are densely determined in the area where the first object is located.

[0037] The key points may be pixels in an image.

[0038] For reference, in the aforementioned solution, the multiple pixel points selected by the user can be appropriately increased or decreased, or remain unchanged, to serve as key points.

[0039] Optionally, after the key points are preliminarily determined, the currently determined key points are displayed to the user, and the user can manually add key points based on the features of the first object, etc., to finally determine all the key points of the first object.

[0040] For example, the first object is the back of the hand. The user can manually add some key points to the meridians on the back of the hand so that the application can track the movement trajectory of the meridians.

[0041] Step 120: Obtain first position information of the first key point in the first frame image; and obtain first feature information of the first key point in the first frame image; the first key point is one of the N key points.

[0042] In this embodiment, the first key point is one of the N key points. Based on this embodiment, the position information and feature information of each key point can be acquired, thereby tracking the movement trajectory of each key point.

[0043] Optionally, the first position information may be first coordinate information.

[0044] Optionally, in this step, a feature extraction backbone (the basic structure of a neural network, used to extract image features) network extracts a feature vector of dimension n (n = 512), which represents the image feature information. Correspondingly, for the first key point, the extracted feature vector is denoted as f1 = N, which represents the first feature information of the first key point in the first frame of the image.

[0045] The first feature information includes: color features of the three channels of red, green and blue in the pixel, Hog features of key points extracted through hog features, shallow features extracted through deep neural networks such as resnet50, etc.

[0046] Step 130: In the second frame image of the first video, second position information is determined, and the first position information matches the second position information.

[0047] In this step, the first frame image and the second frame image are different frame images. Here, the first frame image and the second frame image are analyzed separately to track: the movement trajectory of the first object from the first frame image to the second frame image.

[0048] Therefore, first, the second position information is determined in the second frame image.

[0049] Optionally, the first position information may be second coordinate information.

[0050] The first position information matches the second position information, and the first coordinate information and the second coordinate information may be the same. It should be noted that the first coordinate information and the second coordinate information are located in the same coordinate system, which is established based on the interface for displaying the first video.

[0051] Step 140: Based on the second position information, determine third position information in the second frame image, and match the second feature information of the second key point corresponding to the third position information with the first feature information.

[0052] In this step, the second position information can be used as the center, and within a specified range, according to a certain regular algorithm, the second feature information that matches the first feature information and the position information of the corresponding key point can be found.

[0053] Optionally, the first feature information matches the second feature information, that is, the similarity between the first feature information and the second feature information is the highest compared to the similarity between the first feature information and feature information of other key points within the specified range.

[0054] Optionally, in the second frame image, for the key point corresponding to the position information, the extracted feature vector is recorded as f2=N, which is used to represent the feature information of the key point in the second frame image.

[0055] Alternatively, use the cosine similarity formula:

[0056]

[0057] The value used to represent the similarity is calculated, and the closer the value is to "1", the higher the similarity between the two.

[0058] Among them, the acquisition of similarity involves a detection model, which is a similarity judgment network trained using deep learning.

[0059] Optionally, the specified range is: r=10 pixels.

[0060] Step 150: Determine the movement trajectory of the first key point based on the first position information and the third position information.

[0061] In this step, the first position information and the third position information are connected to form the movement trajectory of the first key point.

[0062] For reference, in Figure 2 In the interface shown, the movement trajectory of each key point is displayed below the first video.

[0063] Among them, Figure 2In the figure, the movement trajectory of any key point is obtained from each frame image, such as the 1 to 6 frames of images in the figure. Furthermore, the user can adjust the position information corresponding to any key point in each frame in the displayed movement trajectory based on the adjustment range provided in this embodiment.

[0064] Thus, in an embodiment of the present application, for the first video, the user can select the first object as the object to be processed in any frame image through a preset input. Thus, based on the user input, the first object is obtained in the first frame image, and N key points of the area where the first object is located are determined in the first frame image to represent the shape, area and other features of the first object. Furthermore, for each key point (taking the first key point as an example), the first position information and first feature information of the first key point in the first frame image are first obtained. Then, in the second frame image of the first video, the second position information matching the first position information is determined, and the third position information is determined with the second position information as the reference position. In the process of determining the third position information, the feature information of the key point on any position information can be matched based on the first feature information until the feature information of the key point on a certain position information matches the first feature information, thereby determining the position information as the third position information, that is, it is considered to be the position information of the first key point in the second frame image. Finally, based on the first position information and the third position information, the movement trajectory of the first key point is determined. It can be seen that in an embodiment of the present application, a method for obtaining the movement trajectory of a first object is provided. When the user replaces or occludes the first object, the background can directly obtain the movement trajectory of the first object and automatically process it in multiple frames of images in the first video without the need for manual operation by the user, thereby simplifying the user operation.

[0065] In the image processing method of another embodiment of the present application, N≧3; step 120 includes:

[0066] Sub-step A1: construct a triangle by connecting every three adjacent key points among the N key points.

[0067] See also Figure 3 , taking the “back of the hand” in the figure as the first object, track the movement trajectory.

[0068] First, the first object is modeled. The modeling method is: for the area where the first object is located, the area is framed, and the framed area must cover most of the area where the first object is located; then, in the area, some key points are marked (such as Figure 3 The key points are triangulated by connecting adjacent key points into triangles (as shown on the left). Figure 3 shown on the right).

[0069] Sub-step A2: obtaining first position information of the first key point in the first frame image according to the triangle where the first key point is located; and obtaining first feature information of the first key point in the first frame image.

[0070] In this step, all the key points after triangulation are combined into a graph G. Where V is used to represent the key point set, V∈{V i =(x,y,z)∈R 3}.

[0071] Furthermore, the adjacency matrix A used to represent the connection relationship between key points is obtained. ij ,ij∈V}. Among them, when there is a connection relationship between two key points, a ij =1; when there is no connection between two key points, a ij =0.

[0072] Furthermore, the degree of each key point is calculated separately The degree of a key point is the number of key points it connects to.

[0073] Based on the relevant parameters of the above key points, the positional relationship between the key points in the constructed model can be obtained, and then based on the obtained positional relationship, the first position information of the first key point can be obtained.

[0074] Optionally, x t It is used to represent the corresponding feature vector of the first key point in frame t (corresponding to the first frame image), and x t It represents not only the first feature information, but also the first position information obtained based on modeling.

[0075] It should be noted that the first position information and the first feature information of this embodiment are both obtained based on the constructed model, and are therefore related to the parameters involved above. Therefore, based on the position information and the first feature information obtained in this embodiment, the position information and the feature information of the frame can be determined in other frame images in combination with the model constructed in the frame. For more detailed explanations, please refer to the following embodiments.

[0076] In this embodiment, N key points in the area of ​​the first object are triangulated to obtain the first position information of any key point (such as the first key point) in the model. This makes the position information of each key point no longer dependent on the image itself as a reference. Instead, the position information of each key point serves as a reference for each other, thereby making the obtained position information more accurate. At the same time, the feature information of the key point corresponding to the position information is also integrated into the model, which is more conducive to matching the feature information through the model, thereby making the matching results more accurate.

[0077] In the image processing method of another embodiment of the present application, step 140 includes:

[0078] Sub-step B1: determining fourth position information associated with the second position information in the second frame image according to the triangle where the first key point is located.

[0079] Sub-step B2: Based on the fourth position information, determine fifth position information associated with the fourth position information in the second frame image.

[0080] Sub-step B3: When the fourth location information matches the fifth location information, determine that the fourth location information or the fifth location information is the third location information.

[0081] In this embodiment, the fourth location information and the fifth location information are both location information predicted by calculation. In order to accurately obtain the final location information, step B2 can be repeated, that is, taking the predicted location message as input, continuing to predict until the output location information matches the input location information (can be the same or close), and then the final output location information is used as the determined third location information.

[0082] Furthermore, the third position information may be determined when the output position information matches the input position information for multiple consecutive times.

[0083] In this embodiment, combined with the modeling method provided in the previous embodiment, the position information of each key point in the first frame image is first copied to the second frame image as the initial position information, and then based on multiple iterations, the position information of each key point in the second frame image is finally found, so that the movement trajectory of each key point based on these two frames of images can be obtained.

[0084] In the image processing method of another embodiment of the present application, step 140 includes:

[0085] Sub-step C1: According to the triangle where the first key point is located, determine the number of key points connected to the first key point and the adjacency matrix of the first key point, and obtain the Laplacian matrix of the first key point.

[0086] Sub-step C2: Obtain the eigenvalues ​​and eigenvectors of the Laplacian matrix of the first key point through orthogonal decomposition.

[0087] Sub-step C3: Obtain the eigenvector of the first key point according to the eigenvalue and eigenvector of the Laplacian matrix of the first key point.

[0088] Sub-step C4: Obtain the feature vector of the matched second key point based on the feature vector of the first key point.

[0089] Sub-step C5: Determine the third position information of the second key point according to the feature vector of the second key point.

[0090] In this embodiment, the following formula is generated through triangulation modeling:

[0091] Formula 2: L=DA;

[0092] Formula 3: L = UΛU T ;

[0093] Formula 4:

[0094] Formula 5:

[0095] In the above formulas, L is the Laplacian matrix of the graph G, which is calculated by subtracting the adjacency matrix from the degree of the key points of the graph G (as shown in Formula 2). Furthermore, the eigenvalue Λ and eigenvector U of L are obtained through orthogonal decomposition (as shown in Formula 3). Then, based on each key point in the graph network, the number of nearby key points can be observed, and different image receptive fields can be obtained, that is, different K values ​​can be selected for network training (as shown in Formula 4). t+1 is the feature vector of the first key point of the input, θ K is the parameter to be optimized, y t+1 is the eigenvector of the first key point calculated by formula 4. Finally, the eigenvector y is calculated by formula 5. t+1 Perform a linear activation and regression function to obtain the position regression point of the key point at time t+1 (corresponding to the second frame image) That is, the third position information of the first key point at frame t+1 (corresponding to the second frame image).

[0096] It should be noted that in this embodiment, L in Formula 2 is obtained based on all key points. Correspondingly, the eigenvalue Λ and eigenvector U of L obtained in Formula 3 are also for all key points. Formulas 4 and 5 are specific to a single key point. Therefore, in this embodiment, the relevant parameters of a key point in Formulas 2 and 3 can be input into Formulas 4 and 5 to obtain the results related to that key point. Thus, by repeating the above process, the relevant results for all key points can be obtained.

[0097] In the calculation process of the above formula 2 to formula 5, for x t+1 , that is, the calculation method of the input corresponding to the t+1 frame is to first extract the feature vector f(P) of the first key point corresponding to the t frame (corresponding to the first frame image) through the feature extraction function f t ) as a fixed value, i.e., x t+1 =t ; Then in P t Randomly sample a point P around t+1 , calculate its eigenvector f(P t+1 ), fuse the two eigenvectors to obtain x t+1 =[f(Pt+1),f(Pt)].

[0098] Furthermore, θ is iteratively optimized by calculating the difference of eigenvectors K and the parameters of the regression function σ, determine The calculation method is:

[0099]

[0100] Optionally, the optimization method includes:

[0101] Unlabeled dataset optimization: that is, through the feature error corresponding to Formula 6, the position information of all key points of the graph G in frame t+1 is obtained from frame t. During the optimization process, θ can be fixed first. K =I (that is, the identity matrix), train a multi-layer perceptron σ so that its output dimension is 3 and its input dimension is y t+1 The corresponding dimension; when the iteration reaches the lowest point and no longer decreases, the parameters of the multilayer perceptron σ are fixed and the iterative optimization θ K , until it reaches a minimum value.

[0102] Optimization of labeled datasets: For labeled datasets, change Formula 6 to:

[0103]

[0104] That is, the feature vector of point P in frame t is changed to the feature vector of point P in frame t+1, where For the acquisition of labeled data, the three-dimensional coordinates of the key points of each frame in the graph G can be obtained through three-dimensional modeling, thereby obtaining the true value.

[0105] In this embodiment, based on the modeling method of triangulating all key points, the position information of each key point in the first frame image is first copied to the second frame image as the initial position information, and then the iterative, optimization and other calculation formulas provided by this embodiment are used to finally find the position information of each key point in the second frame image, so that the determined position information has higher accuracy.

[0106] It should be noted that the triangulated modeling method provided above is used to track the movement trajectory of key points. Since surface objects have a large number of key points, it is more suitable for tracking surface objects.

[0107] In an image processing method of another embodiment of the present application, the time information corresponding to the first frame image in the first video is earlier than the time information corresponding to the second frame image in the first video; and the first frame image and the second frame image are two adjacent frame images.

[0108] In this embodiment, the position information of each key point may be determined in sequence for each frame image after the first frame image, thereby obtaining the movement trajectory of each key point in the first video.

[0109] For each frame of image, the position information is calculated with the previous adjacent frame of image as a reference.

[0110] For example, for the second frame image, the position information is calculated with the adjacent first frame image as a reference.

[0111] See also Figure 4 , respectively show the iterative results of graph G at different times, so that the movement trajectory of each key point can be obtained.

[0112] In this embodiment, based on the determination of the position information of each key point in each frame image, the movement trajectory of each key point of the first object can be obtained, thereby realizing trajectory tracking of the first object, so as to facilitate occlusion processing, replacement processing, etc. of the first object.

[0113] In the process of the image processing method of another embodiment of the present application, if the second feature information of the key point corresponding to the third position information obtained in the second frame image has a similarity with the first feature information less than a first threshold, then the tracking is considered to have failed. In this case, the detection of the next frame can be continued. If the proportion of images that failed to track in multiple consecutive frames is greater than the second threshold, a prompt message is directly output to prompt the user to reselect the object to be tracked; if the proportion of images that failed to track in multiple consecutive frames is less than or equal to the second threshold, and the proportion is greater than the third threshold, then tracking continues, and a prompt message can be output to remind the user that there may be occlusion problems in this tracking segment, resulting in the failure of tracking of some frames.

[0114] In the image processing method of another embodiment of the present application, step 150 includes:

[0115] Sub-step D1: Based on the correction information generated between the first frame image and the second frame image, correct the third position information to obtain sixth position information.

[0116] Optionally, the correction information may be a homography change matrix obtained from a relative position change of the second frame image relative to the first frame image.

[0117] Sub-step D2: Determine the movement trajectory of the first key point based on the first position information and the sixth position information.

[0118] In this embodiment, the change relationship between a series of video frames is obtained by calculating the homography change matrix between corresponding points between different frames, and then the perspective correction is performed on the movement trajectory of each key point according to the homography matrix between adjacent frames.

[0119] Furthermore, after correction, a template of the first video is generated. After uploading, the template can be directly used by more users to replace or block the first object without repeatedly tracking the movement trajectory of the first object in the first video.

[0120] In this embodiment, based on the calculation of the first key point and the third position information of the second frame image, correction information will be generated taking into account phenomena such as image deviation caused by some factors during the shooting process. Therefore, it is necessary to perform perspective correction on the third position information based on the correction information to obtain the sixth position information, and then determine the movement trajectory of the first key point based on the sixth position information to make the movement trajectory more realistic.

[0121] In the process of the image processing method of another embodiment of the present application, when receiving user input on the first object, input on augmented reality (AR) material can also be received, so that after successfully tracking the movement trajectory of the first object, the first object can be replaced with the selected AR material according to the movement trajectory.

[0122] In the process of the image processing method of another embodiment of the present application, in the process of replacing the first object with the selected AR material, corresponding matching and perspective correction are performed on the first object and the AR material in combination with the geometric features of the AR material and the geometric features of the movement trajectory, so that the trajectory points after correction are matched one by one with each frame of the AR material.

[0123] In the process of the image processing method of another embodiment of the present application, the AR material and the lighting in the first video are post-processed and fused using post-processing methods such as Poisson fusion, so that the AR material can match the overall tone of the video; the entire trajectory is filtered using methods such as Euro filtering to ensure the smoothness of the AR material between video frames; and finally, the rendered video that replaces the video content is output.

[0124] In the present application, a tracking algorithm is used to find the position of the same key point in each frame of the first video. Optionally, a SiamFC++ network or an optical flow-based tracking algorithm is used to implement the present application.

[0125] In the image processing method of another embodiment of the present application, this method can also be combined with the cloud server function. For some videos, such as advertisements, animations, etc., in the process of uploading videos, the uploader can choose to use the server's resources to generate recommended points, so that after the video is uploaded to the server, the various points in the movement trajectory of a certain object can be automatically generated. Furthermore, when the user loads such a video, the server will search in the video library. If the video is a video that has been uploaded to the server, the recommended points will be automatically sent to the user, and the user can directly adjust the movement trajectory, select AR materials, and other operations.

[0126] In summary, existing video editing software usually requires a lot of manual operation in the post-editing of film and television production, and requires relatively professional knowledge, and has poor detection and processing capabilities for occlusion; at the same time, non-professional creators prefer to operate on mobile terminals (such as mobile phones), and the superposition and replacement of AR special effects in film and television post-production works on mobile terminals is still a difficulty. In this application, by classifying the usage scenarios, integrating the tracking method with occlusion detection, and superimposing a small amount of user interaction, a simple and easy-to-use video post-fusion method is provided on the mobile terminal, which can quickly place AR elements in the appropriate position in the video, so that users can freely change the elements in the video content, thereby achieving the purpose of simplifying user operations. In actual applications, based on this application, the networking function can be used to provide users with some high-value videos (such as advertisements), so that users can freely replace dynamic content such as portraits in the video, thereby achieving the dynamic replacement effect of embedded advertisements in the video, so as to dynamically integrate advertisements and user interests, thereby forming a more effective advertising material delivery.

[0127] It can be seen that the present application provides a complete editing framework for editing on a mobile terminal, which combines AR elements with video editing to achieve the purpose of post-processing and processing of video content. Among them, the use of tracking, occlusion detection and other methods reduces user participation, can automatically detect occlusion, and can effectively improve the effect of video editing by combining user annotations. In addition, the present application combines the functions of the cloud server to greatly reduce the user threshold for video editing, further reduce user operations, and make it simple to replace video content in the later stage of the video. The use of AR materials also greatly improves the playability of the product, making it more attractive to video creators.

[0128] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the image processing method as an example.

[0129] Figure 5A block diagram of an image processing device according to another embodiment of the present application is shown, the device comprising:

[0130] A first determining module 10 is configured to determine N key points in a region where a first object is located in a first frame of a first video, where N is a positive integer;

[0131] An acquisition module 20 is configured to acquire first position information of a first key point in a first frame of image; and acquire first feature information of the first key point in the first frame of image; the first key point is one of the N key points;

[0132] A second determining module 30 is configured to determine second position information in a second frame image of the first video, where the first position information matches the second position information;

[0133] A third determining module 40 is configured to determine, based on the second position information, third position information in the second frame image, and match second feature information of a second key point corresponding to the third position information with the first feature information;

[0134] The fourth determining module 50 is configured to determine a movement trajectory of the first key point according to the first position information and the third position information.

[0135] Thus, in an embodiment of the present application, for the first video, the user can select the first object as the object to be processed in any frame image through a preset input. Thus, based on the user input, the first object is obtained in the first frame image, and N key points of the area where the first object is located are determined in the first frame image to represent the shape, area and other features of the first object. Furthermore, for each key point (taking the first key point as an example), the first position information and first feature information of the first key point in the first frame image are first obtained. Then, in the second frame image of the first video, the second position information matching the first position information is determined, and the third position information is determined with the second position information as the reference position. In the process of determining the third position information, the feature information of the key point on any position information can be matched based on the first feature information until the feature information of the key point on a certain position information matches the first feature information, thereby determining the position information as the third position information, that is, it is considered to be the position information of the first key point in the second frame image. Finally, based on the first position information and the third position information, the movement trajectory of the first key point is determined. It can be seen that in an embodiment of the present application, a method for obtaining the movement trajectory of a first object is provided. When the user replaces or occludes the first object, the background can directly obtain the movement trajectory of the first object and automatically process it in multiple frames of images in the first video without the need for manual operation by the user, thereby simplifying the user operation.

[0136] Optionally, N≧3; the acquisition module 20 includes:

[0137] A construction unit is used to construct a triangle by connecting every three adjacent key points among the N key points;

[0138] The acquisition unit is used to acquire first position information of the first key point in the first frame image according to the triangle where the first key point is located; and acquire first feature information of the first key point in the first frame image.

[0139] Optionally, the third determining module 40 includes:

[0140] a first determining unit, configured to determine, in the second frame image, fourth position information associated with the second position information based on the triangle where the first key point is located;

[0141] a second determining unit, configured to determine, in the second frame image, fifth position information associated with the fourth position information based on the fourth position information;

[0142] The third determining unit is configured to determine, when the fourth location information matches the fifth location information, that the fourth location information or the fifth location information is the third location information.

[0143] Optionally, the third determining module 40 includes:

[0144] a fourth determining unit, configured to determine, based on the triangle where the first key point is located, the number of key points connected to the first key point and the adjacency matrix of the first key point, and obtain a Laplace matrix of the first key point;

[0145] A first obtaining unit is used to obtain the eigenvalues ​​and eigenvectors of the Laplace matrix of the first key point through orthogonal decomposition;

[0146] A second obtaining unit is used to obtain an eigenvector of the first key point according to the eigenvalue and eigenvector of the Laplace matrix of the first key point;

[0147] A third obtaining unit is used to obtain a feature vector of a matched second key point according to the feature vector of the first key point;

[0148] The fifth determining unit is configured to determine third position information of the second key point according to the feature vector of the second key point.

[0149] Optionally, the fourth determining module 50 includes:

[0150] a correction unit, configured to correct the third position information based on correction information generated between the first frame image and the second frame image to obtain sixth position information;

[0151] The sixth determining unit is configured to determine a moving trajectory of the first key point according to the first position information and the sixth position information.

[0152] The image processing device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0153] The image processing device of the embodiment of the present application may be a device having an action system. The action system may be an Android action system, an iOS action system, or other possible action systems, which are not specifically limited in the embodiment of the present application.

[0154] The image processing device provided in the embodiment of the present application can implement each process implemented in the above method embodiment. To avoid repetition, it will not be described here.

[0155] Alternatively, as Figure 6 As shown, an embodiment of the present application also provides an electronic device 100, including a processor 101, a memory 102, and a program or instruction stored in the memory 102 and executable on the processor 101. When the program or instruction is executed by the processor 101, each step of any of the above-mentioned image processing method embodiments is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0156] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0157] Figure 7 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0158] The electronic device 1000 includes but is not limited to components such as a radio frequency unit 1001 , a network module 1002 , an audio output unit 1003 , an input unit 1004 , a sensor 1005 , a display unit 1006 , a user input unit 1007 , an interface unit 1008 , a memory 1009 , and a processor 1010 .

[0159] Those skilled in the art will understand that the electronic device 1000 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0160] Among them, the processor 1010 is used to determine N key points in the area where the first object is located in the first frame image of the first video, where N is a positive integer; obtain first position information of the first key point in the first frame image; and obtain first feature information of the first key point in the first frame image; the first key point is one of the N key points; in the second frame image of the first video, determine second position information, and the first position information matches the second position information; based on the second position information, determine third position information in the second frame image, and the second feature information of the second key point corresponding to the third position information matches the first feature information; and determine the movement trajectory of the first key point based on the first position information and the third position information.

[0161] Thus, in an embodiment of the present application, for the first video, the user can select the first object as the object to be processed in any frame image through a preset input. Thus, based on the user input, the first object is obtained in the first frame image, and N key points of the area where the first object is located are determined in the first frame image to represent the shape, area and other features of the first object. Furthermore, for each key point (taking the first key point as an example), the first position information and first feature information of the first key point in the first frame image are first obtained. Then, in the second frame image of the first video, the second position information matching the first position information is determined, and the third position information is determined with the second position information as the reference position. In the process of determining the third position information, the feature information of the key point on any position information can be matched based on the first feature information until the feature information of the key point on a certain position information matches the first feature information, thereby determining the position information as the third position information, that is, it is considered to be the position information of the first key point in the second frame image. Finally, based on the first position information and the third position information, the movement trajectory of the first key point is determined. It can be seen that in an embodiment of the present application, a method for obtaining the movement trajectory of a first object is provided. When the user replaces or occludes the first object, the background can directly obtain the movement trajectory of the first object and automatically process it in multiple frames of images in the first video without the need for manual operation by the user, thereby simplifying the user operation.

[0162] Optionally, N≧3; the processor 1010 is further used to construct every three adjacent key points among the N key points into a triangle by connecting them in pairs; based on the triangle where the first key point is located, obtain the first position information of the first key point in the first frame image; and obtain the first feature information of the first key point in the first frame image.

[0163] Optionally, the processor 1010 is further used to determine, in the second frame image, fourth position information associated with the second position information based on the triangle where the first key point is located; determine, in the second frame image, fifth position information associated with the fourth position information based on the fourth position information; and determine, when the fourth position information matches the fifth position information, that the fourth position information or the fifth position information is the third position information.

[0164] Optionally, the processor 1010 is further used to determine the number of key points connected to the first key point and the adjacency matrix of the first key point based on the triangle where the first key point is located, and obtain the Laplace matrix of the first key point; obtain the eigenvalues ​​and eigenvectors of the Laplace matrix of the first key point through orthogonal decomposition; obtain the eigenvector of the first key point based on the eigenvalues ​​and eigenvectors of the Laplace matrix of the first key point; obtain the eigenvector of the matched second key point based on the eigenvector of the first key point; and determine the third position information of the second key point based on the eigenvector of the second key point.

[0165] Optionally, the processor 1010 is further used to correct the third position information based on the correction information generated between the first frame image and the second frame image to obtain sixth position information; and determine the movement trajectory of the first key point based on the first position information and the sixth position information.

[0166] This application provides a complete editing framework for editing on a mobile terminal, which combines AR elements with video editing to achieve the purpose of post-processing and processing of video content. Among them, the use of tracking, occlusion detection and other methods reduces user participation, can automatically detect occlusion, and can effectively improve the effect of video editing by combining user annotations. In addition, this application combines the functions of the cloud server to greatly reduce the user threshold for video editing, further reduce user operations, and make it simple to replace video content in the later stage of the video. The use of AR materials also greatly improves the playability of the product, making it more attractive to video creators.

[0167] It should be understood that in an embodiment of the present application, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of a static picture or video image obtained by an image capture device (such as a camera) in a video image capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of the other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include but are not limited to a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an action stick, which will not be repeated here. The memory 1009 can be used to store software programs and various data, including but not limited to applications and action systems. The processor 1010 may integrate an application processor and a modem processor, wherein the application processor mainly processes the action system, user pages and applications, etc., and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 1010.

[0168] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory x09 may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0169] Processor 1010 may include one or more processing units. Optionally, processor 1010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1010.

[0170] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0171] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0172] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0173] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0174] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0175] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0177] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image processing method, characterized in that: The method comprises: Determine N key points in the area where the first object is located in the first frame image of the first video, where N is a positive integer; Obtaining first position information of a first key point in the first frame image; and obtaining first feature information of the first key point in the first frame image; the first key point is one of the N key points; In a second frame image of the first video, determining second position information, where the first position information matches the second position information; Based on the second position information, determining third position information in the second frame image, where second feature information of a second key point corresponding to the third position information matches the first feature information; determining a movement trajectory of the first key point according to the first position information and the third position information; The determining of third position information in the second frame image based on the second position information includes: In the second frame image, taking the second position information as the center, determine, within a specified range, the position information of the key point corresponding to the second feature information having the highest similarity to the first feature information, as the third position information; The method further comprises: The first object is replaced or blocked according to the movement trajectory.

2. The method according to claim 1, characterized in that N≧3; obtaining first position information of the first key point in the first frame image; And, obtaining first feature information of the first key point in the first frame image includes: Among the N key points, every three adjacent key points are connected into a triangle by connecting two of them; According to the triangle where the first key point is located, first position information of the first key point in the first frame image is obtained; and first feature information of the first key point in the first frame image is obtained.

3. The method according to claim 2, characterized in that The determining, based on the second position information, third position information in the second frame image includes: determining, in the second frame image, fourth position information associated with the second position information based on the triangle where the first key point is located; determining, in the second frame of image, fifth position information associated with the fourth position information based on the fourth position information; In a case where the fourth location information matches the fifth location information, the fourth location information or the fifth location information is determined to be the third location information.

4. The method according to claim 2, characterized in that The determining, based on the second position information, third position information in the second frame image includes: Determine, based on the triangle where the first key point is located, the number of key points connected to the first key point and the adjacency matrix of the first key point, and obtain the Laplacian matrix of the first key point; Obtaining the eigenvalues ​​and eigenvectors of the Laplacian matrix of the first key point through orthogonal decomposition; Obtaining an eigenvector of the first key point according to the eigenvalues ​​and eigenvectors of the Laplacian matrix of the first key point; Obtaining a matched feature vector of the second key point according to the feature vector of the first key point; The third position information of the second key point is determined according to the feature vector of the second key point.

5. The method according to claim 1, wherein The determining the movement trajectory of the first key point according to the first position information and the third position information includes: Correcting the third position information based on correction information generated between the first frame image and the second frame image to obtain sixth position information; A movement trajectory of the first key point is determined according to the first position information and the sixth position information.

6. An image processing device, characterized in that The device comprises: A first determining module is configured to determine N key points in a region where a first object is located in a first frame image of a first video, where N is a positive integer; an acquisition module, configured to acquire first position information of a first key point in the first frame image; and acquire first feature information of the first key point in the first frame image; the first key point being one of the N key points; A second determining module is configured to determine second position information in a second frame image of the first video, where the first position information matches the second position information; a third determining module, configured to determine, based on the second position information, third position information in the second frame image, wherein second feature information of a second key point corresponding to the third position information matches the first feature information; a fourth determining module, configured to determine a movement trajectory of the first key point based on the first position information and the third position information; The third determining module includes: determining, in the second frame image, with the second position information as the center, within a specified range, position information of a key point corresponding to the second feature information having the highest similarity with the first feature information, as the third position information; The device is also used for: The first object is replaced or blocked according to the movement trajectory.

7. The device according to claim 6, characterized in that N≧3; the acquisition module includes: A construction unit, configured to construct a triangle by connecting every three adjacent key points among the N key points; An acquisition unit is used to acquire first position information of the first key point in the first frame image according to the triangle where the first key point is located; and acquire first feature information of the first key point in the first frame image.

8. The device according to claim 7, characterized in that The third determining module includes: a first determining unit, configured to determine, in the second frame image, fourth position information associated with the second position information based on the triangle where the first key point is located; a second determining unit, configured to determine, in the second frame image according to the fourth position information, fifth position information associated with the fourth position information; The third determining unit is configured to determine, when the fourth location information matches the fifth location information, that the fourth location information or the fifth location information is the third location information.

9. The device according to claim 8, characterized in that The third determining module includes: a fourth determining unit, configured to determine, based on the triangle where the first key point is located, the number of key points connected to the first key point and an adjacency matrix of the first key point, and obtain a Laplace matrix of the first key point; A first obtaining unit is used to obtain the eigenvalues ​​and eigenvectors of the Laplacian matrix of the first key point through orthogonal decomposition; a second obtaining unit, configured to obtain an eigenvector of the first key point according to the eigenvalue and eigenvector of the Laplacian matrix of the first key point; a third obtaining unit, configured to obtain a matched feature vector of the second key point according to the feature vector of the first key point; A fifth determining unit is configured to determine the third position information of the second key point based on the feature vector of the second key point.

10. The device according to claim 6, characterized in that The fourth determining module includes: a correction unit, configured to correct the third position information based on correction information generated between the first frame image and the second frame image to obtain sixth position information; A sixth determining unit is configured to determine a movement trajectory of the first key point based on the first position information and the sixth position information.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 5 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 5 are implemented.

Images