Image processing method, device, electronic device and storage medium

Abstract

The present invention discloses an image processing method, device, electronic device and storage medium, the method comprising: determining at least one target limb key point of a current video frame; determining at least one current special effect key point according to at least one target limb key point and the historical limb key points of at least one historical video frame adjacent to the current video frame; obtaining the historical special effect key points of at least one historical video frame according to preset special effect parameters, and determining at least one group of target special effects matching the preset special effect parameters according to at least one current special effect key point and the historical special effect key points; determining the target video frame corresponding to the current video frame based on at least one group of target special effects. The problem that the special effect lines determined according to the motion trajectory of the target subject in the video are relatively stiff, affecting the special effect effect and resulting in a poor user experience is solved, and the special effect lines added according to the motion trajectory of the subject are smoother and softer.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] With the popularity of short videos, more and more users are shooting corresponding video information through terminal devices. In order to further improve the fun of the video content, corresponding special effects are usually added for the users in the video. In specific scenarios, the added special effects can be located by the corresponding limb key points. The existing methods for determining limb key points mainly include 2D limb key point recognition methods and 3D limb key point recognition methods. The principle is to add special effects to each limb key point based on the position of the limb key point in each video frame. However, the special effects determined according to each limb key point have the problem of stiff special effects, which in turn affects the user experience. Summary of the invention

[0003] The present invention provides an image processing method, device, electronic device and storage medium to achieve a technical effect of making special effects added according to the motion trajectory of a subject smoother and softer, thereby improving user experience.

[0004] In a first aspect, an embodiment of the present invention provides an image processing method, the method comprising:

[0005] Determine at least one target limb key point of the current video frame;

[0006] Determine at least one current special effect key point according to the at least one target limb key point and the historical limb key points of at least one historical video frame adjacent to the current video frame;

[0007] Acquire historical special effect key points of at least one historical video frame according to preset special effect parameters, and determine at least one set of target special effects matching the preset special effect parameters according to the at least one current special effect key point and the historical special effect key points;

[0008] Based on the at least one set of target special effects, a target video frame corresponding to the current video frame is determined.

[0009] In a second aspect, an embodiment of the present invention further provides an image processing device, the device comprising:

[0010] A target limb key point determination module, used to determine at least one target limb key point of the current video frame;

[0011] A current special effect key point determination module, used to determine at least one current special effect key point according to the at least one target limb key point and a historical limb key point of at least one historical video frame adjacent to the current video frame;

[0012] A target special effect group determination module, used to obtain historical special effect key points of at least one historical video frame according to preset special effect parameters, and determine at least one group of target special effects matching the preset special effect parameters according to the at least one current special effect key point and the historical special effect key point;

[0013] The target video frame determination module is used to determine the target video frame corresponding to the current video frame based on the at least one set of target special effects.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0015] one or more processors;

[0016] a storage device for storing one or more programs,

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of the embodiments of the present disclosure.

[0018] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the image processing method as described in any one of the embodiments of the present disclosure.

[0019] The technical solution of the embodiment of the present disclosure determines at least one target limb key point of the current video frame, and according to the target limb key point, the historical limb key points in each historical video frame corresponding to the target limb key point can be obtained, and then the limb key point group corresponding to the target limb key point is determined, and then special effects are added to each limb key group. According to the at least one target limb key point and the historical limb key points of at least one historical video frame adjacent to the current video frame, at least one current special effect key point is determined, and then the corresponding weight value is determined according to the generation time of each historical video frame, and each limb key point is smoothed to make the connection between each key point softer. According to the preset special effect parameters, the historical special effect key points of at least one historical video frame are obtained, and according to the at least one current special effect key point and the historical special effect key point, at least one group of target special effects matching the preset special effect parameters are determined, and according to the preset special effect parameters and the historical limb key points, multiple groups of target special effects matching the preset special effect parameters can be obtained. Based on the at least one group of target special effects, the target video frame corresponding to the current video frame is determined, and a smoother and softer video special effect is displayed to the user based on the target video frame. The problem that the special effects lines determined according to the motion trajectory of the target subject in the video are relatively stiff, affecting the special effects effect and causing a poor user experience has been solved. The special effects lines added according to the motion trajectory of the subject are made smoother and softer. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0021] Figure 1 A flowchart of an image processing method provided in Embodiment 1 of the present disclosure;

[0022] Figure 2 A schematic diagram of a connection line between a target limb key point and a historical limb key point before processing provided in the first embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of a connection line between a target limb key point and a historical limb key point after processing provided in the first embodiment of the present disclosure;

[0024] Figure 4 A flowchart of an image processing method provided in Embodiment 2 of the present disclosure;

[0025] Figure 5 A schematic diagram of a target effect curved surface provided in the second embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of a target effect curved surface with its tail end fading provided in the second embodiment of the present disclosure;

[0027] Figure 7 A schematic diagram of a target effect curved surface provided in the second embodiment of the present disclosure after soft processing;

[0028] Figure 8 A flowchart of an image processing method provided in Embodiment 3 of the present disclosure;

[0029] Fig. 9 A schematic diagram of the structure of an image processing device provided in Embodiment 4 of the present disclosure;

[0030] Fig.10 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present disclosure. DETAILED DESCRIPTION

[0031] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0032] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0033] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0034] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0035] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0036] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0037] With the popularity of short videos, in order to make the screen of the short video cooler, special effects can be added to the video. The embodiments of the present disclosure can be applied to the screen that needs to display special effects. For example, the application scenario can be a short video of dancing. When adding special effects to the short video, special effects that follow the swing of the subject's limbs can be added. It can be understood that the subject movement in the video is continuous. When adding special effects according to the subject's limbs, special effects processing can be performed according to the lines of the swing of the limbs. Considering that the lines of special effects added to the video in the prior art are relatively stiff, not beautiful enough, and affect the user experience, the technical solution in the embodiments of the present disclosure can be used to perform special effects processing on each video frame in the video to make the special effects softer.

[0038] Embodiment 1

[0039] Figure 1 This is a flow chart of an image processing method provided in the first embodiment of the present disclosure. This embodiment can be applied to the situation where special effects are added to a video and the special effects are processed more gently. The method can be executed by an image processing device, which can be implemented in the form of software and / or hardware. The hardware can be an electronic device, such as a mobile terminal, a PC or a server.

[0040] like Figure 1 As shown, the method of the embodiment of the present disclosure includes:

[0041] S110: Determine at least one target limb key point of the current video frame.

[0042] It should be noted that the implementation of this technical solution can be performed by the server, or by the client, or by the client and the server. For example, the client shoots the corresponding video frame, and the client processes the video frame to add corresponding special effects to the video frame; or the shot video frame is uploaded to the server, and after the server completes the processing, it is sent to the client, so that the client can display the video frame with special effects added.

[0043] Among them, when adding special effects to the video, the special effects processing can be performed on the video uploaded to the server by the corresponding user. For example, the user can shoot a video through the application software and then send the video to the server. The server can use this technical solution to add corresponding special effects to each video frame in the video; it can also be applied in the process of video shooting, that is, when shooting and playing at the same time, the shot video frame can be uploaded to the server, and the server can execute this technical solution to process the special effects. The current video frame can be understood as a video frame that is currently being processed with special effects by the server or client or a video frame that is about to be processed with special effects. There may or may not be a target subject in the video frame. The target subject can be a person or an object, and its specific content matches the pre-set parameters.

[0044] Exemplarily, a dancing video is processed, and the dancing user can be used as the target subject. The target subject in the video frame can be identified by an image recognition method. If the target subject does not exist, no special effects are added to the video. If the target subject exists, the limb key points of the target subject are identified by a corresponding recognition method to obtain at least one target limb key point. The target limb key point can be understood as the key point of the target subject determined based on the limb key point recognition method. In this embodiment, the limb key point can be each joint point of the user, and the joint point can be the center point of the user's head, neck, shoulder, elbow, hand, hip, leg and foot. The target limb key point can be understood as the specified limb key point of the user in the video image. For example, special effects are added to the limb key point corresponding to the user's wrist, so that the special effects can move according to the swing of the wrist, and the limb key point corresponding to the user's wrist is the target limb key point. It should be noted that the number of target limb key points can be set to one or more. For example, if the limb key points at two wrists are used as target limb key points, when the limb key point recognition algorithm is used to detect the presence of two limb key points at wrists in the video frame, then the number of target limb key points is two; if one of the limb key points at the wrist in the video frame is blocked and the limb key point cannot be detected by the limb key point recognition algorithm, then the number of target limb key points is one.

[0045] Specifically, taking the target video as a dance video as an example, one of the video frames in the dance video is used for special effects processing. First, an image recognition method is used to detect whether there is a target subject in the video frame. If there is no target subject, no special effects processing is performed on the video. If there is a target subject, a limb key point recognition algorithm is further used to determine the limb key points of the target subject and the limb key points recognized in the current video frame are used as the target limb key points.

[0046] It should be noted that by identifying the target limb key points in the video frames, special effects can be added to the identified target limb key points in each video frame, so that when the video is played, the special effects corresponding to the target limb key points can produce a linkage effect.

[0047] Optionally, determining at least one target limb key point of the current video frame includes: using a limb key point recognition algorithm to determine at least one target limb key point in the current video frame; or, using a pre-trained limb key point recognition model to process the current video frame to obtain at least one target limb key point corresponding to the current video frame.

[0048] Among them, the recognition of limb key points can be achieved in two ways. One is a limb key point recognition algorithm, which can include a 2D limb key point recognition algorithm or a 3D limb key point recognition algorithm. Based on the algorithm, the target subject in the video frame is recognized and the limb key points of the target subject can be determined. The second is a neural network model algorithm, which pre-establishes a limb key point recognition model and then uses the model to recognize the limb key points of the target subject in the video frame. Based on any of the above methods, the target subject in the video frame is recognized.

[0049] It should be noted that the obtained limb key points can be represented by the coordinates of pixel points in the video frame. Furthermore, based on the coordinate point position of each limb key point in the corresponding video frame, special effects can be added to each limb key point.

[0050] Specifically, the principle of identifying the target limb key points is mainly through identifying certain feature points in the image. A priori knowledge base of limb key points is established in advance, and the priori knowledge base includes the shape, color, texture, line and other features of each limb key point. After the video frame is input, the video frame is scanned to determine whether the video frame contains feature points corresponding to the priori knowledge base. If it can match the feature points in the priori knowledge base, it can be determined that the video frame contains limb key points. Then at least one target limb key point in the current video frame is determined.

[0051] Alternatively, a training model for limb key point recognition is established in advance, and corresponding parameters are set. Various human images and limb key point images are input into the model, and images containing limb key points are labeled to determine whether the image contains limb key points based on the labels, and then the set training model is used as a limb key point recognition model. Then, the newly input image is recognized based on the recognition model. Exemplarily, when processing a video, the limb key points in the video frame are recognized based on a pre-trained neural network model to determine whether the video frame contains limb key points, and then the pre-set key point recognition rules are used to determine the key points that meet the key point recognition rules as at least one target limb key point in the video frame.

[0052] S120. Determine at least one current special effect key point according to the at least one target limb key point and historical limb key points of at least one historical video frame adjacent to the current video frame.

[0053] The historical video frames can be understood as the video frames before the current video frame based on the current video frame. The limb key points to be detected can be pre-set. The historical limb key points can be understood as the key point coordinates corresponding to the preset limb key points in each historical video frame.

[0054] It can be understood that the wrist part is a preset limb key point, and the historical limb key point can be understood as the coordinates corresponding to the wrist part of the target subject in each historical video frame. The current special effect key point can be understood as the key point that is the same as or has a relative position difference with the limb key point when adding special effects to the limb key point, which can be determined according to each historical frame, that is, the current special effect key point can be understood as the key point for adding special effects to the limb key point. It should be noted that the number of current special effect key points can be one or more, and the specific number can match the number of preset limb key points. For example, taking a dance video as an example, if the special effects you want to add can be linked with the limb key points corresponding to the wrist and ankle areas of the target subject, then the number of current special effect key points is 4, namely wrist 1, wrist 2, foot 1 and foot 2. When determining the special effects corresponding to wrist 1, the coordinate point position of wrist 1 where special effects need to be added can be determined based on the coordinate point position of the limb key point corresponding to wrist 1 in the current video frame and the coordinate point position of wrist 1 in each historical video frame, and the coordinate point position can be used as the current special effects key point.

[0055] It should be noted that when obtaining historical limb key points, all limb key points in each historical video frame may be obtained, or some limb key points in each historical video frame may be obtained. Taking the limb key points of the wrist as an example, the limb key point may be composed of multiple pixels. According to the coordinate point positions of the pixels corresponding to the wrist in each historical video frame, the coordinate point positions of the pixels corresponding to the wrist in the current video frame can be determined, and then the coordinate position of the current special effect key point can be determined based on the coordinate point positions of the pixels corresponding to the wrist.

[0056] Optionally, determining at least one current special effect key point based on the at least one target limb key point and the historical limb key points of at least one historical video frame adjacent to the current video frame includes: determining the limb key points belonging to the same part as a limb key point group; wherein the limb key point group includes target limb key points and / or historical limb key points.

[0057] The limb key point group can be understood as a limb key point group composed of target limb key points of the current video frame and / or historical limb key points. It should be noted that each limb key point of the limb key point group corresponds to the same part. For example, if the limb key point is an ankle, the limb key point group includes the ankle key point in the current video frame and / or the ankle key point in the historical video frame.

[0058] Specifically, when adding special effects to limb key points in a video, in order to maintain the continuity of the video special effects, limb key points belonging to the same part may be grouped as a limb key point group.

[0059] For example, when a video is processed with special effects, the area where the special effects are added is the target limb key point area, and the number of target limb key points in the current video frame can be one or more, so the number of current special effect key points can be determined based on the number of target limb key points. The current video includes two target limb key points, which are the limb key points corresponding to the two wrists, and the target limb key points and / or historical limb key points corresponding to each wrist can be used as a limb key point group, so the current video includes two groups of limb key point groups.

[0060] Optionally, for each limb key point group, the key points to be processed are determined based on the weight values ​​of the historical limb key points in the current limb key point group and the corresponding key point coordinates; the current special effect key points corresponding to the limb key point group are determined based on the key points to be processed and the corresponding target key points corresponding to the limb key point group.

[0061] In the prior art, when adding special effects to each limb key point, the special effects are usually added based on the target limb key point in the current video frame, and then the special effect points in each historical video frame are added to the current video frame, that is, the special effect key points in each video frame are connected to generate a corresponding polyline, and the corresponding special effect effect is displayed according to the polyline. Figure 2 At this time, the generated special effect lines are relatively stiff, and correspondingly, the added special effect lines will be relatively stiff, resulting in a poor user experience. Based on this, this technical solution can be used to perform special effect processing on the key points of the limbs to obtain a softer special effect line effect.

[0062] like Figure 3 As shown, the dotted line in the figure is the original broken line trace for adding special effects to the limb key points based on the existing technology, and the solid line is the connection line between the key points to be processed after processing. It can be seen that the line transition determined based on the present technical solution is gentle, relatively soft and smooth. Correspondingly, the final effect presented is also relatively soft and smooth. The specific implementation method to achieve this effect can be found in the detailed description below.

[0063] There can be multiple limb key point groups, and the processing method for each is the same. Take one group as an example and refer to this group as the current subject key point group. Key point coordinates can be understood as the coordinate position of each limb coordinate point in each video frame image; key points to be processed can be understood as the key points of historical limb key points of each historical video frame after smoothing; target key points can be understood as the key points of target limb key points in the current video frame after smoothing; current special effect key points can be understood as the key points of adding special effects to the current video frame.

[0064] Specifically, when determining the current special effect key point based on the technical solution, and determining the special effect line based on each historical special effect key point and the current special effect key point, considering that the current special effect key point is also affected by the historical special effect points in each historical video frame, the weight value corresponding to each historical video frame can be pre-set, and then the video frame is processed with special effects based on the current video frame and each historical video frame to obtain the special effect key points in each historical video frame, and the special effect key points are connected to obtain a relatively smooth special effect line, and then based on this special effect line, a special effect effect with a smooth transition can be obtained, see Figure 3 The solid line content in .

[0065] On the basis of the above, in order to further obtain softer special effects content, it can be: determine the weight value of each historical video frame according to the generation time of each historical video frame; wherein the size of the weight value is inversely proportional to the generation time; obtain the intermediate value by calculating the product of the key point coordinates of the historical limb key point and the corresponding weight value, and obtain the key point to be processed by accumulating each intermediate value.

[0066] Among them, the video can be composed of multiple video frames arranged in time series. When setting the weight value of each historical video frame, it can be determined according to the interval length between each historical video frame and the current video frame. The longer the interval length, the farther the historical video frame is from the current video frame, and the smaller the impact on the current video frame is, the smaller the weight value of the corresponding video frame is; conversely, the larger the weight value corresponding to the historical video frame is, that is, the size of the video frame weight value is inversely proportional to the generation time of the video frame. Exemplarily, if the weight value of the current video frame is set to 1, the weight value of the penultimate video frame of the current video frame can be set to weight, where weight<1; then the weight value of the penultimate video frame of the current video frame can be set to weight 2 ; The weight value of the third last video frame of the current video frame can be set to weight 3 , and so on, the weight value of each historical video frame is determined.

[0067] Specifically, when combining various historical video frames for special effects processing, the number of historical video frames combined is limited, and can usually be set to 8 frames. Then a circular array with a length of 8 can be set to save the coordinate point positions of each historical video frame. For example, when the 9th video frame is the current video frame, the special effect key points in the 9th frame can be determined based on the coordinate points corresponding to the first 8 historical video frames in the array. Among them, the weight value corresponding to the 8th video frame can be weight, and the weight value corresponding to the 7th historical frame can be weight 2 , the weight value corresponding to the sixth historical frame is weight 3 , and so on, determine the weight value corresponding to each historical video frame. Multiplying the coordinate value in each historical video frame with the weight value corresponding to the historical video frame to which it belongs can obtain the intermediate value of each historical video frame, and by accumulating each intermediate value, the key point to be processed can be obtained.

[0068] It should also be noted that when processing the 10th video frame, the array needs to be updated, and the coordinates of the historical special effects points of the 2nd to 9th frames can be stored in the array, and the first data stored in the array can be eliminated to obtain an updated array, and based on the updated array, the current special effects key points of the 10th frame are determined. In other words, the array always stores a preset number of historical video frames before the current video frame, and the generation time interval between the historical video frames and the current video frames is small. The advantage of this setting is that the influence of each historical video frame on the special effects point of the current video frame is taken into account, and a special effects line with a natural transition can be obtained, thereby showing a softer special effects effect.

[0069] S130. Obtain historical special effect key points of at least one historical video frame according to preset special effect parameters, and determine at least one group of target special effects that match the preset special effect parameters according to the at least one current special effect key point and the historical special effect key points.

[0070] Among them, the preset special effect parameters can be understood as the length and width of the pre-set special effect area. The specific preset special effect parameters can be set according to the length of the special effect area that you want to display, and the length and width of the special effect area are set accordingly. The historical special effect key points can be understood as the key points for adding special effects in each historical video frame; the target special effect can be understood as the special effect displayed in the current video, which can be a special effect bar or a special effect pattern, wherein the number of target special effects corresponds to the number of limb key point groups, and the target special effects match the preset special effect parameters. At least one group of target special effects can be understood as one or more target special effects added to the video frame, and the number of them matches the number of limb key point groups.

[0071] Specifically, when adding special effects to a video, each video frame includes not only the current special effect point in the current video frame, but also the historical special effect key points of each historical video frame, and the number of current special effect points included in the video frame can be one or more. According to the preset length and width of the special effect bar, the historical special effect key points of each historical frame and at least one current special effect key point are displayed on the special effect area accordingly, and at least one set of target special effects matching the preset special effect parameters is determined.

[0072] Optionally, the preset special effect parameters include a special effect bar display length, and obtaining historical special effect key points of at least one historical video frame according to the preset special effect parameters includes: obtaining historical special effect key points of at least one historical video frame before the current video frame according to the special effect bar display length.

[0073] The special effect bar can be understood as a specific special effect. The preset special effect display parameters include the display shape, length, width, etc. of the special effect bar. The special effect bar can be a rectangle, a triangle, or any personalized shape, such as a fish shape, a dragon shape, etc.

[0074] Specifically, according to the preset special effect parameters, not only the length and width of the special effect bar can be included, but in order to make the special effect more beautiful, geometric figures or patterns can also be included. For example, the pattern can be a triangle, a dragon pattern, a butterfly pattern, a rainbow bar, and any other pattern with set width and length. It should be noted that the length and width of the special effect bar are matched with the preset special effect parameters, and the length and width of the special effect bar are determined according to the preset special effect parameters. When displaying special effects in a video frame, the length of the special effect is not infinite, but is matched according to the preset special effect parameters. When displaying special effects, the historical special effect key points in each historical video frame before the current video frame can be obtained according to the display length of the special effect bar, and the historical special effect key points can be displayed within the length range of the special effect bar.

[0075] Optionally, determining at least one group of target special effects that match the preset special effects parameters based on the at least one current special effects key point and the historical special effects key point includes: obtaining multiple groups of target special effects that match the preset special effects parameters by processing the historical special effects key points and the current special effects key points of the same part.

[0076] Specifically, the number of target special effects in the video frame matches the number of current special effect key points. If two wrist parts and two foot areas of the human body are used as target limb key points, and special effects are added to the target limb key points, the target limb key points with added special effects can be used as current special effect key points. Considering that the target subject in the video frame may have a situation where the target limb key points are blocked during the movement, the number of current special effect key points and the number of target limb key points can be equal or unequal. The number of current special effect key points of the target special effect in the video frame is at most 4, and each part can correspond to a group of target special effects. Further, by processing the historical special effect key points and the current special effect key points of the same part, multiple groups of target special effects that match the preset special effect parameters can be obtained. Exemplarily, if the number of target limb key points is a group, the length of the special effect bar can be determined according to the preset special effect parameters, and by processing the historical special effect key points and the current special effect key points of the limb key points of the same part, a group of target special effects that match the preset special effect parameters can be obtained.

[0077] S140: Determine a target video frame corresponding to the current video frame based on the at least one set of target special effects.

[0078] Specifically, after determining at least one set of target special effects, the at least one set of target special effects is added to corresponding positions of the current video frame, and then the video frame after the special effects are added is used as the target video frame.

[0079] Optionally, determining the target video frame corresponding to the current video frame based on the at least one group of target special effects includes: adding each group of target special effects to the corresponding target limb key points to obtain the target video frame corresponding to the current video frame.

[0080] Specifically, based on the above example, at least one group of target special effects includes a group of target special effects corresponding to wrist 1, a group of target special effects corresponding to wrist 2, a group of target special effects corresponding to foot 1, and a group of target special effects corresponding to foot 2. Then, the target special effects can be added to the corresponding target limb key points to obtain the target video frame.

[0081] The technical solution of the embodiment of the present disclosure determines at least one target limb key point of the current video frame, and according to the target limb key point, the historical limb key points in each historical video frame corresponding to the target limb key point can be obtained, and then the limb key point group corresponding to the target limb key point is determined, and then special effects are added to each limb key group. According to the at least one target limb key point and the historical limb key points of at least one historical video frame adjacent to the current video frame, at least one current special effect key point is determined, and then the corresponding weight value is determined according to the generation time of each historical video frame, and each limb key point is smoothed to make the connection between each key point softer. According to the preset special effect parameters, the historical special effect key points of at least one historical video frame are obtained, and according to the at least one current special effect key point and the historical special effect key point, at least one group of target special effects matching the preset special effect parameters are determined, and according to the preset special effect parameters and the historical limb key points, multiple groups of target special effects matching the preset special effect parameters can be obtained. Based on the at least one group of target special effects, the target video frame corresponding to the current video frame is determined, and a smoother and softer video special effect is displayed to the user based on the target video frame. The problem that the special effects lines determined according to the motion trajectory of the target subject in the video are relatively stiff, affecting the special effects effect and causing a poor user experience has been solved. The special effects lines added according to the motion trajectory of the subject are made smoother and softer.

[0082] Embodiment 2

[0083] Figure 4 This is a flow chart of an image processing method provided in the second embodiment of the present disclosure. Based on the above embodiments, the special effect processing can be further optimized to make the obtained special effect effect softer, wherein the technical terms that are the same as or corresponding to the above embodiments are not repeated here.

[0084] like Figure 4 As shown, the method includes:

[0085] S210: Determine at least one target limb key point of the current video frame.

[0086] S220. Determine at least one current special effect key point according to the at least one target limb key point and the historical limb key points of at least one historical video frame adjacent to the current video frame.

[0087] S230, determining the current to-be-processed rendering effect of the current special effect key point according to the target effect surface corresponding to the current video frame, and updating the historical rendering effects of the historical special effect key points in the current video frame.

[0088] Among them, the target effect surface can be a pre-set surface, or it can be a special effect surface dynamically generated when determining the target special effect of each video frame. The target effect surface can include effect parameters corresponding to the special effect, such as the color and transparency of the special effect. Parameters such as the length, width and shape of the target effect surface can also be set. The current rendering effect to be processed can be understood as the special effect information corresponding to the current special effect point determined based on the target effect surface, and the special effect information can be information such as the color, width and transparency of the special effect. It can be understood that each historical video frame also has a corresponding historical special effect key point, and each historical special effect key point can correspond to a pixel point in the current video frame, and the special effect display information of the corresponding pixel point can be updated based on the target effect surface.

[0089] Specifically, the special effect information related to the special effect can be filled into the preset target effect surface. In the target effect surface, the relevant special effect information can be color information, transparency information, width information, etc. Since the special effect has continuity, that is, each historical special effect key point can be displayed on the current video frame accordingly, considering the influence of each historical video frame on the current video frame, based on the rendering processing of the current special effect key point, the historical rendering effect of the historical special effect key point in the current video frame can be updated accordingly.

[0090] Optionally, determining the current rendering effect to be processed of the current special effect key point based on the target effect surface corresponding to the current video frame also includes: creating a surface to be processed corresponding to the target special effect; determining the vertical coordinate values ​​corresponding to each horizontal coordinate in the surface to be processed, and determining the surface to be used based on each vertical coordinate value; determining the rendering effect of each point in the surface to be used to obtain the target effect surface.

[0091] The surface to be processed is a pre-created surface, specifically a surface corresponding to a certain length and width, and having a certain curvature. The horizontal coordinate value is set for the created surface to be processed, and the vertical coordinate value corresponding to each horizontal coordinate is determined accordingly, and the surface to be processed corresponding to the horizontal and vertical coordinate values ​​is used as the surface to be used.

[0092] Specifically, a surface to be processed corresponding to the target special effect can be created in advance, the surface has a corresponding length and width, the length direction is used as the horizontal coordinate, and the value of the vertical coordinate corresponding to each horizontal coordinate is set accordingly to obtain the surface to be used. Furthermore, in order to make the video more vivid, special effects processing can be performed on each pixel point in the surface to be used, and corresponding color values ​​and transparency information can be added to each pixel point in the surface to be used, so as to obtain the target effect surface.

[0093] For example, see Figure 5 , establish a coordinate system with the origin of the surface to be processed as the center, with the horizontal direction as the X axis and the vertical direction as the Y axis. Based on this coordinate system, determine the vertical coordinate value corresponding to the horizontal coordinate, which is the width of the surface to be processed. For example, the change step of the X axis is 0.1, and the Y axis changes by 0.01 for every 0.1 increase in the X axis. The changes of the X and Y axes can be set according to the actual situation. Of course, the smaller the change rate, the better the smoothness of the special effect. For the schematic diagram of the special effect after the change, see Figure 6 .

[0094] It should also be noted that the change in Y value is Figure 6 The dotted line in the figure changes along the central symmetry axis to ensure the coordination of the special effects display.

[0095] At this time, there is no corresponding color and transparency in the special effect diagram. You can set the corresponding color and transparency for each point in the used surface based on the X-axis to obtain a target effect surface with better special effect. Figure 7 .

[0096] Optionally, determining the current rendering effect to be processed of the current special effect key point based on the generated target effect surface, and updating the historical rendering effects of the historical special effect key points in the current video frame, includes: determining the horizontal coordinate corresponding to the target effect surface based on the generation time information of the historical video frames to which the historical special effect key points belong and the current video frame; determining the current rendering effect to be processed of the current video frame based on the horizontal coordinate, and updating the historical rendering effects of the historical special effect key points corresponding to the historical video frames.

[0097] Specifically, the special effects information displayed in the target effect surface includes not only the special effects information of the current special effects key points, but also the special effects information of the historical special effects key points. Based on the generation time information of each historical video frame, the corresponding horizontal coordinate of each historical video frame in the target effect surface is determined. The earlier the generation time of the historical video frame, the smaller the value of the horizontal coordinate of the corresponding target effect surface. Furthermore, the transparency, color or shape of the corresponding position of the historical special effects key point in the current video frame can be set to determine the current rendering effect to be processed of the current video frame.

[0098] S240: Determine a plurality of groups of target special effects that match the preset special effect parameters according to the historical rendering effects of the same part and the current rendering effect to be processed.

[0099] Among them, the total length of each target field in the target effect surface matches the length parameter of the special effect bar in the preset special effect parameter.

[0100] Among them, each target field in the target effect surface can be understood as the field of each historical video frame displayed in the current video frame, including the historical rendering effects in each historical video frame and the number of each historical video frame. The total length of each target field matches the length parameter of the special effects bar in the preset special effects parameters.

[0101] Specifically, according to the pixel points corresponding to the historical special effects key points in each historical video in the current video frame, and then based on the special effects display information in the preset special effects parameters, such as display length, width, transparency and other information, each pixel point is displayed according to the corresponding special effects display information, and the target special effects corresponding to the current video frame can be obtained.

[0102] S250: Determine a target video frame corresponding to the current video frame based on the at least one set of target special effects.

[0103] The technical solution of the disclosed embodiment determines the current pending rendering effect of the current special effect key point according to the target effect surface corresponding to the current video frame, and updates the historical rendering effect of the historical special effect key point in the current video frame, and displays the rendering effect of the current special effect key point and the updated historical special effect key point in the current video frame. Then, based on the historical rendering effect of the same part and the current pending rendering effect, multiple groups of target special effects that match the preset special effect parameters are determined; wherein the total length of each target field in the target effect surface matches the length parameter of the special effect bar in the preset special effect parameters, and special effects are added to each key point in the video frame, and the target video corresponding to the current video frame can be obtained based on each video frame after the special effects are added. This solves the problem that the special effect effect is not soft enough when displaying special effects, and achieves a smoother and softer special effect effect.

[0104] Embodiment 3

[0105] Figure 8 A flow chart of an image processing method provided for the third embodiment of the present disclosure is provided. Based on the above two embodiments, the special effect processing can be further optimized to make the obtained special effect effect softer, wherein the technical terms that are the same as or corresponding to the above embodiments are not repeated here.

[0106] like Figure 8As shown, the method specifically includes:

[0107] S310: Determine at least one target limb key point of the current video frame.

[0108] S320: Determine at least one current special effect key point according to the at least one target limb key point and the historical limb key points of at least one historical video frame adjacent to the current video frame.

[0109] S330, obtaining historical target effect surfaces of a plurality of historical video frames that are adjacent to the current video frame and before the current video frame.

[0110] Specifically, each video frame has a corresponding target effect surface, and the target effect surface of each historical video frame is called a historical target effect surface.

[0111] It is understandable that in actual applications, if the rendering effect of a frame of special effects key points is displayed on the screen, the special effects display effect is already relatively soft and smooth. However, in order to further improve the softening effect, the target effect surface of multiple historical video frames with a smaller interval time before the current video frame can be obtained for further processing.

[0112] S340, determining the current to-be-processed rendering effect of the current special effect key point according to each historical target effect surface and the target effect surface, and updating the historical rendering effect of the historical special effect key point in the current video frame.

[0113] Specifically, by superimposing the target effect surface of the current video frame and the target effect surface corresponding to each historical video frame, the display information of the historical special effect key points in the current video frame can be determined, and the historical rendering effect of the historical special effect key points in the current video frame can be obtained.

[0114] Optionally, determining the current rendering effect to be processed of the current special effect key point according to each historical target effect surface and the target effect surface, and updating the historical rendering effects of each historical special effect key point in the current video frame, includes: determining the special effect superposition weight value corresponding to each historical target effect surface; obtaining the rendering effect to be superimposed of each pixel point in each group of historical target special effects according to the special effect superposition weight value and the corresponding target effect surface; determining the current rendering effect to be processed of the current special effect key point according to each rendering effect to be superimposed and the target effect surface of the current video frame, and updating the historical rendering effects of the historical special effect key point in the current video frame.

[0115] Among them, when performing special effects processing, the special effects of each historical video frame will also affect the special effects of the current video frame. When determining the special effects of the current video frame, the special effects of each historical video frame can be superimposed. The special effects superposition weight value can be understood as the influence value of the historical target effect surface of each historical video frame on the current video frame. Generally, the historical video frame with a greater time distance from the current video frame generation time has less impact on the current video frame image. In this case, the special effects superposition weight value of the historical video frame with a longer time interval from the current video frame can be set to the minimum value, and so on, to determine the special effects superposition weight value of each historical video frame. The rendering effect to be superimposed can be understood as the rendering effect after the superposition of each historical target effect surface.

[0116] It should also be noted that at this time, the number of historical video frames can be pre-set, optionally, 5 historical video frames.

[0117] Specifically, the longer the interval with the current video frame, the smaller the impact of the historical video frame on the current video frame. In this case, the weight value corresponding to each historical video frame is determined according to the interval between each historical video frame and the current video frame. The interval is inversely proportional to the weight, that is, the longer the interval, the smaller the weight. Each historical effect surface corresponds to multiple pixel points in the video frame. The weight values ​​of the corresponding special effects of the multiple pixel points corresponding to each historical effect surface can be superimposed. The pixel points corresponding to each point in the current video frame and the special effect display information of the pixel points can be determined. Based on this, the pending rendering effect of the current special effect key point can be updated, and the historical rendering effect of the historical special effect key point in the current video frame can be updated.

[0118] Optionally, determining the current rendering effect to be processed of the current special effect key point according to each rendering effect to be superimposed and the target effect surface of the current video frame, and updating the historical rendering effects of the historical special effect key points in the current video frame, includes: determining the historical rendering effects of each historical special effect key point in the current video frame according to the target effect surface of the current video frame, and determining the rendering effect to be processed of the current special effect key point; superimposing each rendering effect to be superimposed on the corresponding pixel point in the current video frame, and updating each historical rendering effect and the rendering effect to be processed.

[0119] Specifically, in order to obtain a better rendering effect, the special effects can be tailed to make the special effects more lively. Record the coordinate value position corresponding to each historical special effect key point, and determine the coordinate value position of the historical special effect key point in the current video frame, and then superimpose the historical rendering effect of the historical special effect key point on the corresponding pixel point in the current video frame, and combine each historical rendering effect with the current effect to be processed. Set the horizontal coordinate of the target effect surface according to the time series, and set the value of the vertical coordinate corresponding to each horizontal coordinate. According to the generation time of each historical video frame, the width value corresponding to the current video frame is the largest, and the farther the historical video frame is from the current video frame, the smaller the special effect width value corresponding to the historical video frame is. Superimposing the historical rendering effect of each historical video frame with the pending rendering effect of the current video frame can produce a tailing effect. As the video frame is played, the same processing is performed on each video frame, and each historical rendering effect and the current pending rendering effect are updated.

[0120] Optionally, if it is detected that the rendering effect value corresponding to each pixel point is less than a preset rendering effect threshold, the rendering effect value of the pixel point is cleared to obtain a target video frame corresponding to the current video frame.

[0121] The preset rendering effect threshold may be understood as a value of the rendering effect pre-set for each pixel.

[0122] Specifically, the rendering effect of the current video frame includes the special effect display information corresponding to each historical video frame. Since the special effect will be processed in combination with the target effect surface of the historical video frame, there may be a situation where the special effect value of some pixels is relatively small. However, if the special effect corresponding to the pixel is still displayed at this time, there will be a problem of unclean picture. Based on this, it can be determined whether to clear it according to the special effect value of the corresponding pixel. The preset rendering effect threshold can be set to 0.01. When the rendering effect corresponding to the pixel is lower than 0.01, it means that the rendering effect corresponding to the pixel can be cleared. The advantage of this is that if the rendering effects in each historical video frame are retained when displaying the video special effects, the rendering effect in the video will be messy, affecting the display of the rendering effect. Therefore, the rendering effect threshold corresponding to each pixel can be set to clear the rendering effects of pixels below this value, thereby obtaining a video frame with both special effects and a relatively clean picture.

[0123] S350: Determine a plurality of groups of target special effects that match the preset special effect parameters according to the historical rendering effects of the same part and the current rendering effect to be processed.

[0124] S360: Determine a target video frame corresponding to the current video frame based on the at least one set of target special effects.

[0125] The technical solution of the disclosed embodiment obtains the historical target effect surfaces of multiple historical video frames adjacent to the current video frame and before the current video frame, and then determines the current rendering effect to be processed according to the corresponding position of the current special effect key point in each historical target effect surface. According to each historical target effect surface and the target effect surface, the current rendering effect to be processed of the current special effect key point is determined, and the historical rendering effects of the historical special effect key point in the current video frame are updated, and the historical rendering effects with a longer interval with the current video frame are cleared, and then multiple groups of target special effects are determined based on each historical rendering effect and the rendering effect to be processed in the current video frame. The problem of unsightly special effects caused by the long retention time of historical rendering effects on the screen is solved, and the impact of historical video rendering effects is reduced, thereby softening the trailing effect of special effects.

[0126] Embodiment 4

[0127] Fig. 9 This is a structural schematic diagram of an image processing device provided in Embodiment 4 of the present disclosure, and the device specifically includes: a target limb key point determination module 410, a current special effect key point determination module 420, a target special effect group determination module 430 and a target video frame determination module 440.

[0128] A target limb key point determination module 410 is used to determine at least one target limb key point of the current video frame;

[0129] A current special effect key point determination module 420, configured to determine at least one current special effect key point based on the at least one target limb key point and a historical limb key point of at least one historical video frame adjacent to the current video frame;

[0130] A target special effect group determination module 430 is used to obtain historical special effect key points of at least one historical video frame according to preset special effect parameters, and determine at least one group of target special effects that matches the preset special effect parameters according to the at least one current special effect key point and the historical special effect key point;

[0131] The target video frame determination module 440 is configured to determine a target video frame corresponding to the current video frame based on the at least one set of target special effects.

[0132] The technical solution of the embodiment of the present disclosure determines at least one target limb key point of the current video frame, and according to the target limb key point, the historical limb key points in each historical video frame corresponding to the target limb key point can be obtained, and then the limb key point group corresponding to the target limb key point is determined, and then special effects are added to each limb key group. According to the at least one target limb key point and each historical limb key point of at least one historical video frame adjacent to the current video frame, at least one current special effect key point is determined, and then the corresponding weight value is determined according to the generation time of each historical video frame, and each limb key point is smoothed to make the connection between each key point softer. According to the preset special effect parameters, the historical special effect key points of at least one historical video frame are obtained, and according to the at least one current special effect key point and the historical special effect key point, at least one group of target special effects matching the preset special effect parameters are determined, and according to the preset special effect parameters and the historical limb key points, multiple groups of target special effects matching the preset special effect parameters can be obtained. Based on the at least one group of target special effects, the target video frame corresponding to the current video frame is determined, and a smoother and softer video special effect is displayed to the user based on the target video frame. The problem that the special effects lines determined according to the motion trajectory of the target subject in the video are relatively stiff, affecting the special effects effect and causing a poor user experience has been solved. The special effects lines added according to the motion trajectory of the subject are made smoother and softer.

[0133] On the basis of the above technical solution, optionally, the target limb key point determination module is used to:

[0134] A limb key point recognition algorithm is used to determine at least one target limb key point in the current video frame; or a pre-trained limb key point recognition model is used to process the current video frame to obtain at least one target limb key point corresponding to the current video frame.

[0135] On the basis of the above technical solution, optionally, the current special effect key point determination module specifically includes:

[0136] A limb key point group determination submodule is used to determine the limb key points belonging to the same part as a limb key point group; wherein the limb key point group includes target limb key points and / or historical limb key points;

[0137] The key point determination submodule to be processed is used to determine the key points to be processed for each limb key point group according to the weight values ​​of the historical limb key points in the current limb key point group and the corresponding key point coordinates;

[0138] The current special effect key point determination submodule is used to determine the current special effect key point corresponding to each limb key point group according to the to-be-processed key points corresponding to each limb key point group and the corresponding target key points.

[0139] On the basis of the above technical solution, optionally, the submodule for determining the key points to be processed specifically includes:

[0140] A weight value determination unit, used to determine the weight value of each historical video frame according to the generation time of each historical video frame; wherein the size of the weight value is inversely proportional to the generation time;

[0141] The key point determination unit to be processed is used to obtain an intermediate value by calculating the product of the key point coordinates of the historical limb key point and the corresponding weight value, and obtain the key point to be processed by accumulating each intermediate value.

[0142] On the basis of the above technical solution, optionally, the target special effect group determination module is used to:

[0143] According to the display length of the special effects bar, historical special effects key points of at least one historical video frame before the current video frame are obtained.

[0144] On the basis of the above technical solution, optionally, the target special effect group determination module is used to:

[0145] By processing the historical special effect key points and the current special effect key points of the same part, multiple groups of target special effects matching the preset special effect parameters are obtained.

[0146] On the basis of the above technical solution, optionally, the target special effect group determination module specifically includes:

[0147] A rendering effect determination submodule, used to determine the current to-be-processed rendering effect of the current special effect key point according to the target effect surface corresponding to the current video frame, and to update the historical rendering effect of the historical special effect key point in the current video frame;

[0148] The target special effect determination submodule is used to determine multiple groups of target special effects that match the preset special effect parameters based on the historical rendering effects of the same part and the current rendering effects to be processed; wherein the total length of each target field in the target effect surface matches the length parameter of the special effect bar in the preset special effect parameters.

[0149] On the basis of the above technical solution, optionally, the rendering effect determination submodule specifically includes:

[0150] A to-be-processed curved surface creation unit, used to create a to-be-processed curved surface corresponding to the target special effect;

[0151] A curved surface determination unit to be used, used to determine the ordinate value corresponding to each abscissa in the curved surface to be processed, and determine the curved surface to be used based on each ordinate value;

[0152] The target effect curved surface determining unit is used to determine the rendering effect of each point in the surface to be used to obtain the target effect curved surface.

[0153] On the basis of the above technical solution, optionally, the rendering effect determination submodule specifically includes:

[0154] A horizontal coordinate determining unit, used to determine the horizontal coordinate corresponding to the target effect surface according to the generation time information of the three-dimensional historical video frames and the current video frame to which the historical special effect key points belong;

[0155] The rendering effect determination unit is used to determine the current rendering effect to be processed of the current video frame according to the horizontal coordinate, and to update the historical rendering effects of the historical special effect key points corresponding to each historical video frame.

[0156] On the basis of the above technical solution, optionally, the rendering effect determination submodule specifically includes:

[0157] A historical target effect curved surface acquisition unit, used to acquire historical target effect curved surfaces of a plurality of historical video frames adjacent to the current video frame and before the current video frame;

[0158] The current pending rendering effect determination unit is used to determine the current pending rendering effect of the current special effect key point according to each historical target effect surface and the target effect surface, and to update the historical rendering effect of the historical special effect key point in the current video frame.

[0159] On the basis of the above technical solution, optionally, the historical target effect surface acquisition unit specifically includes:

[0160] The special effect superposition weight value subunit is used to determine the special effect superposition weight value corresponding to each historical target effect surface;

[0161] A subunit for determining rendering effects to be superimposed, used for obtaining the rendering effects to be superimposed for each pixel point in each group of historical target special effects according to the special effect superposition weight value and the corresponding target effect curved surface;

[0162] The current rendering effect determination subunit is used to determine the current rendering effect to be processed of the current special effect key point according to each rendering effect to be superimposed and the target effect surface of the current video frame, and to update the historical rendering effects of each historical special effect key point in the current video frame.

[0163] On the basis of the above technical solution, optionally, the current rendering effect determination subunit to be processed specifically includes:

[0164] The to-be-processed rendering effect layer is used to determine the historical rendering effects of each historical special effect key point in the current video frame according to the target effect surface of the current video frame, and to determine the to-be-processed rendering effect of the current special effect key point;

[0165] The pending rendering effect update layer is used to superimpose each pending rendering effect on a corresponding pixel point in the current video frame, and to update each historical rendering effect and the pending rendering effect.

[0166] Based on the above technical solution, optionally, the to-be-processed rendering effect update layer is further used to:

[0167] If it is detected that the rendering effect value corresponding to each pixel point is less than a preset rendering effect threshold, the rendering effect value of the pixel point is cleared to obtain a target video frame corresponding to the current video frame.

[0168] On the basis of the above technical solution, optionally, the target video frame determination module is used to:

[0169] Each group of target special effects is added to the corresponding target limb key point to obtain a target video frame corresponding to the current video frame.

[0170] The image processing device provided in the embodiments of the present disclosure can execute the image processing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0171] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0172] Embodiment 5

[0173] Fig.10 This is a schematic diagram of the structure of an electronic device provided by Embodiment 5 of the present disclosure. Fig.10 , which shows an electronic device (eg, Fig.10 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0174] like Fig.10 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0175] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10 All the devices shown are provided. More or fewer devices may be implemented or provided alternatively.

[0176] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the image processing method of the embodiment of the present disclosure are executed.

[0177] Embodiment 6

[0178] Embodiment 6 of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the image processing method provided in the above embodiment is implemented.

[0179] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0180] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0181] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0182] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains at least two Internet Protocol addresses; sends a node evaluation request including the at least two Internet Protocol addresses to a node evaluation device, wherein the node evaluation device selects an Internet Protocol address from the at least two Internet Protocol addresses and returns it; receives the Internet Protocol address returned by the node evaluation device; wherein the obtained Internet Protocol address indicates an edge node in a content distribution network.

[0183] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device:

[0184] Determine at least one target limb key point of the current video frame;

[0185] Determine at least one current special effect key point according to the at least one target limb key point and the historical limb key points of at least one historical video frame adjacent to the current video frame;

[0186] Acquire historical special effect key points of at least one historical video frame according to preset special effect parameters, and determine at least one set of target special effects matching the preset special effect parameters according to the at least one current special effect key point and the historical special effect key points;

[0187] Based on the at least one set of target special effects, a target video frame corresponding to the current video frame is determined.

[0188] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0189] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0190] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0191] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0192] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0193] According to one or more embodiments of the present disclosure, [Example 1] provides an image processing method, the method comprising:

[0194] Determine at least one target limb key point of the current video frame;

[0195] Determine at least one current special effect key point according to the at least one target limb key point and the historical limb key points of at least one historical video frame adjacent to the current video frame;

[0196] Acquire historical special effect key points of at least one historical video frame according to preset special effect parameters, and determine at least one set of target special effects matching the preset special effect parameters according to the at least one current special effect key point and the historical special effect key points;

[0197] According to one or more embodiments of the present disclosure, [Example 2] provides an image processing method, further comprising:

[0198] Using a limb key point recognition algorithm to determine at least one target limb key point in the current video frame; or,

[0199] The current video frame is processed using a pre-trained limb key point recognition model to obtain at least one target limb key point corresponding to the current video frame.

[0200] According to one or more embodiments of the present disclosure, [Example 3] provides an image processing method, further comprising:

[0201] Determine the limb key points belonging to the same part as a limb key point group; wherein the limb key point group includes the target limb key points and / or the historical limb key points;

[0202] For each limb key point group, determine the key points to be processed according to the weight values ​​of the historical limb key points in the current limb key point group and the corresponding key point coordinates;

[0203] According to the key points to be processed and the corresponding target key points corresponding to each limb key point group, the current special effect key points corresponding to each limb key point group are determined.

[0204] According to one or more embodiments of the present disclosure, [Example 4] provides an image processing method, further comprising:

[0205] Determine the weight value of each historical video frame according to the generation time of each historical video frame; wherein the size of the weight value is inversely proportional to the generation time;

[0206] The intermediate value is obtained by calculating the product of the key point coordinates of the historical limb key point and the corresponding weight value, and the key point to be processed is obtained by accumulating the intermediate values.

[0207] According to one or more embodiments of the present disclosure, [Example 5] provides an image processing method, further comprising:

[0208] According to the display length of the special effects bar, historical special effects key points of at least one historical video frame before the current video frame are obtained.

[0209] According to one or more embodiments of the present disclosure, [Example 6] provides an image processing method, further comprising:

[0210] By processing the historical special effect key points and the current special effect key points of the same part, multiple groups of target special effects matching the preset special effect parameters are obtained.

[0211] According to one or more embodiments of the present disclosure, [Example 7] provides an image processing method, further comprising:

[0212] Determine the current rendering effect to be processed of the current special effect key point according to the target effect surface corresponding to the current video frame, and update the historical rendering effect of the historical special effect key point in the current video frame;

[0213] Determining multiple groups of target special effects that match the preset special effect parameters according to the historical rendering effects of the same part and the current rendering effect to be processed;

[0214] Among them, the total length of each target field in the target effect surface matches the length parameter of the special effect bar in the preset special effect parameter.

[0215] According to one or more embodiments of the present disclosure, [Example 8] provides an image processing method, further comprising:

[0216] Creating a surface to be processed corresponding to the target special effect;

[0217] Determine the ordinate value corresponding to each abscissa in the to-be-processed curved surface, and determine the to-be-used curved surface based on each ordinate value;

[0218] Determine the rendering effect of each point in the surface to be used to obtain a target effect surface.

[0219] According to one or more embodiments of the present disclosure, [Example 9] provides an image processing method, further comprising:

[0220] Determine the horizontal coordinate corresponding to the target effect surface according to the generation time information of the three-dimensional historical video frames and the current video frame to which the historical special effect key points belong;

[0221] The current rendering effect to be processed of the current video frame is determined according to the horizontal coordinate, and the historical rendering effects of the historical special effect key points corresponding to each historical video frame are updated.

[0222] According to one or more embodiments of the present disclosure, [Example 10] provides an image processing method, further comprising:

[0223] Acquire historical target effect surfaces of a plurality of historical video frames adjacent to the current video frame and before the current video frame;

[0224] According to each historical target effect surface and the target effect surface, the current to-be-processed rendering effect of the current special effect key point is determined, and the historical rendering effect of the historical special effect key point in the current video frame is updated.

[0225] According to one or more embodiments of the present disclosure, [Example 11] provides an image processing method, further comprising:

[0226] Determine the special effect superposition weight value corresponding to each historical target effect surface;

[0227] According to the special effect superposition weight value and the corresponding target effect surface, the rendering effect to be superimposed of each pixel point in each group of historical target special effects is obtained;

[0228] According to each rendering effect to be superimposed and the target effect surface of the current video frame, the current rendering effect to be processed of the current special effect key point is determined, and the historical rendering effects of the historical special effect key points in the current video frame are updated.

[0229] According to one or more embodiments of the present disclosure, [Example 12] provides an image processing method, further comprising:

[0230] Determine, according to the target effect surface of the current video frame, the historical rendering effect of the historical special effect key point in the current video frame, and determine the to-be-processed rendering effect of the current special effect key point;

[0231] Each rendering effect to be superimposed is superimposed on a corresponding pixel point in the current video frame, and each historical rendering effect and the rendering effect to be processed are updated.

[0232] According to one or more embodiments of the present disclosure, [Example 13] provides an image processing method, further comprising:

[0233] If it is detected that the rendering effect value corresponding to each pixel point is less than a preset rendering effect threshold, the rendering effect value of the pixel point is cleared to obtain a target video frame corresponding to the current video frame.

[0234] According to one or more embodiments of the present disclosure, [Example 14] provides an image processing method, further comprising:

[0235] Each group of target special effects is added to the corresponding target limb key point to obtain a target video frame corresponding to the current video frame.

[0236] According to one or more embodiments of the present disclosure, [Example 15] provides an image processing device, the device comprising:

[0237] A target limb key point determination module, used to determine at least one target limb key point of the current video frame;

[0238] A current special effect key point determination module, used to determine at least one current special effect key point according to the at least one target limb key point and a historical limb key point of at least one historical video frame adjacent to the current video frame;

[0239] A target special effect group determination module, used to obtain historical special effect key points of at least one historical video frame according to preset special effect parameters, and determine at least one group of target special effects matching the preset special effect parameters according to the at least one current special effect key point and the historical special effect key point;

[0240] The target video frame determination module is used to determine the target video frame corresponding to the current video frame based on the at least one set of target special effects.

[0241] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0242] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0243] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image processing method, characterized in that: include: Determine at least one target limb key point of the current video frame; Determine at least one current special effect key point according to the at least one target limb key point and the historical limb key points of at least one historical video frame adjacent to the current video frame; The current special effect key point is determined based on the key point to be processed and the corresponding target key point corresponding to the limb key point group; the key point to be processed is determined based on the weight value and corresponding key point coordinates of the historical limb key point in the limb key point group; the limb key point group includes the target limb key point and / or the historical limb key point; The target key point is the key point of the target limb key point in the current video frame after smoothing; Acquire historical special effect key points of at least one historical video frame according to preset special effect parameters, and determine at least one set of target special effects matching the preset special effect parameters according to the at least one current special effect key point and the historical special effect key points; Based on the at least one set of target special effects, a target video frame corresponding to the current video frame is determined.

2. The method according to claim 1, characterized in that The determining of at least one target limb key point of the current video frame comprises: Using a limb key point recognition algorithm to determine at least one target limb key point in the current video frame; or, The current video frame is processed using a pre-trained limb key point recognition model to obtain at least one target limb key point corresponding to the current video frame.

3. The method according to claim 1, characterized in that The determining at least one current special effect key point according to the at least one target limb key point and the historical limb key points of at least one historical video frame adjacent to the current video frame comprises: Determine the limb key points belonging to the same part as a limb key point group; wherein the limb key point group includes the target limb key points and / or the historical limb key points; For each limb key point group, determine the key points to be processed according to the weight values ​​of the historical limb key points in the current limb key point group and the corresponding key point coordinates; According to the key points to be processed and the corresponding target key points corresponding to each limb key point group, the current special effect key points corresponding to each limb key point group are determined.

4. The method according to claim 3, characterized in that Determining the key points to be processed according to the weight values ​​of the historical limb key points in the current limb key point group and the corresponding key point coordinates includes: Determine the weight value of each historical video frame according to the generation time of each historical video frame; wherein the size of the weight value is inversely proportional to the generation time; The intermediate value is obtained by calculating the product of the key point coordinates of the historical limb key point and the corresponding weight value, and the key point to be processed is obtained by accumulating the intermediate values.

5. The method according to claim 1, characterized in that The preset special effect parameters include the display length of the special effect bar, and obtaining the historical special effect key points of at least one historical video frame according to the preset special effect parameters includes: According to the display length of the special effects bar, historical special effects key points of at least one historical video frame before the current video frame are obtained.

6. The method according to claim 1, characterized in that The determining, according to the at least one current special effect key point and the historical special effect key point, at least one group of target special effects matching the preset special effect parameters comprises: By processing the historical special effect key points and the current special effect key points of the same part, multiple groups of target special effects matching the preset special effect parameters are obtained.

7. The method according to claim 1, characterized in that Determining at least one set of target special effects matching the preset special effect parameters according to the at least one current special effect key point and the historical special effect key point, comprising: Determine the current rendering effect to be processed of the current special effect key point according to the target effect surface corresponding to the current video frame, and update the historical rendering effect of the historical special effect key point in the current video frame; Determining multiple groups of target special effects that match the preset special effect parameters according to the historical rendering effects of the same part and the current rendering effect to be processed; Among them, the total length of each target field in the target effect surface matches the length parameter of the special effect bar in the preset special effect parameter.

8. The method according to claim 7, characterized in that Also includes: Creating a surface to be processed corresponding to the target special effect; Determine the ordinate value corresponding to each abscissa in the to-be-processed curved surface, and determine the to-be-used curved surface based on each ordinate value; Determine the rendering effect of each point in the surface to be used to obtain a target effect surface.

9. The method according to claim 7, characterized in that: The step of determining the current to-be-processed rendering effect of the current special effect key point according to the target effect curved surface corresponding to the current video frame, and updating the historical rendering effect of the historical special effect key point in the current video frame, comprises: Determine the horizontal coordinate corresponding to the target effect surface according to the generation time information of each historical video frame to which the historical special effect key point belongs and the current video frame; The current rendering effect to be processed of the current video frame is determined according to the horizontal coordinate, and the historical rendering effects of the historical special effect key points corresponding to each historical video frame are updated.

10. The method according to claim 7, characterized in that The step of determining the current to-be-processed rendering effect of the current special effect key point according to the target effect curved surface corresponding to the current video frame, and updating the historical rendering effect of the historical special effect key point in the current video frame, comprises: Acquire historical target effect surfaces of a plurality of historical video frames adjacent to the current video frame and before the current video frame; According to each historical target effect surface and the target effect surface, the current to-be-processed rendering effect of the current special effect key point is determined, and the historical rendering effect of the historical special effect key point in the current video frame is updated.

11. The method according to claim 10, characterized in that The step of determining the current to-be-processed rendering effect of the current special effect key point according to each historical target effect curved surface and the target effect curved surface, and updating the historical rendering effect of the historical special effect key point in the current video frame, comprises: Determine the special effect superposition weight value corresponding to each historical target effect surface; According to the special effect superposition weight value and the corresponding target effect surface, the rendering effect to be superimposed of each pixel point in each group of historical target special effects is obtained; According to each rendering effect to be superimposed and the target effect surface of the current video frame, the current rendering effect to be processed of the current special effect key point is determined, and the historical rendering effects of the historical special effect key points in the current video frame are updated.

12. The method according to claim 11, characterized in that The step of determining the current rendering effect to be processed of the current special effect key point according to each rendering effect to be superimposed and the target effect surface of the current video frame, and updating the historical rendering effect of the historical special effect key point in the current video frame, comprises: Determine, according to the target effect surface of the current video frame, the historical rendering effect of the historical special effect key point in the current video frame, and determine the to-be-processed rendering effect of the current special effect key point; Each rendering effect to be superimposed is superimposed on a corresponding pixel point in the current video frame, and each historical rendering effect and the rendering effect to be processed are updated.

13. The method according to claim 12, characterized in that Also includes: If it is detected that the rendering effect value corresponding to each pixel point is less than a preset rendering effect threshold, the rendering effect value of the pixel point is cleared to obtain a target video frame corresponding to the current video frame.

14. The method according to any one of claims 1 to 13, characterized in that: The determining, based on the at least one set of target special effects, a target video frame corresponding to the current video frame comprises: Each group of target special effects is added to the corresponding target limb key point to obtain a target video frame corresponding to the current video frame.

15. An image processing device, characterized in that: include: A target limb key point determination module, used to determine at least one target limb key point of the current video frame; A current special effect key point determination module, used to determine at least one current special effect key point according to the at least one target limb key point and a historical limb key point of at least one historical video frame adjacent to the current video frame; The current special effect key point is determined based on the key point to be processed and the corresponding target key point corresponding to the limb key point group; the key point to be processed is determined based on the weight value and corresponding key point coordinates of the historical limb key point in the limb key point group; the limb key point group includes the target limb key point and / or the historical limb key point; The target key point is the key point of the target limb key point in the current video frame after smoothing; A target special effect group determination module, used to obtain historical special effect key points of at least one historical video frame according to preset special effect parameters, and determine at least one group of target special effects matching the preset special effect parameters according to the at least one current special effect key point and the historical special effect key point; The target video frame determination module is used to determine the target video frame corresponding to the current video frame based on the at least one set of target special effects.

16. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1 to 14.

17. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the image processing method according to any one of claims 1 to 14 when executed by a computer processor.

Images