Image processing method, device, electronic device and storage medium
By deforming a preset open lip makeup image and overlaying it onto the closed lip area, the problem of lip color flickering in the video was solved, achieving color consistency and improved user experience.
Patent Information
- Application Number
- CN202111276359.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-10-29
AI Technical Summary
When mapping lip makeup in a video, inconsistencies in color or transparency between open and closed lip makeup can cause color flickering, negatively impacting the user experience.
By acquiring video frames of the closed lip area, the lip region is determined, and a preset open lip makeup image is deformed and superimposed onto the closed lip area to ensure color consistency.
This avoids color flickering in the lip area of the video, improving the user experience.
Smart Images

Figure CN113920007B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art
[0002] With the popularity of smartphones and the development of artificial intelligence technology, people are increasingly fond of using filters, makeup, stickers, etc. to beautify selfies and record their beautiful lives. With the popularity of short videos, people will also edit short videos to record their lives. In other words, the application of makeup has expanded from a picture to a video.
[0003] Taking the application of makeup to lips in a video as an example, it is necessary to determine whether the lip area in each video frame of the video is in an open or closed state. If it is in an open state, the drawn open lip makeup is mapped to the lip area in the video frame; if it is in a closed state, the drawn closed lip makeup is mapped to the lip area in the video frame. Since both open and closed lip makeup need to be drawn in advance, when the color or transparency of the open and closed lip makeup is inconsistent, the lip area in the mapped video may have a sudden color change, resulting in color flickering, which affects the user experience. Summary of the invention
[0004] The present disclosure provides an image processing method, an apparatus, an electronic device and a storage medium, which can solve the problem of color flickering in the lip area after video mapping in the prior art.
[0005] The technical solution of the embodiment of the present disclosure is as follows:
[0006] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, which may include: obtaining a video frame to be processed including a lip area in a closed state; determining a first upper lip area and a first lower lip area of the lip area in the video frame to be processed; obtaining a preset open lip makeup image including a second upper lip area and a second lower lip area; deforming the second upper lip area and the second lower lip area, wherein the shape of the deformed second upper lip area matches the shape of the first upper lip area, and the shape of the deformed second lower lip area matches the shape of the first lower lip area; superimposing the deformed second upper lip area on the first upper lip area, and superimposing the deformed second lower lip area on the first lower lip area.
[0007] Optionally, the image processing method may also include: determining multiple second key points in the second upper lip area that respectively correspond to multiple first key points of the video frame to be processed, and determining multiple fourth key points in the second lower lip area that respectively correspond to multiple third key points of the video frame to be processed; multiple first key points are used to characterize the first upper lip area, multiple second key points are used to characterize the second upper lip area, multiple third key points are used to characterize the first lower lip area, and multiple fourth key points are used to characterize the second lower lip area; the above-mentioned "superimposing the deformed second upper lip area on the first upper lip area, and superimposing the deformed second lower lip area on the first lower lip area" specifically includes: according to multiple first key points and multiple second key points, superimposing the deformed second upper lip area on the first upper lip area, and according to multiple third key points and multiple fourth key points, superimposing the deformed second lower lip area on the first lower lip area.
[0008] Optionally, the above-mentioned "superimposing the deformed second upper lip area on the first upper lip area according to multiple first key points and multiple second key points, and superimposing the deformed second lower lip area on the first lower lip area according to multiple third key points and multiple fourth key points" includes: according to the position correspondence between multiple first key points and multiple second key points, superimposing multiple second key points on corresponding first key points respectively, so that the deformed second upper lip area covers the first upper lip area; according to the position correspondence between multiple third key points and multiple fourth key points, superimposing multiple fourth key points on corresponding third key points respectively, so that the deformed second lower lip area covers the first lower lip area.
[0009] Optionally, when there is an overlapping area between the first upper lip area and the first lower lip area in the superimposed video frames, the method further includes: performing color fading processing on the overlapping area.
[0010] Optionally, the method also includes: when the first upper lip area completely covers the first lower lip area, deleting the second lower lip area and superimposing the deformed second upper lip area on the first upper lip area; when the first lower lip area completely covers the first upper lip area, deleting the second upper lip area and superimposing the deformed second lower lip area on the second lip area.
[0011] Optionally, when the first upper lip area in the video frame to be processed covers part of the first lower lip area, or the first lower lip area covers part of the first upper lip area, the above-mentioned "determining the first upper lip area and the first lower lip area of the lip area in the video frame to be processed" includes: determining the first lip line and the second lip line of the covered area, the first lip line being the lip line of the covering lip area and connected with the covered lip area, and the second lip line being the lip line that is not blocked by the covered lip area; when the closed area formed by the first lip line and the second lip line is located above the covering lip area, the closed area is taken as the first upper lip area, and the covering lip area is taken as the first lower lip area; when the closed area formed by the first lip line and the second lip line is located below the covering lip area, the closed area is taken as the first lower lip area, and the covering area is taken as the first upper lip area.
[0012] Optionally, when the first upper lip area covers part of the first lower lip area, the above-mentioned "deforming the second upper lip area and the second lower lip area" includes: deforming the second upper lip area so that the shape of the second upper lip area after the deformation matches the shape of the first upper lip area, and deforming the second lower lip area so that the shape of the second lower lip area after the deformation matches the shape of the closed area; the above-mentioned "superimposing the deformed second upper lip area on the first upper lip area, and superimposing the deformed second lower lip area on the first lower lip area" includes: superimposing the second upper lip area on the first upper lip area, and superimposing the deformed second lower lip area on the closed area domain; when the first lower lip area covers part of the first upper lip area, the above-mentioned "deforming the second upper lip area and the second lower lip area" includes: deforming the second upper lip area so that the shape of the second upper lip area after the deformation matches the shape of the closed area, and deforming the second lower lip area so that the shape of the second lower lip area after the deformation matches the shape of the first lower lip area; the above-mentioned "superimposing the deformed second upper lip area on the first upper lip area, and superimposing the deformed second lower lip area on the first lower lip area" includes: superimposing the second upper lip area on the closed area, and superimposing the deformed second lower lip area on the first lower lip area.
[0013] According to a second aspect of an embodiment of the present disclosure, an image processing device is provided, which may include: an acquisition unit, a determination unit, and a processing unit; the acquisition unit is used to acquire a video frame to be processed including a lip area in a closed state; the determination unit is used to determine a first upper lip area and a first lower lip area of the lip area in the video frame to be processed; the acquisition unit is also used to acquire a preset open lip makeup image including a second upper lip area and a second lower lip area; the processing unit is used to deform the second upper lip area and the second lower lip area, wherein the shape of the deformed second upper lip area matches the shape of the first upper lip area, and the shape of the deformed second lower lip area matches the shape of the first lower lip area; the processing unit is also used to superimpose the deformed second upper lip area on the first upper lip area, and superimpose the deformed second lower lip area on the first lower lip area.
[0014] Optionally, the processing unit is also used to determine multiple second key points in the second upper lip area that respectively correspond to multiple first key points of the video frame to be processed, and determine multiple fourth key points in the second lower lip area that respectively correspond to multiple third key points of the video frame to be processed; multiple first key points are used to characterize the first upper lip area, multiple second key points are used to characterize the second upper lip area, multiple third key points are used to characterize the first lower lip area, and multiple fourth key points are used to characterize the second lower lip area; the processing unit is specifically used to superimpose the deformed second upper lip area on the first upper lip area according to the multiple first key points and the multiple second key points, and superimpose the deformed second lower lip area on the first lower lip area according to the multiple third key points and the multiple fourth key points.
[0015] Optionally, the processing unit is specifically used to superimpose multiple first key points onto corresponding second key points according to the position correspondence between multiple first key points and multiple second key points, so that the first upper lip area is covered with the second upper lip area after the deformation processing; and superimpose multiple third key points onto corresponding fourth key points according to the position correspondence between multiple third key points and multiple fourth key points, so that the second lower lip area after the deformation processing covers the first lower lip area.
[0016] Optionally, when there is an overlapping area between the first upper lip area and the first lower lip area in the superimposed video frames, the processing unit is further configured to perform color fading processing on the overlapping area.
[0017] Optionally, when the first upper lip area completely covers the first lower lip area, the processing unit is also used to delete the second lower lip area and superimpose the deformed second upper lip area on the first upper lip area; when the first lower lip area completely covers the first upper lip area, the processing unit is also used to delete the second upper lip area and superimpose the deformed second lower lip area on the second lip area.
[0018] Optionally, when the first upper lip area covers part of the first lower lip area, or the first lower lip area covers part of the first upper lip area, the determination unit is specifically used to determine the first lip line and the second lip line of the covered area, the first lip line is the lip line of the covering lip area and connected with the covered lip area, and the second lip line is the lip line of the covered lip area that is not blocked; when the closed area formed by the first lip line and the second lip line is located above the covering lip area, the closed area is taken as the first upper lip area, and the covering lip area is taken as the first lower lip area; when the closed area formed by the first lip line and the second lip line is located below the covering lip area, the closed area is taken as the first lower lip area, and the covering area is taken as the first upper lip area.
[0019] Optionally, when the first upper lip area covers a portion of the first lower lip area, the processing unit is specifically used to deform the second lower lip area so that the shape of the deformed second lower lip area matches the shape of the closed area; superimpose the second upper lip area on the first upper lip area, and superimpose the deformed second lower lip area on the closed area; when the first lower lip area covers a portion of the first upper lip area, the processing unit is specifically used to deform the second upper lip area so that the shape of the deformed second upper lip area matches the shape of the closed area; superimpose the shape of the deformed second upper lip area on the first upper lip area, and superimpose the second lower lip area on the first lower lip area.
[0020] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, which may include: a processor and a memory for storing processor executable instructions; wherein the processor is configured to execute the instructions to implement any one of the optional image processing methods in the above-mentioned first aspect.
[0021] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which instructions are stored. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any one of the optional image processing methods in the above-mentioned first aspect.
[0022] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the image processing method as described in any optional implementation of the first aspect.
[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.
[0024] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0025] Based on any of the above aspects, in the present disclosure, after acquiring a video frame to be processed including a closed lip area, the image processing device can deform the upper lip area of the preset open lip makeup image and superimpose it on the upper lip area of the lip area in the closed state in the video frame to be processed, and deform the lower lip area of the lip makeup image and superimpose it on the lower lip area of the lip area in the closed state in the video frame to be processed. In the technical solution provided by the present disclosure, a preset open lip makeup image can be mapped to lips in a closed state, and can also be mapped to lips in an open state. Since the lips in a closed state and lips in an open state use the same open lip makeup image, the color of the mapped lip area can be kept consistent, thereby avoiding the phenomenon of color flickering of the lips in the processed video. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0027] Figure 1 A schematic diagram of a lip makeup image provided by an embodiment of the present disclosure is shown;
[0028] Figure 2 A schematic diagram of a lip region of a video frame provided by an embodiment of the present disclosure is shown;
[0029] Figure 3 A schematic diagram of the structure of a communication system provided by an embodiment of the present disclosure is shown;
[0030] Figure 4 A schematic diagram of a flow chart of an image processing method provided by an embodiment of the present disclosure is shown;
[0031] Figure 5 A schematic diagram showing a flow chart of another image processing method provided by an embodiment of the present disclosure;
[0032] Figure 6A schematic diagram showing key points of a lip region of a video frame to be processed provided by an embodiment of the present disclosure;
[0033] Figure 7 A schematic diagram showing key points of an open lip makeup image provided by an embodiment of the present disclosure;
[0034] Figure 8 A schematic diagram showing a flow chart of another image processing method provided by an embodiment of the present disclosure;
[0035] Fig. 9 A schematic diagram showing a flow chart of another image processing method provided by an embodiment of the present disclosure;
[0036] Fig.10 A schematic diagram showing a flow chart of another image processing method provided by an embodiment of the present disclosure;
[0037] Fig.11 A schematic diagram showing a flow chart of another image processing method provided by an embodiment of the present disclosure;
[0038] Fig.12 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure is shown;
[0039] Fig.13 A schematic structural diagram of another image processing device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0040] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0042] It should also be understood that the term “comprising” indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or components.
[0043] The data involved in the present disclosure may be data authorized by the user or fully authorized by all parties.
[0044] As described in the background technology, with the continuous development of short video applications, more and more people like to share their life preferences through short videos. At the same time, in order to increase the beauty of people in short videos, people will apply beauty makeup to short videos.
[0045] For example, in order to enhance the lip color of a person in a video, the drawn lip makeup image can be mapped to the lips of the person in the video. Generally speaking, developers of short video applications need to draw open lip makeup and closed lip makeup first. Then, the image processing device (which can be a terminal, such as a mobile phone, etc.) identifies whether the lips in the video are in an open state or a closed state. If the lips in the video are in an open state, the open lip makeup is mapped to the open lips of the video; if the lips in the video are in a closed state, the closed lip makeup is mapped to the closed lips of the video. In this way, the effect of beautifying the lips in the video can be achieved.
[0046] In one example, Figure 1 In the figure, a is the closed lip makeup, and b is the open lip makeup. Figure 2 In the figure, a represents the lips in the closed state in the video frame to be processed, and b represents the lips in the closed state in the video frame to be processed.
[0047] The image processing device first obtains the lip state in the video frame to be processed. If the image processing device detects that the lips in the video frame to be processed are as follows Figure 2 The lips shown in a in the figure are Figure 1 The closed lip makeup of a in is mapped to Figure 2 The lips shown in a in the figure are obtained by mapping the lips; if the image processing device detects that the lips in the video frame to be processed are as shown in a Figure 2 The lips shown in b in the figure are Figure 1 The open lip makeup of b in is mapped to Figure 2 The lips shown in b are obtained after mapping.
[0048] Based on the above mapping process, the image processing device can map the lips in multiple video frames to be processed to obtain a video with special effects.
[0049] However, if Figure 1 A's lip makeup and Figure 1 During the drawing process of the lip makeup of b, the color or transparency of the lip makeup is inconsistent, and the lips in the video may appear to flicker. For example, the lips in the first video frame are in a closed state, and the lips in the second video frame adjacent to the first video frame are in an open state. Since the lip color in the mapped first video frame is inconsistent with the lip color in the mapped second video frame, when the video is played, the color flickering phenomenon will occur, affecting the user experience.
[0050] In view of this, the present disclosure provides an image processing method, which includes: obtaining a video frame to be processed including a lip area in a closed state; determining a first upper lip area and a first lower lip area of the lip area in the video frame to be processed; obtaining a preset open lip makeup image including a second upper lip area and a second lower lip area; deforming the second upper lip area and the second lower lip area, wherein the shape of the deformed second upper lip area matches the shape of the first upper lip area, and the shape of the deformed second lower lip area matches the shape of the first lower lip area; superimposing the deformed second upper lip area on the first upper lip area, and superimposing the deformed second lower lip area on the first lower lip area. Since the lip areas in different states in the video frame can be superimposed using the same lip makeup image, the color flickering phenomenon in the superimposed video frame can be avoided.
[0051] The following is an exemplary description of the image processing method provided by the embodiment of the present disclosure in conjunction with the accompanying drawings:
[0052] Figure 3 A communication system schematic diagram provided in an embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the communication system may include: a server 310 and a terminal 320. The server 310 may establish a connection with the terminal 320 via a wired network or a wireless network.
[0053] Among them, the server 310 can be used to store the drawn lip makeup images and send the lip makeup images to the terminal 320. The server 310 can be a multimedia resource platform server, which can be used to store and process multimedia resources. For example, the multimedia resource service platform can be a short-time frequency service platform, etc. The multimedia resources provided by the short video service platform can be short videos, etc. The present disclosure does not limit the specific type of the multimedia resource service platform. The data involved in the present disclosure can be data authorized by the user or fully authorized by all parties.
[0054] In the present disclosure, the server 310 may include a short video application service platform, each short video application service platform uniquely corresponds to an application, and the application is installed on the terminal. The server 310 is mainly used to store the relevant business data of the application installed on the terminal 320, and send the business data to the terminal.
[0055] In some embodiments, the server may be a single server, or a server cluster composed of multiple servers. In some implementations, the server cluster may also be a distributed cluster. The present disclosure does not limit the specific implementation of the server 310.
[0056] The terminal 320 may be a mobile phone, tablet computer, desktop, laptop, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), augmented reality (AR)\virtual reality (VR) device, etc., which can install and use content community applications (such as Kuaishou), and the present disclosure does not impose any special restrictions on the specific form of the terminal. It can interact with the user through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device.
[0057] Optionally, the above Figure 3 In the communication system shown, the server 310 may be connected to at least one terminal 320. The present disclosure does not limit the number and type of the terminal 320.
[0058] The image processing method provided by the embodiment of the present disclosure can be applied to the aforementioned Figure 3 The terminal or server shown.
[0059] It should be noted that the image processing method provided by the present disclosure is executed by an image processing device. The image processing device may be a server, or a chip or system on chip in a server, etc. Alternatively, the image processing device may be a terminal, or a chip or system on chip in a terminal, etc., without limitation. For example, the image processing device may be Figure 3 The server 310 or terminal 320 is not limited.
[0060] like Figure 4 As shown, the image processing and transmission method provided by the embodiment of the present disclosure may include:
[0061] S401: Obtain a video frame to be processed.
[0062] The video frame to be processed may include a lip area, and the lips are in a closed state.
[0063] The apparatus for executing S401 may be the aforementioned image processing apparatus. Different image processing apparatuses obtain the video frame to be processed in different ways, which are described below.
[0064] 1. When the image processing device is a terminal, the image processing device can obtain a video to be processed in response to a user's video shooting operation, and use a preset tool to extract multiple video frames in the processed video. The preset tool can be used to decompose the video into multiple continuous video frames.
[0065] 2. When the image processing device is a server, the image processing device can obtain the video to be processed through the terminal. For example, the terminal sends a first request message including the video to be processed to the image processing device. The first request message is used to request beauty processing of lips in the video to be processed. The image processing device receives the first request message from the terminal and decomposes the video to be processed in the first request message into multiple continuous video frames.
[0066] After the image processing device acquires a plurality of continuous video frames, it can identify the plurality of continuous video frames and use the video frame including the lips as the video frame to be processed.
[0067] S402: Determine a first upper lip region and a first lower lip region of the lip region in the video frame to be processed.
[0068] The first upper lip region and the second lower lip region may form lips.
[0069] In a possible implementation, the first upper lip region and the second upper lip region of the lips in the video frame to be processed can be determined by using a preset face recognition algorithm. The preset face recognition algorithm can refer to the prior art and will not be described in detail.
[0070] S403: Obtain a preset open lip makeup image.
[0071] The preset open lip makeup image may include a second upper lip area and a second lower lip area.
[0072] For example, the preset open lip makeup image can be Figure 1 The lip makeup image shown in b.
[0073] In one example, the lip makeup image may be pre-stored in the above-mentioned image processing device, or may be obtained by the image processing device from other devices. For example, when the image processing device is a terminal, the image processing device may obtain the open lip makeup image from a server.
[0074] S404: Perform deformation processing on the second upper lip region and the second lower lip region.
[0075] The shape of the second upper lip region after the deformation process matches the shape of the first upper lip region, and the shape of the second lower lip region after the deformation process matches the shape of the first lower lip region.
[0076] S405: superimpose the second upper lip region after the deformation processing on the first upper lip region, and superimpose the second lower lip region after the deformation processing on the first lower lip region.
[0077] For example, you can Figure 1The upper lip area of the open lip makeup in b is deformed and superimposed on Figure 2 The upper lip area of the lips in the closed state in a, and Figure 1 The lower lip area of the open lip makeup in b is deformed and superimposed on Figure 2 The lower lip area in closed state in a.
[0078] It should be pointed out that when the lips in the video frame to be processed are in an open state, the upper lip area of the open lip makeup image can be superimposed on the upper lip area of the lip area of the video frame to be processed, and the lower lip area of the open lip makeup image can be superimposed on the lower lip area of the lip area of the video frame to be processed.
[0079] For example, you can Figure 1 The upper lip area of the open lip makeup in b is superimposed on Figure 2 The upper lip area of the lips in the open state in b, and Figure 1 The lower lip area in the open state in b is superimposed on Figure 2 The lower lip area in the open state in b.
[0080] The technical solution provided by the above-mentioned embodiments brings at least the following beneficial effects: As can be seen from S401-S405, in the present disclosure, in the present disclosure, after acquiring a video frame to be processed including a closed lip area, the image processing device can deform the upper lip area of the preset open lip makeup image and superimpose it on the upper lip area of the lip area in the closed state in the video frame to be processed, and deform the lower lip area of the lip makeup image and superimpose it on the lower lip area of the lip area in the closed state in the video frame to be processed. In the technical solution provided by the present disclosure, a preset open lip makeup image can be superimposed on the lips in the closed state, and can also be superimposed on the lips in the open state. Since the lips in the closed state and the lips in the open state use the same open lip makeup image, the color of the superimposed lip area can be kept consistent, thereby avoiding the phenomenon of color flickering of the lips in the processed video.
[0081] In one embodiment, in combination Figure 4 ,like Figure 5 As shown, the method may also include:
[0082] S501, determining a plurality of second key points in the second upper lip region that respectively correspond to a plurality of first key points of the video frame to be processed, and determining a plurality of fourth key points in the second lower lip region that respectively correspond to a plurality of third key points of the video frame to be processed.
[0083] Among them, a plurality of first key points can be used to characterize the first upper lip region, a plurality of second key points can be used to characterize the second upper lip region, a plurality of third key points can be used to characterize the first lower lip region, and a plurality of fourth key points can be used to characterize the second lower lip region.
[0084] In one example, if the lips in the video frame to be processed are Figure 2 As shown in a in , then Figure 6 As shown in a, multiple first key points can be A1A2A3A4A5A6A7A 20 A 18 A 17 A 14 A 13 Among them, the connection lines between three adjacent key points among the multiple first key points can form a geometric figure as a triangle. For example, A1, A2, A 13 Can form triangle A1A2A 13 The plurality of geometric figures formed by the plurality of first key points may be spliced into a first upper lip region.
[0085] like Figure 6 As shown in a in FIG. 1 , the third key points of the lips in the video frame to be processed may be A1A 12 A 11 A 10 A9A8A7A 20 A 18 A 16 A 15 A 13 Among them, the lines between three adjacent key points among the plurality of third key points can form a triangle. For example, A1, A 12 , A 13 Can form triangle A1A 12 A 13 The plurality of geometric figures formed by the plurality of third key points may be spliced into a first lower lip region.
[0086] In another example, if the lips in the video frame to be processed are Figure 2 As shown in b in , Figure 6 As shown in b, the multiple first key points can be B1B2B3B4B5B6B7B 20 B 18 B 17 B 14 B 13 Among them, the connection lines between three adjacent key points among the multiple first key points can form a geometric figure as a triangle. For example, B1, B2, B 13 Can form triangle A1A2A 13The plurality of geometric figures formed by the plurality of first key points may be spliced into a first upper lip region.
[0087] like Figure 6 As shown in b in FIG. 1 , the third key points of the lips in the video frame to be processed may be B1B 12 B 11 B 10 B9B8B7B 20 B 18 B 16 B 15 B 13 Among them, the lines between three adjacent key points among the plurality of third key points can form a triangle. For example, B1, B 12 , B 13 Can form triangle B1B 12 B 13 The plurality of geometric figures formed by the plurality of third key points may be spliced into a first lower lip region.
[0088] Correspondingly, the open lip makeup image is Figure 1 If b in Figure 7 As shown, the second key points of the open lip makeup image may be: b1b2b3b4b5b6b7b 20 b 19 b 16 b 15 b 13 , the fourth key points of the open lip makeup image can be: b1b 12 b 11 b10b9b8b7b 20 b 18 b 17 b 14 b 13 .
[0089] It should be noted that the above Figure 7 The upper lip region and the lower lip region of the open lip makeup image in FIG are both deformed.
[0090] In a possible implementation, a preset texture rendering algorithm can be used to render multiple second key points and multiple fourth key points of the open lip makeup image. The preset texture rendering algorithm can be set as needed, for example, it can be an OpenGL texture rendering algorithm.
[0091] In the above S405, superimposing the second upper lip region after deformation processing on the first upper lip region, and superimposing the second lower lip region after deformation processing on the first lower lip region may specifically include S502.
[0092] S502. According to the plurality of first key points and the plurality of second key points, the deformed second upper lip region is superimposed on the first upper lip region of the video frame to be processed, and according to the plurality of third key points and the plurality of fourth key points, the deformed second lower lip region is superimposed on the first lower lip region.
[0093] For example, combined with Figure 7 Multiple key points of the lip makeup image and Figure 6 The above-mentioned “according to the multiple first key points and the multiple second key points, superimposing the deformed second upper lip region to the first upper lip region of the video frame to be processed” may refer to Figure 7 The third key point of the medium-open lip makeup image b1b2b3b4b5b6b7b 20 b 19 b 16 b 15 b 13 and Figure 6 A plurality of first key points A1A2A3A4A5A6A7A in the upper lip region of a 20 A 18 A 17 A 14 A 13 The position correspondence between the upper lip regions is achieved by superimposing the deformed second upper lip region onto the first upper lip region.
[0094] The above-mentioned “according to the plurality of third key points and the plurality of fourth key points, superimposing the second lower lip region after the deformation processing onto the first lower lip region” may refer to Figure 7 Multiple fourth key points b1b of the open lip makeup image 12 b11b 10 b9b8b7b 20 b 18 b 17 b 14 b 13 and Figure 6 The second key point A1A in the upper lip area of a 12 A 11 A 10 A9A8A7A 20 A 18 A 16 A15A 13 The position correspondence between the two is based on the deformation processing of the second lower lip area and the deformation processing of the second lower lip area is superimposed on the first lower lip area.
[0095] The technical solution provided by the above embodiment brings at least the following beneficial effects: As can be seen from S501 to S502, multiple key points of the open lip makeup image can be determined according to multiple key points used to characterize the lip area in the video frame to be processed. Since the multiple key points of the video frame to be processed correspond to the multiple key points of the open lip makeup image one-to-one, the open lip makeup image can be accurately and completely covered to the lip area in the video frame to be processed according to the multiple key points of the video frame to be processed and the multiple key points of the open lip makeup image.
[0096] In one embodiment, in the above S502, according to the plurality of first key points and the plurality of second key points, the second upper lip region after deformation processing is superimposed on the first upper lip region of the video frame to be processed, and according to the plurality of third key points and the plurality of fourth key points, the second lower lip region after deformation processing is superimposed on the first lower lip region, including:
[0097] According to the position correspondence between multiple first key points and multiple second key points, the multiple second key points are respectively superimposed on the corresponding first key points so that the second upper lip area after deformation covers the first upper lip area; according to the position correspondence between multiple third key points and multiple fourth key points, the multiple fourth key points are superimposed on the multiple third key points so that the second lower lip area after deformation covers the first lower lip area.
[0098] The technical solution provided by the above embodiment brings at least the following beneficial effects: As can be seen from the above, since the multiple key points of the video frame to be processed and the multiple key points of the open lip makeup image are one-to-one corresponding in position, the multiple key points of the open lip makeup image can be superimposed on the multiple key points of the video frame to be processed according to the positional correspondence between the key points, so that the open lip makeup image can completely cover the lip area of the video frame to be processed.
[0099] In one embodiment, in combination Figure 5 ,like Figure 8 As shown, when an overlapping area appears between the first upper lip area of the superimposed video frame and the first lower lip area of the superimposed video frame, the method may further include S801.
[0100] S801, performing color reduction processing on the overlapping area.
[0101] The overlapping area may refer to an area in the superimposed video frame to be processed whose color is darker than that of other areas.
[0102] For example, in combination with the first example of S502 above, it can be seen from the description of S502 that the key point A of the lips in the superimposed video frame 13 A 14 A17 A 18 A 20 A 19 A 16 A 15 The constructed area is superimposed twice during superposition, thus causing the lips in that area to be darker in color.
[0103] In a possible implementation, the color of the adjacent area of the overlapping area may be used as a reference color to perform a color fading process on the overlapping area, and the color of the overlapping area after the process may be the same as or similar to the color of the adjacent area.
[0104] The technical solution provided by the above embodiment brings at least the following beneficial effects: It can be seen from S801 that when the lips of the video frame to be processed are in a closed state, in order to make the effect of the lip area of the superimposed video frame better, the overlapping area in the superimposed lip area can be color-faded to make the lips in the processed video frame more natural and beautiful.
[0105] In one embodiment, in combination Figure 4 ,like Fig. 9 As shown, when the first upper lip area completely covers the first lower lip area, the method may further include:
[0106] S901. Delete the second upper lip region, and superimpose the deformed second lower lip region onto the first lower lip region.
[0107] When the first lower lip area completely covers the first upper lip area, the method may further include:
[0108] S902: Delete the second lower lip region, and superimpose the deformed second upper lip region onto the first lower lip region.
[0109] In another possible embodiment, when the first upper lip area completely covers the first lower lip area, or when the first upper lip area completely covers the first lower lip area, the method may further include: dividing the identified lip area into a first lip area and a second lip area, wherein the first lip area is located above the second lip area, and the first lip area is used as the first upper lip area, and the second lip area is used as the first lower lip area.
[0110] In the above S404, the method of superimposing the second upper lip area in the lip makeup image on the first upper lip area of the video frame to be processed, and superimposing the second lower lip area in the lip makeup image on the first lower lip area of the video frame to be processed may specifically include: superimposing the first upper lip area on the first lip area, and superimposing the first lower lip area on the second lip area.
[0111] The specific implementation process may refer to the above step S405 and will not be described in detail.
[0112] The technical solution provided by the above embodiment brings at least the following beneficial effects: It can be seen from S901 to S902 that when the identified video frame to be processed only includes the upper lip or the lower lip, the corresponding lip area of the open lip makeup image can be deformed and superimposed on the recognizable lip area, so that the incomplete lips in the superimposed video frame can also have a beauty effect.
[0113] In one embodiment, in combination Figure 4 ,like Fig.10 As shown, when the first upper lip area in the to-be-processed video frame covers a portion of the first lower lip area, or the first lower lip area covers a portion of the first upper lip area, in the above S402, determining the first upper lip area and the first lower lip area of the lip area in the to-be-processed video frame may specifically include:
[0114] S1001. Determine a first lip line and a second lip line of a covered lip area.
[0115] The first lip line is a lip line that covers the lip area and connects with the covered lip area, and the second lip line is a lip line that is not blocked by the covered lip area.
[0116] For example, when a portion of the first lower lip region is covered by the first upper lip region, the first lip line may be the lower lip line of the first upper lip, and the second lip line may be the lower lip line of the first lower lip. When a portion of the first upper lip region is covered by the first lower lip region, the first lip line may be the upper lip line of the first lower lip, and the second lip line may be the upper lip line of the first upper lip.
[0117] S1002. When the closed area formed by the first lip line and the second lip line is located above the covered lip area, the closed area is used as the first upper lip area, and the covered lip area is used as the first lower lip area.
[0118] S1003. When the closed area formed by the first lip line and the second lip line is located below the covered lip area, the closed area is used as the first lower lip area, and the covered lip area is used as the first upper lip area.
[0119] The technical solution provided by the above embodiment brings at least the following beneficial effects: It can be seen from S1001 to S1003 that when a lip area in a video frame to be processed is partially covered by another lip area, the exposed part of the covered area can be regarded as a complete lip area, and then the open lip makeup image can be accurately and completely covered to the entire lip area in the video frame to be processed.
[0120] In one embodiment, in combination Fig.10 ,like Fig.11 As shown, when the first upper lip area covers a portion of the first lower lip area, in the above S404, the second upper lip area and the second lower lip area are deformed, including:
[0121] S1101. Perform deformation processing on the second upper lip area so that the shape of the second upper lip area after the deformation processing matches the shape of the first upper lip area, and perform deformation processing on the second lower lip area so that the shape of the second lower lip area after the deformation processing matches the shape of the closed area.
[0122] In the above S405, superimposing the second upper lip region after deformation processing onto the first upper lip region, and superimposing the second lower lip region after deformation processing onto the first lower lip region, comprises:
[0123] S1102: superimpose the second upper lip region after deformation processing onto the first upper lip region, and superimpose the second lower lip region after deformation processing onto the closed region.
[0124] When the first lower lip region covers a portion of the first upper lip region, in the above S404, the second upper lip region and the second lower lip region are deformed, including:
[0125] S1103. Perform deformation processing on the second upper lip area so that the shape of the second upper lip area after the deformation processing matches the shape of the closed area, and perform deformation processing on the second lower lip area so that the shape of the second lower lip area after the deformation processing matches the shape of the first lower lip area.
[0126] In the above S405, superimposing the second upper lip region after deformation processing onto the first upper lip region, and superimposing the second lower lip region after deformation processing onto the first lower lip region, comprises:
[0127] S1104: superimpose the second upper lip region after deformation processing onto the closed region, and superimpose the second lower lip region after deformation processing onto the first lower lip region.
[0128] The technical solution provided by the above embodiment brings at least the following beneficial effects: From S1011 to S1014, it can be known that when a lip area in a video frame to be processed is partially covered by another lip area, the exposed part of the covered area can be regarded as a complete lip area, and the open lip makeup image can be deformed so that the deformed open lip makeup image matches the shape of the lip area in the video frame. In this way, the special effects of the lip area of the superimposed video frame can be more realistic.
[0129] It can be understood that, in actual implementation, the image processing device described in the embodiment of the present disclosure may include one or more hardware structures and / or software modules for implementing the aforementioned corresponding image processing method, and these execution hardware structures and / or software modules may constitute an electronic device. It should be easily appreciated by those skilled in the art that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present disclosure.
[0130] Based on such understanding, an embodiment of the present disclosure also provides an image processing device. Fig.12 FIG. 2 shows a schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure. Fig.12 As shown, the image processing device may include: an acquisition unit 1201 , a determination unit 1202 , and a processing unit 1203 .
[0131] The acquisition unit 1201 is used to acquire a video frame to be processed including a lip area in a closed state; the determination unit 1202 is used to determine a first upper lip area and a first lower lip area of the lip area in the video frame to be processed; the acquisition unit 1201 is also used to acquire a preset open lip makeup image, which includes a second upper lip area and a second lower lip area; the processing unit 1203 is used to deform the second upper lip area and the second lower lip area, so that the shape of the deformed second upper lip area matches the shape of the first upper lip area, and the shape of the deformed second lower lip area matches the shape of the first lower lip area. The processing unit 1203 is also used to superimpose the deformed second upper lip area on the first upper lip area, and superimpose the deformed second lower lip area on the first lower lip area. For example, in combination with Figure 4 , the acquisition unit 1201 is used to execute S401 and S403.
[0132] Optionally, the processing unit 1203 is further used to determine a plurality of second key points in the second upper lip region that respectively correspond to a plurality of first key points of the video frame to be processed, and to determine a plurality of fourth key points in the second lower lip region that respectively correspond to a plurality of third key points of the video frame to be processed; the plurality of first key points are used to characterize the first upper lip region, the plurality of second key points are used to characterize the second upper lip region, the plurality of third key points are used to characterize the first lower lip region, and the plurality of fourth key points are used to characterize the second lower lip region. The processing unit 1203 is specifically used to superimpose the deformed second upper lip region onto the first upper lip region based on the plurality of first key points and the plurality of second key points, and to superimpose the deformed second lower lip region onto the first lower lip region based on the plurality of third key points and the plurality of fourth key points.
[0133] Optionally, the processing unit 1203 is specifically used to superimpose multiple first key points onto corresponding second key points according to the position correspondence between multiple first key points and multiple second key points, so that the first upper lip area is covered with the second upper lip area after the deformation processing; and to superimpose multiple third key points onto corresponding fourth key points according to the position correspondence between multiple third key points and multiple fourth key points, so that the second lower lip area after the deformation processing covers the first lower lip area.
[0134] Optionally, when the first upper lip region and the first lower lip region in the superimposed video frames have an overlapping area, the processing unit 1203 is further configured to perform color fading processing on the overlapping area.
[0135] Optionally, when the first upper lip area completely covers the first lower lip area, the processing unit 1203 is also used to delete the second lower lip area and superimpose the deformed second upper lip area on the first upper lip area; when the first lower lip area completely covers the first upper lip area, the processing unit 1203 is also used to delete the second upper lip area and superimpose the deformed second lower lip area on the second lip area.
[0136] Optionally, when the first upper lip area covers part of the first lower lip area, or the first lower lip area covers part of the first upper lip area, the determination unit 1202 is specifically used to determine the first lip line and the second lip line of the covered area, the first lip line is the lip line of the covering lip area and connected with the covered lip area, and the second lip line is the lip line of the covered lip area that is not blocked; when the closed area formed by the first lip line and the second lip line is located above the covering lip area, the closed area is taken as the first upper lip area, and the covering lip area is taken as the first lower lip area; when the closed area formed by the first lip line and the second lip line is located below the covering lip area, the closed area is taken as the first lower lip area, and the covering area is taken as the first upper lip area.
[0137] Optionally, when the first upper lip area covers part of the first lower lip area, the processing unit 1203 is specifically used to deform the second lower lip area so that the shape of the deformed second lower lip area matches the shape of the closed area; superimpose the second upper lip area on the first upper lip area, and superimpose the deformed second lower lip area on the closed area; when the first lower lip area covers part of the first upper lip area, the processing unit 1203 is specifically used to deform the second upper lip area so that the shape of the deformed second upper lip area matches the shape of the closed area; superimpose the shape of the deformed second upper lip area on the first upper lip area, and superimpose the second lower lip area on the first lower lip area.
[0138] As described above, the embodiment of the present disclosure can divide the functional modules of the image processing device according to the above method example. Among them, the above integrated modules can be implemented in the form of hardware or in the form of software functional modules. In addition, it should be noted that the division of modules in the embodiment of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module.
[0139] Regarding the image processing device in the above embodiment, the specific manner in which each module performs operations and the beneficial effects possessed have been described in detail in the aforementioned method embodiment and will not be repeated here.
[0140] The embodiment of the present disclosure also provides an image processing device. Fig.13 The schematic diagram of the structure of the image processing device provided by the embodiment of the present disclosure is shown. The image processing device may include at least one processor 131, a communication bus 132, a memory 133 and at least one communication interface 134.
[0141] Processor 131 may be a processor (central processing unit, CPU), microprocessor unit, ASIC, or one or more integrated circuits for controlling the execution of the program of the disclosed solution. Fig. 9 The function implemented by the determination unit 1202 in the image processing device is similar to Fig.13 The functions implemented by the processor 131 in are the same.
[0142] The communication bus 132 may include a pathway for transmitting information between the above-mentioned components.
[0143] The communication interface 134 uses any transceiver or other device for communicating with other devices or communication networks, such as servers, Ethernet, radio access networks (RAN), wireless local area networks (WLAN), etc. As an example, in combination with Fig.13 The function implemented by the acquisition unit 902 in the image processing device is similar to Fig.13 The communication interface 134 in FIG.
[0144] The memory 133 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processing unit through a bus. The memory may also be integrated with the processing unit.
[0145] The memory 133 is used to store application code for executing the solution of the present disclosure, and the execution is controlled by the processor 131. The processor 131 is used to execute the application code stored in the memory 133, so as to realize the functions in the method of the present disclosure.
[0146] In a specific implementation, as an embodiment, the processor 131 may include one or more CPUs, such as Fig.13 CPU0 and CPU1 in.
[0147] In a specific implementation, as an embodiment, the terminal may include multiple processors, such as Fig.13 1 and 135. Each of these processors may be a single-CPU processor or a multi-CPU processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).
[0148] In a specific implementation, as an embodiment, the image processing apparatus may further include an input device 136 and an output device 137. The input device 136 communicates with the output device 137 and may receive user input in a variety of ways. For example, the input device 136 may be a mouse, a keyboard, a touch screen device, or a sensor device. The output device 137 communicates with the processor 131 and may display information in a variety of ways. For example, the output device 131 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, etc.
[0149] Those skilled in the art will understand that Fig.13 The structure shown in the figure does not constitute a limitation on the image processing device, and may include more or less components than those shown in the figure, or combine certain components, or adopt a different component arrangement.
[0150] The present disclosure also provides a computer-readable storage medium including instructions, and the computer-readable storage medium stores instructions. When the instructions in the computer-readable storage medium are executed by the processor of a computer device, the computer can execute the image processing method provided by the above-mentioned embodiment. For example, the computer-readable storage medium may be a memory 133 including instructions, and the above instructions may be executed by the processor 131 of the image processing device to complete the above method. For another example, the computer-readable storage medium may be a memory 132 including instructions, and the above instructions may be executed by the processor 131 of the server to complete the above method. Optionally, the computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
[0151] The present disclosure also provides a computer program product, which includes computer instructions. When the computer instructions are executed on an image processing device, the image processing device executes the above Figure 4 , Figure 5 , Figure 8-Figure 10 An image processing method as shown in any one of the accompanying drawings.
[0152] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0153] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. An image processing method, characterized in that: The method comprises: Acquire a video frame to be processed, wherein the video frame to be processed includes a lip region; the lip region is in a closed state; Determine a first upper lip region and a first lower lip region of the lip region; Acquire a preset open lip makeup image, wherein the preset open lip makeup image includes a second upper lip area and a second lower lip area; Performing deformation processing on the second upper lip region and the second lower lip region; wherein the shape of the second upper lip region after the deformation processing matches the shape of the first upper lip region, and the shape of the second lower lip region after the deformation processing matches the shape of the first lower lip region; superimposing the second upper lip region after deformation processing onto the first upper lip region, and superimposing the second lower lip region after deformation processing onto the first lower lip region, to obtain a superimposed video frame; Determine a plurality of second key points in the second upper lip region that respectively correspond to a plurality of first key points of the video frame to be processed, and determine a plurality of fourth key points in the second lower lip region that respectively correspond to a plurality of third key points of the video frame to be processed; the plurality of first key points are used to characterize the first upper lip region, the plurality of second key points are used to characterize the second upper lip region, the plurality of third key points are used to characterize the first lower lip region, and the plurality of fourth key points are used to characterize the second lower lip region; The step of superimposing the second upper lip region after deformation onto the first upper lip region, and superimposing the second lower lip region after deformation onto the first lower lip region, comprises: According to the multiple first key points and the multiple second key points, the second upper lip area after deformation is superimposed on the first upper lip area, and according to the multiple third key points and the multiple fourth key points, the second lower lip area after deformation is superimposed on the first lower lip area.
2. The method according to claim 1, characterized in that The method of superimposing the deformed second upper lip region onto the first upper lip region according to the plurality of first key points and the plurality of second key points, and superimposing the deformed second lower lip region onto the first lower lip region according to the plurality of third key points and the plurality of fourth key points, comprises: According to the position correspondence between the plurality of first key points and the plurality of second key points, the plurality of second key points are respectively superimposed on the corresponding first key points, so that the second upper lip region after deformation covers the first upper lip region; According to the position correspondence between the plurality of third key points and the plurality of fourth key points, the plurality of fourth key points are respectively superimposed on the corresponding third key points, so that the second lower lip area after deformation covers the first lower lip area.
3. The method according to claim 1, characterized in that When there is an overlapping area between the first upper lip area in the superimposed video frame and the first lower lip area in the superimposed video frame, the method further includes: The overlapping area is subjected to color fading processing.
4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: When the first upper lip region completely covers the first lower lip region, deleting the second lower lip region, and superimposing the deformed second upper lip region onto the first upper lip region; When the first lower lip region completely covers the first upper lip region, the second upper lip region is deleted, and the deformed second lower lip region is superimposed on the second lip region.
5. The method according to any one of claims 1 to 3, characterized in that: When the first upper lip area covers a portion of the first lower lip area, or the first lower lip area covers a portion of the first upper lip area, determining a first upper lip area and a first lower lip area of the lip area in the to-be-processed video frame includes: Determine a first lip line and a second lip line of the covered lip area, wherein the first lip line is a lip line of the covering lip area that is connected to the covered lip area, and the second lip line is a lip line of the covered lip area that is not blocked; When the closed area formed by the first lip line and the second lip line is located above the covered lip area, the closed area is used as the first upper lip area, and the covered lip area is used as the first lower lip area; When a closed area formed by the first lip line and the second lip line is located below the covered lip area, the closed area is used as the first lower lip area, and the covered lip area is used as the first upper lip area.
6. The method according to claim 5, characterized in that When the first upper lip area covers a portion of the first lower lip area, the deforming the second upper lip area and the second lower lip area includes: Performing deformation processing on the second upper lip region so that the shape of the second upper lip region after the deformation processing matches the shape of the first upper lip region, and performing deformation processing on the second lower lip region so that the shape of the second lower lip region after the deformation processing matches the shape of the closed region; The step of superimposing the second upper lip region after the deformation process onto the first upper lip region, and superimposing the second lower lip region after the deformation process onto the first lower lip region, comprises: superimposing the second upper lip region after deformation processing onto the first upper lip region, and superimposing the second lower lip region after deformation processing onto the closed region; When the first lower lip area covers a portion of the first upper lip area, the deforming the second upper lip area and the second lower lip area includes: Performing deformation processing on the second upper lip region so that the shape of the second upper lip region after the deformation processing matches the shape of the closed region, and performing deformation processing on the second lower lip region so that the shape of the second lower lip region after the deformation processing matches the shape of the first lower lip region; The step of superimposing the second upper lip region after deformation onto the first upper lip region, and superimposing the second lower lip region after deformation onto the first lower lip region, comprises: The second upper lip region after deformation processing is superimposed on the closed region, and the second lower lip region after deformation processing is superimposed on the first lower lip region.
7. An image processing device, characterized in that: The device comprises an acquisition unit, a determination unit and a processing unit; The acquisition unit is used to acquire a video frame to be processed, wherein the video frame to be processed includes a lip area; the lip area is in a closed state; The determining unit is used to determine a first upper lip region and a first lower lip region of the lip region; The acquisition unit is further used to acquire a preset open lip makeup image, wherein the preset open lip makeup image includes a second upper lip area and a second lower lip area; The processing unit is used to perform deformation processing on the second upper lip area and the second lower lip area; wherein the shape of the second upper lip area after the deformation processing matches the shape of the first upper lip area, and the shape of the second lower lip area after the deformation processing matches the shape of the first lower lip area; The processing unit is further configured to superimpose the second upper lip region after deformation processing onto the first upper lip region, and superimpose the second lower lip region after deformation processing onto the first lower lip region, to obtain a superimposed video frame; The determination unit is further used to: determine a plurality of second key points in the second upper lip region that respectively correspond to a plurality of first key points of the video frame to be processed, and determine a plurality of fourth key points in the second lower lip region that respectively correspond to a plurality of third key points of the video frame to be processed; the plurality of first key points are used to characterize the first upper lip region, the plurality of second key points are used to characterize the second upper lip region, the plurality of third key points are used to characterize the first lower lip region, and the plurality of fourth key points are used to characterize the second lower lip region; The processing unit is specifically used to: superimpose the second upper lip area after deformation processing on the first upper lip area according to the multiple first key points and the multiple second key points, and superimpose the second lower lip area after deformation processing on the first lower lip area according to the multiple third key points and the multiple fourth key points.
8. The device according to claim 7, characterized in that The processing unit is specifically used for: According to the position correspondence between the plurality of first key points and the plurality of second key points, the plurality of second key points are respectively superimposed on the corresponding first key points, so that the second upper lip region after deformation covers the first upper lip region; According to the position correspondence between the plurality of third key points and the plurality of fourth key points, the plurality of fourth key points are respectively superimposed on the corresponding third key points, so that the second lower lip area after deformation covers the first lower lip area.
9. The device according to claim 7, characterized in that When there is an overlapping area between the first upper lip area in the superimposed video frame and the first lower lip area in the superimposed video frame, the processing unit is further configured to: The overlapping area is subjected to color fading processing.
10. The device according to any one of claims 7 to 9, characterized in that: The processing unit is further used for: When the first upper lip region completely covers the first lower lip region, deleting the second lower lip region, and superimposing the deformed second upper lip region onto the first upper lip region; When the first lower lip region completely covers the first upper lip region, the second upper lip region is deleted, and the deformed second lower lip region is superimposed on the second lip region.
11. The device according to any one of claims 7 to 9, characterized in that: When the first upper lip area covers a portion of the first lower lip area, or the first lower lip area covers a portion of the first upper lip area, the determining unit is specifically configured to: Determine a first lip line and a second lip line of the covered lip area, wherein the first lip line is a lip line of the covering lip area that is connected to the covered lip area, and the second lip line is a lip line of the covered lip area that is not blocked; When the closed area formed by the first lip line and the second lip line is located above the covered lip area, the closed area is used as the first upper lip area, and the covered lip area is used as the first lower lip area; When a closed area formed by the first lip line and the second lip line is located below the covered lip area, the closed area is used as the first lower lip area, and the covered lip area is used as the first upper lip area.
12. The device according to claim 11, characterized in that When the first upper lip area covers a portion of the first lower lip area, the processing unit is specifically configured to: Performing deformation processing on the second upper lip region so that the shape of the second upper lip region after the deformation processing matches the shape of the first upper lip region, and performing deformation processing on the second lower lip region so that the shape of the second lower lip region after the deformation processing matches the shape of the closed region; superimposing the second upper lip area after the deformation process onto the first upper lip area, and superimposing the second lower lip area after the deformation process onto the closed area; When the first lower lip area covers a portion of the first upper lip area, the processing unit is specifically configured to: Performing deformation processing on the second upper lip region so that the shape of the second upper lip region after the deformation processing matches the shape of the closed region, and performing deformation processing on the second lower lip region so that the shape of the second lower lip region after the deformation processing matches the shape of the first lower lip region; The second upper lip area after the deformation process is superimposed on the closed area, and the second lower lip area after the deformation process is superimposed on the first lower lip area.
13. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method according to any one of claims 1 to 6.
14. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image processing method according to any one of claims 1 to 6.
15. A computer program product comprising instructions, characterized in that When the instruction is executed on an electronic device, the electronic device executes the image processing method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Three-dimensional grid simplifying method and device, electronic equipment and storage medium
CN112614228A
System and Method for Extremely Efficient Image and Pattern Recognition and Artificial Intelligence Platform
US20200184278A1
