Image processing method and electronic device

By identifying and processing specific points in images or videos to formulate display strategies, the problem of poor user experience in existing technologies is solved, achieving higher processing accuracy and user satisfaction.

CN112132085BActive Publication Date: 2024-11-29LENOVO (BEIJING) LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202011052499.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-29
Publication Date
2024-11-29
Estimated Expiration
2040-09-29

AI Technical Summary

Technical Problem

Current image or video processing technologies cannot meet user needs and negatively impact user experience.

Method used

By identifying specific points in the captured video, corresponding operations are performed, and display strategies are formulated based on specific conditions to achieve different processing effects for different areas.

Benefits of technology

It improves the accuracy of image or video processing and user experience, meeting users' optimization needs for different areas.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112132085B_ABST
    Figure CN112132085B_ABST
Patent Text Reader

Abstract

The present application discloses an image processing method and an electronic device, the method comprising: obtaining a first captured video; identifying at least one specific point of a first object in the first captured video; obtaining a first operation, the first operation being an operation on a first specific point among the at least one specific point; performing a first processing on the first captured video according to the first operation to obtain a first processed video; the first processing at least comprising processing an area corresponding to a first feature point; analyzing the obtained second captured video; formulating a display strategy when the area corresponding to the first specific point in the second captured video meets specific conditions; the display strategy at least comprising: displaying a visual effect after the first processing on the area corresponding to the second specific point; displaying a visual effect without the first processing on the area corresponding to the third specific point; processing the second captured video or the first processed video according to the display strategy to obtain a second processed video and displaying the second processed video.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to image processing technology, and relate to but are not limited to an image processing method and an electronic device. Background Art

[0002] The processing of images or videos in the related art cannot achieve the effect expected by users, which affects the user experience. Therefore, how to meet the user's demand for image or video processing and improve the user experience is an issue that technicians in this field need to focus on. Summary of the invention

[0003] In view of this, embodiments of the present application provide an image processing method and an electronic device.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Obtaining the first acquisition video;

[0007] identifying at least one specific point of a first object in the first captured video;

[0008] Acquire a first operation, where the first operation is an operation on a first specific point among the at least one specific point;

[0009] Performing a first process on the first captured video according to the first operation to obtain a first processed video and displaying the first processed video; wherein the first process at least includes processing an area corresponding to the first feature point;

[0010] Analyzing the obtained second acquisition video;

[0011] When the area corresponding to the first specific point in the second captured video meets a specific condition, a display strategy is formulated; the display strategy at least includes: displaying the visual effect after the first processing for the area corresponding to the second specific point; and displaying the visual effect without the first processing for the area corresponding to the third specific point;

[0012] The second acquired video or the first processed video is processed according to the display strategy to obtain a second processed video and display the second processed video.

[0013] On the other hand, an embodiment of the present application provides an image processing device, the device comprising:

[0014] A first acquisition module, used to obtain a first captured video;

[0015] A first recognition module, used to recognize at least one specific point of a first object in the first captured video;

[0016] A second obtaining module is used to obtain a first operation, where the first operation is an operation on a first specific point among the at least one specific point;

[0017] A third obtaining module is used to perform a first processing on the first collected video according to the first operation to obtain a first processed video and display the first processed video; wherein the first processing at least includes processing an area corresponding to the first feature point;

[0018] An analysis module, used for analyzing the obtained second collected video;

[0019] A display module is used to formulate a display strategy when the area corresponding to the first specific point in the second captured video meets a specific condition; the display strategy at least includes: displaying the visual effect after the first processing for the area corresponding to the second specific point; and displaying the visual effect without the first processing for the area corresponding to the third specific point;

[0020] A processing module is used to process the second collected video or the first processed video according to the display strategy to obtain a second processed video and display the second processed video.

[0021] On the other hand, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and the processor implements the steps in the above method when executing the program.

[0022] On the other hand, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps in the above method when executed by a processor.

[0023] The present disclosure provides a computer program product, including a computer-readable code. When the computer-readable code is executed in an electronic device, a processor in the electronic device executes the image processing method in one or more of the above embodiments.

[0024] In an embodiment of the present application, on the one hand, a first operation is obtained, and the first operation is an operation on a first specific point among the at least one specific point; a first processing is performed on the first captured video according to the first operation to obtain a first processed video and display the first processed video; wherein the first processing at least includes processing the area corresponding to the first feature point; in this way, the first specific point and the area corresponding to the first specific point can be processed through the first operation, thereby ensuring the certainty of the processing area, improving the accuracy of processing the image or video, meeting user needs, and improving the user experience. On the other hand, when the area corresponding to the first specific point in the second captured video meets specific conditions, a display strategy is formulated; the display strategy at least includes: displaying the visual effect after the first processing for the area corresponding to the second specific point; and displaying the visual effect without the first processing for the area corresponding to the third specific point. In this way, different processing operations can be performed on the areas corresponding to different specific points, meeting the user's needs for different optimizations of different areas, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of the implementation process of the image processing method of the embodiment of the present application;

[0026] Figure 2A This is a schematic diagram of the implementation process of the image processing method of the embodiment of the present application;

[0027] Figure 2B This is a schematic diagram of the implementation process of the image processing method of the embodiment of the present application;

[0028] Figure 2C This is a schematic diagram of the implementation effect of the image processing method of the embodiment of the present application;

[0029] Figure 2D This is a schematic diagram of the implementation effect of the image processing method of the embodiment of the present application;

[0030] Figure 3A This is a schematic diagram of the implementation process of the image processing method of the embodiment of the present application;

[0031] Figure 3B This is a schematic diagram of the implementation effect of the image processing method of the embodiment of the present application;

[0032] Figure 3C This is a schematic diagram of the implementation process of the image processing method of the embodiment of the present application;

[0033] Figure 4A This is a schematic diagram of the implementation process of the image processing method of the embodiment of the present application;

[0034] Figure 4B This is a schematic diagram of the high-density facial key point detection result in the image processing method of the embodiment of the present application;

[0035] Figure 5 This is a schematic diagram of the structure of the image processing device according to an embodiment of the present application;

[0036] Figure 6 A schematic diagram of a hardware entity of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0038] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] If similar descriptions of "first / second" appear in the application documents, the following instructions are added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0041] The technical solution of the present application is further described in detail below with reference to the accompanying drawings and embodiments.

[0042] The present application provides an image processing method. Figure 1 Schematic diagram of the implementation process of the image processing method of the present application embodiment. Figure 1 As shown, the method includes:

[0043] Step S101, obtaining a first captured video;

[0044] Here, the first captured video is an original video shot by an image capture device. Here, the image capture device may be a camera, a video camera, or other mobile terminal with a camera function. The original video is a video that has not been edited, clipped, or processed with special effects.

[0045] Here, the first captured video includes complete facial video information of at least one person.

[0046] Step S102, identifying at least one specific point of the first object in the first captured video;

[0047] Here, the first object may be a person, and the at least one specific point may be a facial key point obtained by identifying facial information in a frame image including the person. Here, the identification may be achieved by high-density facial key point detection.

[0048] Step S103, obtaining a first operation, where the first operation is an operation on a first specific point among the at least one specific point;

[0049] Here, the first specific point may be a detected key point of the face, for example, an edge of an eyebrow.

[0050] The first operation may be a beautification operation performed on the identified first specific point, or may be an enlargement operation performed on the first specific point.

[0051] Step S104, performing a first process on the first collected video according to the first operation to obtain a first processed video and display the first processed video; wherein the first process at least includes processing an area corresponding to the first feature point;

[0052] Here, the first processing is performed on the first captured video according to the first operation. Here, the object of the first processing may be the area corresponding to the first specific point. Here, the first processed video is the video processed by the first operation. For example, it may be a beautified video of the first specific point, or it may be an enlarged video.

[0053] Step S105, analyzing the obtained second captured video;

[0054] Here, the second captured video may be a video in which the key point changes, for example, when the user blocks the key point with any object, or when the key point moves.

[0055] Step S106, when the area corresponding to the first specific point in the second captured video meets a specific condition, formulate a display strategy; the display strategy at least includes: displaying the visual effect after the first processing for the area corresponding to the second specific point; and displaying the visual effect without the first processing for the area corresponding to the third specific point;

[0056] Here, the specific condition may be to determine that the area corresponding to the first specific point is blocked by an object, or to determine that the area corresponding to the first specific point has moved. Here, the second specific point may be a non-blocked area, or an area to be enlarged, for example, when beautifying facial information, it is an unblocked eyebrow area; in a zoom-in operation, it is an enlarged area. The third specific point may be an blocked area, or a non-zoomed area. For example, it may be an blocked eyebrow area and a non-eyebrow area, or an area where the zoom-in operation is not performed.

[0057] Here, the display strategy may be to add a beautifying pattern to the non-blocked area, and to display the blocked area without adding a beautifying pattern. Alternatively, the display strategy may be to amplify the area to be amplified.

[0058] Step S107: Process the second captured video or the first processed video according to the display strategy to obtain a second processed video and display the second processed video.

[0059] Here, the second processed video may be to add a beautification pattern to the non-blocked area but not to other blocked areas; or, may be to amplify the area to be amplified.

[0060] In an embodiment of the present application, on the one hand, a first operation is obtained, and the first operation is an operation on a first specific point among the at least one specific point; a first processing is performed on the first captured video according to the first operation to obtain a first processed video and display the first processed video; wherein the first processing at least includes processing the area corresponding to the first feature point; in this way, the first specific point and the area corresponding to the first specific point can be processed through the first operation, thereby ensuring the certainty of the processing area, improving the accuracy of processing the image or video, meeting user needs, and improving the user experience. On the other hand, when the area corresponding to the first specific point in the second captured video meets specific conditions, a display strategy is formulated; the display strategy at least includes: displaying the visual effect after the first processing for the area corresponding to the second specific point; and displaying the visual effect without the first processing for the area corresponding to the third specific point. In this way, different processing operations can be performed on the areas corresponding to different specific points, meeting the user's needs for different optimizations of different areas, and improving the user experience.

[0061] The present application provides an image processing method. Figure 2A Schematic diagram of the implementation process of the image processing method of the present application embodiment. Figure 2A As shown, the method includes:

[0062] Step S201, obtaining a first captured video;

[0063] Step S202, identifying at least one specific point of the first object in the first captured video;

[0064] Step S203, obtaining a first operation, where the first operation is an operation of adding a decoration pattern to the first specific point;

[0065] Here, the operation of adding a dressing pattern may be an operation of beautifying the facial features, for example, an operation of drawing eyebrows.

[0066] Step S204, according to the operation of adding the dressing pattern, adding the dressing pattern to the area corresponding to the first feature point in the first collected video to obtain a first processed video;

[0067] Here, the first processed video may be a video after beautifying the facial features, for example, a video after adding eyebrow shapes to the eyebrows.

[0068] Step S205, analyzing the obtained second captured video;

[0069] Step S206, when the area corresponding to the first specific point in the second captured video meets a specific condition, formulate a display strategy; the display strategy at least includes: displaying the visual effect after the first processing for the area corresponding to the second specific point; and displaying the visual effect without the first processing for the area corresponding to the third specific point;

[0070] Step S207: Process the second captured video or the first processed video according to the display strategy to obtain a second processed video and display the second processed video.

[0071] The image processing method provided in the embodiment of the present application adds a dressing pattern to the area corresponding to the first feature point in the first captured video according to the operation of adding the dressing pattern, so as to obtain a first processed video. In this way, the effect of beautifying the characteristics of the characters in the specific area can be achieved, thereby achieving the beautification of the specific area.

[0072] The present application provides an image processing method. Figure 2B Schematic diagram of the implementation process of the image processing method of the present application embodiment. Figure 2B As shown, the method includes:

[0073] Step S210, obtaining a first captured video;

[0074] Step S220, identifying at least one specific point of the first object in the first captured video;

[0075] Step S230, obtaining a first operation, where the first operation is an operation of adding a decoration pattern to the first specific point;

[0076] Step S240, determining the location of the operation of adding the dress pattern;

[0077] Here, the position may be the position of any part of the five sense organs.

[0078] Step S250, determining a dressing area to be added with a pattern on the first captured video according to the position;

[0079] In the implementation process, the position is used to determine the characteristic edge corresponding to the position, and the edge area is determined as the area to be dressed up. For example, when the position is determined to be the left eyebrow, the edge of the detected left eyebrow is determined as the area to be dressed up.

[0080] Step S260, adding a dressing pattern to the dressing area to be added with the pattern, obtaining the first processed video and displaying the first processed video; wherein the first processing at least includes processing the area corresponding to the first feature point;

[0081] For example, an eyebrow shape is added to the left eyebrow to obtain a first processed video, and the video after the eyebrow shape is added is displayed.

[0082] Step S270, analyzing the obtained second captured video;

[0083] Step S280, if there is an occluded area in the area corresponding to the first specific point in the second captured video, a display strategy is formulated; the display strategy at least includes: displaying the visual effect after the first processing for the unoccluded area; and displaying the visual effect without the first processing for the occluded area;

[0084] For example, the occluded area may be an area irrelevant to the determined feature. Figure 2C Schematic diagram of the effect of implementing the image processing method of the embodiment of the present application. Figure 2D Schematic diagram of the implementation effect of the image processing method of the embodiment of the present application, as shown in Figure 2C As shown, when the determined feature is an eyebrow, the occluded area 21 may be a finger on the eyebrow. Figure 2D As shown, the unobstructed area 22 may be the eyebrows except the fingers. The display strategy may be: erasing the eyebrow shape of the eyebrow part obstructed by the fingers, and adding the eyebrow shape to the unobstructed part.

[0085] Step S290, processing the second captured video or the first processed video according to the display strategy to obtain a second processed video and display the second processed video, wherein the unobstructed area in the second processed video displays the visual effect after the first processing, and the obstructed area displays the visual effect before the first processing.

[0086] In the implementation process, after different processing is performed on the unobstructed area and the obstructed area, the processed video needs to be displayed. For example, the part obstructed by the finger is erased, and the unobstructed part is still displayed. In this way, the makeup effect of the video character can be close to the actual makeup, and the display effect is more realistic.

[0087] In some embodiments, the step S290, processing the first processed video according to the display strategy to obtain a second processed video, includes:

[0088] Step S291, adding a dressing pattern to the dressing area in the first processed video, not displaying the dressing pattern in the blocked area in the dressing area, to obtain a second processed video; or

[0089] Step S292: adding a dressing pattern to the unblocked area in the dressing area in the first processed video.

[0090] During implementation, there are two ways to prevent the unobstructed area from displaying the decoration pattern: 1) adding a decoration pattern to the decoration area, but not displaying the decoration pattern of the obstructed area in the decoration area; for example, making it transparent; 2) when adding a decoration pattern, only adding the decoration pattern to the unobstructed area.

[0091] In some embodiments, the step S291 of adding a dressing pattern to the dressing area in the first processed video and not displaying the dressing pattern of the blocked area in the dressing area to obtain the second processed video includes:

[0092] Step S2911, using a three-dimensional fitting algorithm to fit the dress pattern to the corresponding dress area;

[0093] Here, the three-dimensional fitting algorithm can overlay the selected eyebrow pattern on the face video stream.

[0094] Step S2912, retouching the attached dressing pattern to remove the dressing pattern in the blocked area of ​​the dressing area;

[0095] Step S2913, after the image editing is completed or during the image editing process, the dressing pattern of the blocked area in the dressing area is not displayed to obtain a second processed video.

[0096] For example, when the eyebrow pattern is overlaid on the face video stream, the eyebrow pattern in the area blocked by fingers etc. on the eyebrows can be made transparent after the overlay is completed, so as to obtain a video after the blocked area is processed.

[0097] The image processing method provided in the embodiment of the present application processes the second acquired video or the first processed video according to the display strategy to obtain the second processed video and display the second processed video. In this way, the video can be processed by the display strategy so that the dressing effect of the characters in the video is close to the actual makeup and the display effect is more realistic.

[0098] The present application provides an image processing method, the method comprising:

[0099] Step S21, obtaining a face video;

[0100] Step S22, identifying at least one specific point of the first object in the face video;

[0101] Step S23, obtaining a first operation, where the first operation is an operation of adding a decoration pattern to the first specific point;

[0102] Step S24, determining the location of the operation of adding the dress pattern;

[0103] Step S25, determining a dressing area to be added with a pattern on the face video according to the position;

[0104] Step S26, adding a dressing pattern to the dressing area to be added with the pattern, obtaining the first processed video and displaying the first processed video; wherein the first processing at least includes processing the area corresponding to the first feature point;

[0105] Step S27, analyzing the acquired face video;

[0106] Step S28, when there is an occluded area in the area corresponding to the first specific point in the face video, formulate a display strategy; the display strategy at least includes: displaying the visual effect after the first processing for the unoccluded area, and displaying the visual effect without the first processing for the occluded area;

[0107] Step S29, processing the first processed video according to the display strategy to obtain a second processed video and display the second processed video, wherein the unobstructed area in the second processed video displays the visual effect after the first processing, and the obstructed area displays the visual effect before the first processing.

[0108] In some embodiments, the method further includes step S30 and step S40: step S30, performing facial key point detection on the dress area to obtain a facial area;

[0109] In some embodiments, the step S30 of performing facial key point detection on the dress area to obtain the facial area includes:

[0110] Step S31, performing facial key point detection on the first image to be processed to obtain facial edges;

[0111] Here, the facial key point detection may be high-density facial key point detection, which may be a 5000 to 10000 point algorithm for scanning the key points of the entire face.

[0112] Step S32, performing face detection on the first image to be processed to determine facial features of the specific object;

[0113] Here, the face detection algorithm can be used to obtain the face filling information after obtaining the face edge.

[0114] Step S33, superimposing the facial features and the face edge to obtain a face region.

[0115] During implementation, the face edge and the facial features are superimposed to obtain an image of the face area.

[0116] Step S40, performing image recognition on the face area to obtain a recognition result, wherein the recognition result includes whether there is an obstructed area in the dressing area.

[0117] In some embodiments, the step S40, performing image recognition on the face area to obtain a recognition result, wherein the recognition result includes whether there is an occluded area in the dress area, comprises: step S41, performing key feature recognition on the face area to determine key features of the face;

[0118] During the implementation process, key features of the face such as eyebrows and eyes can be obtained through key feature recognition.

[0119] Step S42, performing dense key point detection on the face area to determine the edges of the key features of the face;

[0120] Here, the density key point detection algorithm is a contour detection algorithm, which is the density of the key features of the face. For example, for the eyebrow edge, the 1000 key point algorithm is used to detect the eyebrow edge.

[0121] Here, a dense key point algorithm is used to select S dense points other than feature points on the face image, and the edges of the key features of the face are more accurately identified based on the S dense points and the determined key points.

[0122] Step S43, determining a key feature region in the face region according to the key features of the face and the edges of the key features of the face;

[0123] Step S44, when the key feature area includes a non-key feature, determining that the non-key feature is an obstruction;

[0124] For example, the edge of the eyebrow is detected, and when feature points of an abnormal eyebrow shape or an edge line composed of feature points are detected, it is determined that the eyebrow area includes non-eyebrows, and the non-eyebrow area is determined to be an occluder.

[0125] Step S45: perform edge detection on the obstruction to obtain the obstruction area.

[0126] For example, when the eyebrows are blocked by hands, it is detected that the recognized eyebrows have hand feature information, and it is determined that the eyebrows are blocked by the hands. Edge detection is performed on the blocker to identify the shape of the blocker.

[0127] The image processing method provided in the embodiment of the present application can identify the blocked parts of eyebrows or eyes through high-density facial key point detection; and can accurately identify the blocked parts by performing edge detection on the blocked objects to obtain the blocked areas, and then use different display strategies according to the blocked areas and non-blocked areas. In this way, the dressing effect of the characters in the video can be close to the actual makeup, and the display effect is more realistic.

[0128] The present application provides an image processing method. Figure 3A Schematic diagram of the implementation process of the image processing method of the present application embodiment. Figure 3A As shown, the method includes:

[0129] Step S310, obtaining a first captured video;

[0130] Step S320, identifying at least one specific point of the first object in the first captured video;

[0131] Step S330, obtaining a first operation, where the first operation is a zoom-in operation on the first specific point;

[0132] For example, the first specific point may be a focal point in the middle of an eyebrow, and the first operation may be an operation of enlarging the eyebrow.

[0133] Step S340, performing a magnification process on the first captured video according to the magnification operation to obtain a first processed video; wherein the first processing at least includes processing an area corresponding to the first feature point;

[0134] Here, the first processed video includes a first window, and the image in the first window is an enlarged image centered on the first specific point.

[0135] For example, when a region centered on an eyebrow is enlarged, the region centered on the eyebrow will be displayed in the local enlargement window.

[0136] In some embodiments, the step S340, performing a magnification process on the first captured video according to the magnification operation to obtain a first processed video, includes:

[0137] Step S341, determining the position of the zoom-in operation;

[0138] During implementation, the key points in the face can be identified through facial key point recognition technology, and when it is detected that the user clicks on any specific point on the face, the position coordinates of the specific point can be determined.

[0139] Step S342, determining an image area to be displayed in the first captured video according to the coordinates of the position;

[0140] Here, the display range may be preset. For example, the range covered by a circle with the position coordinate as the center and a specific length as the radius may be preset as the display range.

[0141] During implementation, an image area within a display range is determined according to the coordinates of the position.

[0142] In some embodiments, the step S342, determining the image area to be displayed in the first captured video according to the coordinates of the position, includes:

[0143] Step S3421, identifying key points of the target object in the first captured video;

[0144] Here, the key point of the target object may be a key point in a human face, for example, the beginning of an eyebrow, or the end of an eyebrow.

[0145] Step S3422, determining a feature point corresponding to the position on the target object according to the relative position relationship between the coordinates of the position and the key point;

[0146] During implementation, when the first specific point is not a key point, it is necessary to determine the position change of the first specific point based on the positional relationship between the first specific point and the key point to obtain the coordinates of the position change.

[0147] Step S3423, determining the first coordinates of the feature point in the first captured video;

[0148] Here, the first coordinates are the coordinates of the original position, for example, the eyebrow coordinates of the eyebrow (50, 51).

[0149] Step S3424, determining an image area to be displayed in the first captured video according to the first coordinates;

[0150] For example, according to the eyebrow coordinates (50, 51), the area to be displayed is determined to be a display area with (50, 51) as the center and 50 as the radius.

[0151] Step S343, tracking the image area to be displayed and displaying the image area to be displayed in a specific window to obtain the first processed video;

[0152] In some embodiments, the step S343 of tracking the image area to be displayed and displaying the image area to be displayed in a specific window to obtain the first processed video includes:

[0153] Step S3431, tracking the image area to be displayed, and using the feature point as the center point of the specific window, displaying the image area to be displayed in the specific window, and obtaining the first processed video;

[0154] Here, in the case where the eyebrow coordinates (50, 51) are a specific point, the first processed video is an enlarged video centered on the eyebrow coordinates.

[0155] Step S350, analyzing the obtained second captured video;

[0156] Step S360, when the area corresponding to the first specific point in the second captured video moves, formulate a display strategy; the display strategy at least includes: displaying the visual effect after the first processing for the enlarged area; and displaying the visual effect without the first processing for the non-enlarged area;

[0157] Here, it should be noted that, when the area corresponding to the first specific point moves, the center point of the specific window is always the first specific point. For example, after determining that the first specific point is the center of the right eyebrow, the coordinates of the center are detected to be (55, 60). When the head of the person moves, the coordinates of the center point of the specific window also change. When the coordinates of the center of the eyebrow after the movement are determined to be (65, 70), the coordinates of the center point of the specific window also change to (65, 70).

[0158] Step S370, processing the second captured video or the first processed video according to the display strategy to obtain a second processed video and display the second processed video, wherein the second processed video displays the visual effect after the first processing for the enlarged area; and displays the visual effect before the first processing for the non-enlarged area.

[0159] For example, Figure 3B Schematic diagram of the implementation effect of the image processing method of the embodiment of the present application, as shown in Figure 3B As shown, the middle point of the right eyebrow is taken as the first specific point 31, the area where the first specific point is located is magnified, and the non-magnified area is not magnified.

[0160] The image processing method provided by the embodiment of the present application tracks the image area to be displayed, and uses the feature point as the center point of the specific window to display the image area to be displayed in the specific window to obtain the first processed video. In this way, when the user clicks on the area of ​​a specific point, the coordinates of this point relative to the face are identified through the face key recognition technology. When the face moves, this coordinate also moves with the movement of the face. The local magnification window reads the coordinate point of the specific point continuously, and always uses the specific point as the midpoint of the display area to display the area corresponding to this coordinate point, thereby ensuring that the magnified content of the corresponding display area will not follow the movement of the face or the head to cause the magnified area to exceed the range, thereby bringing greater convenience to the user.

[0161] In the related art, when partially zooming in on an image or video, because the partially zoomed-in window displays a fixed area with the screen as the reference coordinate, the content of the partially zoomed-in window will move with the movement of the main window content, so that the part to be observed may move out of the window due to the movement of the main window content, which reduces the user's experience of using the partially zoomed-in function. To solve the above problem, the embodiment of the present application provides an image processing method, Figure 3C Schematic diagram of the implementation process of the image processing method of the present application embodiment. Figure 3C As shown, the method includes:

[0162] Step S301, determining a first specific point as a center point of a local magnification window;

[0163] During the implementation process, when it is detected that the user clicks on the first specific point of the face image in the video, the area centered on the first specific point will be displayed in the local magnification window.

[0164] Step S302, calculating the first coordinates of the first specific point;

[0165] Here, the first coordinates are coordinates of the first specific point relative to the face.

[0166] During implementation, the coordinates of the first specific point relative to the face can be calculated through facial key point recognition technology.

[0167] Step S303, calculating the offset coordinates of the first specific point;

[0168] For example, the first coordinate of the first specific point is calculated to be (X1, Y1). When the head moves, the offset coordinate (X2, Y2) of the first coordinate (X1, Y1) can be calculated through face recognition and face key point recognition technology.

[0169] Step S304, calculating the second coordinate of the first specific point;

[0170] Here, the second coordinate is the coordinate of the first specific point after the head moves. In the implementation process, the second coordinate can be calculated based on the first coordinate and the offset coordinate of the first specific point. For example, the second coordinate can be calculated based on the first coordinate of the first specific point being (X1, Y1) and the offset coordinate being (X2, Y2).

[0171] Step S305, determining the first specific point corresponding to the second coordinate as the center point of the local magnification window;

[0172] During the implementation process, the local magnification window displays an image centered on the first specific point corresponding to the second coordinate, thereby ensuring that the specific part of the human body is always in the center of the local magnification window.

[0173] In the related art, in the scene of optimizing the display effect, when the recognized facial features are blocked by other objects, because the existing intelligent optimization display effect is achieved by identifying the position of the facial features and then adding the display effect to the corresponding position, when the recognized facial features are blocked by other objects, the display effect will be added to the corresponding blocked object, which cannot achieve the effect expected by the user and affects the user experience. To solve the above problems, the embodiment of the present application provides an image processing method, Figure 4A Schematic diagram of the implementation process of the image processing method of the present application embodiment. Figure 4A As shown, the method includes:

[0174] Step 401: obtaining a face video stream input by an image acquisition device;

[0175] Here, the image acquisition device is described by taking the camera of a mobile terminal as an example. To obtain the face video stream of the mobile terminal camera in real time when recording a video is to obtain the frame image of the current video of the face in the video stream.

[0176] Step 402: high-density facial key point detection to identify the occluded parts of eyebrows or eyes;

[0177] Here, the relative positions of key points such as eyebrows or eyes in the face can be detected by face key point detection. When eyebrows or eyes in the face are partially blocked, the blocked parts of eyebrows or eyes can be identified based on the feature information of the key points.

[0178] Figure 4B Schematic diagram of high-density facial key point detection results in the image processing method of the present application embodiment. Figure 4B As shown, through high-density facial key point detection, an algorithm with 5000 to 10000 points can be used to scan the key points 40, 41, 42, ... of the entire face.

[0179] For example, the color and shape of the eyebrows can be used as the characteristic information of the eyebrows to determine the part of the face that meets the characteristic information of the eyebrows. When it is detected that the eyebrows include non-eyebrow characteristic information, the part of the eyebrows that is blocked can be identified based on the non-eyebrow characteristic information. For example, when the eyebrows are blocked by hands, it is detected that the identified eyebrows have hand characteristic information, and it is determined that the eyebrows are blocked by the hands. The edge detection of the blocking object is performed to identify the shape of the blocking object.

[0180] Step 403: Overlaying the selected eyebrow pattern on the face video stream through face detection, face key point detection, dense key points (edges), face attributes and three-dimensional fitting algorithm and other technologies;

[0181] Here, first, a face attribute algorithm is used to select an eyebrow pattern; then, face detection, face key point detection, dense key points and a three-dimensional fitting algorithm are used to overlay the selected eyebrow pattern on the face video stream. Here, the selected eyebrow pattern is overlaid on the current frame image. Here, the face detection algorithm can be used to find the position information of all faces on the frame image. The face key point detection algorithm is used to determine the relative position of the key points in the face. The dense key point algorithm is used to select S dense points other than the feature points on the face image, and the face is more accurately identified based on these S dense points and the determined key points.

[0182] For example, first, when the facial attribute algorithm detects the facial features such as beard, the gender attribute of the face is determined; then, when the gender is determined to be male, the face shape attribute of the face is determined. For example, square face, square face and melon-seed face; finally, the eyebrow shape that matches the face shape attribute is selected to complete the eyebrow shape pattern selection.

[0183] For example, first, a face model is established based on the face detection key point detection algorithm and the dense key point algorithm. For example, the relative position of the eyes in the face and the relative position of the eyebrows relative to the eyes are located; then, the selected eyebrow shape is fitted to the face using a three-dimensional fitting algorithm, and the eyebrow shape is adjusted based on the relative positions of key points such as the nose or eyes of the fitted eyebrow shape. Furthermore, the position of the eyebrows is dynamically detected. When the face is in motion, the position of the eyebrows is dynamically detected based on the posture of the face turning left or right, and a close fit is performed.

[0184] Step 404: Erasing the portion blocked by the finger through AI image editing;

[0185] Here, through the eyebrow shape fitting in the previous step, the selected eyebrow shape pattern can be fitted on the face. Here, according to the shape of the obtained occlusion, the part of the original fitted image where the occlusion is located is erased or displayed as a transparent color.

[0186] Step 405: Display the makeup effect.

[0187] The embodiment of the present application provides an image processing method, which detects the blocked parts of eyebrows or eyes through high-density facial key points; and erases the blocked parts by fingers through AI image editing, while the unblocked parts are still displayed. In this way, the makeup effect of the characters in the video can be close to the actual makeup, and the display effect is more realistic.

[0188] Based on the foregoing embodiments, an embodiment of the present application provides an image processing device, which includes the units included and the modules included in the units, and can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0189] Figure 5 Schematic diagram of the composition structure of the image processing device according to the embodiment of the present application. Figure 5 As shown, the device 500 includes a first obtaining module 501, a first identifying module 502, a second obtaining module 503, a third obtaining module 504, an analyzing module 505, a display module 506 and a processing module 507, wherein:

[0190] A first obtaining module 501 is used to obtain a first collected video;

[0191] A first recognition module 502, configured to recognize at least one specific point of a first object in the first captured video;

[0192] A second obtaining module 503 is used to obtain a first operation, where the first operation is an operation on a first specific point among the at least one specific point;

[0193] A third obtaining module 504 is used to perform a first processing on the first collected video according to the first operation to obtain a first processed video and display the first processed video; wherein the first processing at least includes processing the area corresponding to the first feature point;

[0194] An analysis module 505 is used to analyze the obtained second captured video;

[0195] Display module 506, used for formulating a display strategy when the area corresponding to the first specific point in the second captured video meets a specific condition; the display strategy at least includes: displaying the visual effect after the first processing for the area corresponding to the second specific point; and displaying the visual effect without the first processing for the area corresponding to the third specific point;

[0196] The processing module 507 is used to process the second collected video or the first processed video according to the display strategy to obtain a second processed video and display the second processed video.

[0197] In some embodiments, the third obtaining module 504 is further configured to add a dressing pattern to the area corresponding to the first feature point in the first collected video according to the operation of adding the dressing pattern, so as to obtain a first processed video.

[0198] In some embodiments, the third obtaining module 504 includes a first determining unit, a second determining unit and a beautifying unit, wherein: the first determining unit is used to determine the location of the operation of adding the dressing pattern; the second determining unit is used to determine the dressing area to be added with the pattern on the first captured video according to the location; the beautifying unit is used to add the dressing pattern to the dressing area to be added with the pattern to obtain the first processed video.

[0199] In some embodiments, the processing module 507 is further used to add a dressing pattern to the dressing area in the first processed video, without displaying the dressing pattern in the blocked area in the dressing area, to obtain a second processed video; or, to add a dressing pattern to the unblocked area in the dressing area in the first processed video.

[0200] In some embodiments, the processing module 507 includes a pasting unit, a retouching unit and an obtaining unit, wherein: the pasting unit is used to use a three-dimensional pasting algorithm to paste the dressing pattern on the corresponding dressing area; the retouching unit is used to retouch the pasted dressing pattern to remove the dressing pattern in the occluded area in the dressing area; the obtaining unit is used to not display the dressing pattern in the occluded area in the dressing area after the retouching is completed or during the retouching process, so as to obtain a second processed video.

[0201] In some embodiments, the device 500 further includes a detection module and a second recognition module, wherein: the detection module is used to perform facial key point detection on the dressing area to obtain a facial area; the second recognition module is used to perform image recognition on the facial area to obtain a recognition result, and the recognition result includes whether there is an occluded area in the dressing area.

[0202] In some embodiments, the detection module includes a first detection unit, a second detection unit and a superposition unit, wherein: the first detection unit is used to perform face key point detection on the first image to be processed to obtain face edges; the second detection unit is used to perform face detection on the first image to be processed to determine the facial features of the specific object; the superposition unit is used to superimpose the facial features and the face edges to obtain a face area.

[0203] In some embodiments, the second recognition module includes a recognition unit, a third detection unit, a third determination unit, a fourth determination unit and a fourth detection unit, wherein: the recognition unit is used to perform key feature recognition on the face area and determine the key features of the face; the third detection unit is used to perform dense key point detection on the face area and determine the edges of the key features of the face; the third determination unit is used to determine the key feature area in the face area based on the key features of the face and the edges of the key features of the face; the fourth determination unit is used to determine that the non-key features are occluders when the key feature area includes non-key features; the fourth detection unit is used to perform edge detection on the occluder to obtain the occluded area.

[0204] In some embodiments, the third obtaining module 504 is used to enlarge the first captured video according to the enlargement operation to obtain a first processed video.

[0205] In some embodiments, the third acquisition module 504 includes a fifth determination unit, a sixth determination unit and a tracking unit, wherein: the fifth determination unit is used to determine the position of the zoom operation; the sixth determination unit is used to determine the image area to be displayed in the first acquired video according to the coordinates of the position; the tracking unit is used to track the image area to be displayed and display the image area to be displayed in a specific window to obtain the first processed video.

[0206] In some embodiments, the sixth determination unit includes an identification subunit, a first determination subunit, a second determination subunit and a third determination subunit, wherein: the identification subunit is used to identify the key points of the target object in the first acquired video; the first determination subunit is used to determine the feature points corresponding to the position on the target object based on the relative position relationship between the coordinates of the position and the key points; the second determination subunit is used to determine the first coordinates of the feature points in the first acquired video; the third determination subunit is used to determine the image area to be displayed in the first acquired video based on the first coordinates; the tracking unit is also used to track the image area to be displayed, and use the feature points as the center point of the specific window to display the image area to be displayed in the specific window to obtain the first processed video.

[0207] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0208] It should be noted that in the embodiments of the present application, if the above-mentioned image processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a mobile phone, a personal assistant, a personal computer, a server, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0209] Correspondingly, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and the processor implements the steps in the above method when executing the program.

[0210] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above method are implemented.

[0211] The present disclosure provides a computer program product including a computer-readable storage medium storing program code, wherein the program code includes instructions that can be used to execute the steps of the image processing method described in the above method embodiment. Please refer to the above method embodiment and no further details will be given here.

[0212] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0213] It should be noted that Figure 6 is a schematic diagram of a hardware entity of an electronic device in an embodiment of the present application, such as Figure 6 As shown, the hardware entity of the electronic device 600 includes: a processor 601, a communication interface 602 and a memory 603, wherein

[0214] The processor 601 generally controls the overall operation of the electronic device 600 .

[0215] The communication interface 602 enables the electronic device to communicate with other terminals or servers through a network.

[0216] The memory 603 is configured to store instructions and applications executable by the processor 601, and can also cache data to be processed or processed by the processor 601 and various modules in the electronic device 600 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0217] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0218] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0219] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0220] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0221] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0222] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0223] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0224] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Obtaining the first acquisition video; identifying at least one specific point of a first object in the first captured video; Acquire a first operation, where the first operation is an operation on a first specific point among the at least one specific point; Performing a first processing on the first captured video according to the first operation to obtain a first processed video and displaying the first processed video; wherein the first processing at least includes processing an area corresponding to the first specific point; if the first operation is a zoom-in operation on the first specific point, performing a first processing on the first captured video according to the first operation to obtain a first processed video and displaying the first processed video includes: determining a location of the zoom-in operation; Determining an image area to be displayed in the first acquired video according to the coordinates of the position; Tracking the image area to be displayed and displaying the image area to be displayed in a specific window to obtain the first processed video; wherein when the area corresponding to the first specific point moves, determining the corresponding coordinate change in the specific window, the first specific point is the center point of the specific window; Analyzing the obtained second acquisition video; In the case where there is an occluded area in the area corresponding to the first specific point in the second captured video, a display strategy is formulated; the display strategy at least includes: displaying the visual effect after the first processing for the unoccluded area; and displaying the visual effect without the first processing for the occluded area; The second acquired video or the first processed video is processed according to the display strategy to obtain a second processed video and display the second processed video.

2. The method according to claim 1, characterized in that: If the first operation is to add a decoration pattern to the first specific point, The performing a first processing on the first captured video according to the first operation to obtain a first processed video and displaying the first processed video includes: according to the operation of adding a dressing pattern, adding a dressing pattern to an area corresponding to the first specific point in the first captured video to obtain a first processed video.

3. The method according to claim 2, characterized in that The adding of the dressing pattern to the area corresponding to the first specific point in the first captured video according to the operation of adding the dressing pattern to obtain a first processed video includes: Determine the location of the operation of adding the dress pattern; Determining a dressing area to which a pattern is to be added on the first captured video according to the position; Adding a dressing pattern to the dressing area to which the pattern is to be added, to obtain the first processed video.

4. The method according to claim 1, characterized in that: Processing the first processed video according to the display strategy to obtain a second processed video includes: Adding a dressing pattern to the dressing area in the first processed video, and not displaying the dressing pattern of the blocked area in the dressing area, to obtain a second processed video; or, A dressing pattern is added to an unblocked area in the dressing area in the first processed video.

5. The method according to claim 4, characterized in that The step of adding a dressing pattern to the dressing area in the first processed video and not displaying the dressing pattern of the blocked area in the dressing area to obtain the second processed video comprises: Adopting a three-dimensional fitting algorithm to fit the dress pattern to the corresponding dress area; Retouching the attached dressing pattern to remove the dressing pattern in the blocked area of ​​the dressing area; After the image editing is completed or during the image editing process, the dressing pattern of the blocked area in the dressing area is not displayed to obtain a second processed video.

6. The method according to claim 1, characterized in that The determining, in the first acquired video according to the coordinates of the position, an image area to be displayed includes: Identifying key points of a target object in the first captured video; Determining a specific point on the target object corresponding to the position according to a relative position relationship between the coordinates of the position and the key point; Determining first coordinates of the specific point in the first captured video; Determining an image area to be displayed in the first acquired video according to the first coordinates; Correspondingly, the tracking of the image area to be displayed and displaying the image area to be displayed in a specific window to obtain the first processed video includes: tracking the image area to be displayed, and using the specific point as the center point of the specific window, displaying the image area to be displayed in the specific window to obtain the first processed video.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps in the method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Video figure dressing method and device, computing device and computer memory medium

    CN107820027A