Image Processing Method, Image Processing Apparatus, and Storage Medium

Abstract

An image processing method, an image processing device, and a storage medium. The image processing method includes: obtaining facial key points and a face orientation of a face in an input image; obtaining a nose bridge line of the face based on the facial key points; obtaining at least one adjusted nose bridge line based on the face orientation and the nose bridge line; and performing a deformation process on the face in the input image according to the nose bridge line before adjustment and the at least one adjusted nose bridge line to achieve nose augmentation. The image processing method can achieve the effect of nose augmentation and beauty for the face in the image.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to an image processing method, an image processing apparatus, and a storage medium. Background Art

[0002] With the continuous improvement of the level of electronic technology, face beautification has become a common function in many photo-taking or photo-editing software. For example, traditional beautification algorithms can usually implement multiple functions such as enlarging eyes and slimming the face, adjusting skin color, skin smoothing and whitening, removing spots and acne, lightening dark circles under the eyes, and slimming the nose, greatly improving the beautification effect of pictures. Therefore, it is more and more widely used in various electronic devices for users to select and use. Such electronic devices can be, for example, smartphones, tablet computers, digital cameras, etc.

[0003] The nose occupies an important position among the facial features of a human face. A good nose shape can effectively improve the appearance value of a user, that is, it is more in line with the aesthetic taste of the target audience (group), and the target audience can be the audience in a certain region, the audience with a certain cultural background, the audience with a certain occupation, etc. Improving a person's nose shape through rhinoplasty can improve this person's appearance value. Then, in the process of face beautification, how to achieve rhinoplasty has become a problem to be solved. Summary of the Invention

[0004] At least one embodiment of the present disclosure provides an image processing method, including: obtaining facial key points and a face orientation of a human face in an input image; obtaining a nasal bridge line of the human face based on the facial key points; obtaining at least one adjusted nasal bridge line based on the face orientation and the nasal bridge line; and performing a deformation process on the human face in the input image according to the nasal bridge line before adjustment and the at least one adjusted nasal bridge line to achieve rhinoplasty.

[0005] For example, in the image processing method provided in an embodiment of the present disclosure, the facial key points include nasal bridge key points, and obtaining the nasal bridge line of the human face based on the facial key points includes: extracting the nasal bridge key points from the facial key points, and fitting the nasal bridge line of the human face based on the nasal bridge key points.

[0006] For example, in the image processing method provided in an embodiment of the present disclosure, the facial key points at least include nasal shape contour key points, and obtaining the nasal bridge line of the human face based on the facial key points includes: extracting a left nasal wing key point from the nasal shape contour key points; extracting a right nasal wing key point from the nasal shape contour key points; obtaining a nasal bridge key point based on the left nasal wing key point and the right nasal wing key point; and fitting the nasal bridge line of the human face based on the nasal bridge key point.

[0007] For example, in the image processing method provided in an embodiment of the present disclosure, obtaining a nasal bridge key point based on the left nasal wing key point and the right nasal wing key point includes:

[0008] x0 = (x1 + x2) / 2

[0009] Wherein, x0 represents the key point of the nose bridge, x1 represents the key point of the left nostril, and x2 represents the key point of the right nostril that is symmetric to the key point of the left nostril.

[0010] For example, in the image processing method provided in an embodiment of the present disclosure, based on the face orientation and the nose bridge line, at least one adjusted nose bridge line is obtained, including: determining a tip point and a root point on the nose bridge line, and rotating the nose bridge line with the tip point and / or the root point as the rotation center to obtain the at least one adjusted nose bridge line.

[0011] For example, in the image processing method provided in an embodiment of the present disclosure, rotating the nose bridge line with the tip point and / or the root point as the rotation center to obtain the at least one adjusted nose bridge line includes: rotating the nose bridge line with the tip point as the rotation center by a first angle in a plane passing through the nose bridge line and perpendicular to the main plane of the face to obtain one adjusted nose bridge line; and / or rotating the nose bridge line with the root point as the rotation center by a second angle in a plane passing through the nose bridge line and perpendicular to the main plane of the face to obtain one adjusted nose bridge line.

[0012] For example, in the image processing method provided in an embodiment of the present disclosure, rotating the nose bridge line with the tip point and / or the root point as the rotation center to obtain the at least one adjusted nose bridge line further includes: dividing the face orientation in the three-dimensional space into multiple intervals, respectively determining corresponding first region angles and second region angles for each of the multiple intervals; determining the interval to which the face orientation of the face in the input image belongs among the multiple intervals, so that the first angle and the second angle are respectively the first region angle and the second region angle corresponding to the belonging interval.

[0013] For example, in the image processing method provided in an embodiment of the present disclosure, the first region angle and the second region angle corresponding to each of the multiple intervals are determined by a predefined or dynamically adjusted method.

[0014] For example, in the image processing method provided in an embodiment of the present disclosure, according to the nose bridge line before adjustment and the at least one adjusted nose bridge line, performing a deformation process on the face in the input image to achieve rhinoplasty, including: determining original control points on the nose bridge line before adjustment, and respectively determining target control points corresponding one-to-one to the original control points on the at least one adjusted nose bridge line; performing a deformation process on the face in the input image according to the original control points and the target control points to achieve rhinoplasty.

[0015] For example, in the image processing method provided in an embodiment of the present disclosure, performing a deformation process on a human face in the input image to achieve nose augmentation according to the original control points and the target control points includes: performing a meshing process on the input image to obtain a mesh image; and performing a deformation process on the mesh image according to the multiple original control points and the multiple target control points to obtain an image after nose augmentation.

[0016] At least one embodiment of the present disclosure further provides an image processing apparatus, including: a human face key point detection unit configured to obtain the human face key points and the human face orientation of a human face in an input image; a nose bridge line positioning unit configured to obtain the nose bridge line of the human face based on the human face key points; a nose bridge adjustment unit configured to obtain at least one adjusted nose bridge line based on the human face orientation and the nose bridge line; and an image deformation unit configured to perform a deformation process on the human face in the input image to achieve nose augmentation according to the nose bridge line before adjustment and the at least one adjusted nose bridge line.

[0017] For example, the image processing apparatus provided in an embodiment of the present disclosure further includes: a control point generation unit configured to determine original control points on the nose bridge line before adjustment and respectively determine target control points corresponding one-to-one to the original control points on the at least one adjusted nose bridge line.

[0018] At least one embodiment of the present disclosure further provides an image processing apparatus, including: a processor; a memory; and one or more computer program modules, where the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include instructions for executing the image processing method provided in any embodiment of the present disclosure.

[0019] At least one embodiment of the present disclosure further provides a storage medium that non-temporarily stores computer-readable instructions, and when the non-temporary computer-readable instructions are executed by a computer, they can execute the instructions of the image processing method provided in any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description only relate to some embodiments of the present invention and do not limit the present invention.

[0021] Figure 1 It is a flowchart of an image processing method provided in an embodiment of the present disclosure;

[0022] Figure 2 It is a schematic diagram of a nose augmentation operation provided in an embodiment of the present disclosure;

[0023] Figure 3 For Figure 1 a flowchart of an example of step S120 shown in

[0024] Figure 4 For Figure 1 a flowchart of an example of step S130 shown in

[0025] Figure 5 For Figure 1 a flowchart of another example of step S130 shown in

[0026] Figure 6 For Figure 1 a flowchart of another example of step S140 shown in

[0027] Figure 7 a schematic diagram of a control point generation operation provided by an embodiment of the present disclosure;

[0028] Figure 8 For Figure 6 a flowchart of an example of step S142 shown in

[0029] Figure 9 a system flowchart of an image processing method provided by an embodiment of the present disclosure;

[0030] Figure 10A a schematic block diagram of an image processing device provided by an embodiment of the present disclosure; and

[0031] Figure 10B a schematic block diagram of another image processing device provided by an embodiment of the present disclosure. Detailed implementation manners

[0032] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the described embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0033] Unless otherwise defined, technical or scientific terms used in this disclosure shall have the ordinary meanings as understood by those of ordinary skill in the art to which this invention belongs. The terms "first", "second" and similar terms used in this disclosure do not denote any order, quantity or importance, but are only used to distinguish different components. Similarly, the terms such as "a", "an" or "the" do not denote a limitation of quantity, but mean that there is at least one. The terms such as "comprising" or "including" mean that the elements or items appearing before this term cover the elements or items listed after this term and their equivalents, without excluding other elements or items. The terms such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms such as "upper", "lower", "left" and "right" are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0034] The following describes this disclosure through several specific embodiments. To keep the following description of the embodiments of the present invention clear and concise, the detailed descriptions of known functions and known components may be omitted. When any component of the embodiments of the present invention appears in more than one drawing, the component is denoted by the same reference numeral in each drawing.

[0035] An embodiment of this disclosure provides an image processing method, including: obtaining face key points and face orientation of a face in an input image; obtaining a nose bridge line of the face based on the face key points; obtaining at least one adjusted nose bridge line based on the face orientation and the nose bridge line; and performing a deformation process on the face in the input image according to the nose bridge line before adjustment and the at least one adjusted nose bridge line to achieve nose augmentation. At least one embodiment of this disclosure further provides an image processing apparatus and a storage medium corresponding to the above image processing method.

[0036] The image processing method provided by the above embodiment of this disclosure can estimate the shape of the nose bridge in three-dimensional space by analyzing the key points of the nose bridge or nose wing part of the face in the input image and in cooperation with the three-dimensional orientation of the face, and perform nose augmentation operations such as padding the root of the nose and / or lifting the tip of the nose based on this shape, so as to simulate a nose augmentation surgery in a real environment and achieve the effect of nose augmentation and beauty enhancement for the face in the input image.

[0037] The following describes in detail the embodiments of this disclosure and some of their examples with reference to the drawings.

[0038] Figure 1A flowchart of an example of an image processing method provided by an embodiment of the present disclosure. The image processing method can be implemented in software or hardware, and is loaded and executed by a processor in a device such as a mobile phone, a laptop computer, a desktop computer, a network server, a digital camera, etc., to achieve the effect of rhinoplasty beauty for the face in the input image and output the image after beauty processing. As Figure 1 shown, the image processing method includes steps S110 to S140.

[0039] Step S110: Obtain the facial key points and the facial orientation of the face in the input image.

[0040] Step S120: Obtain the nasal bridge line of the face based on the facial key points.

[0041] Step S130: Obtain at least one adjusted nasal bridge line based on the facial orientation and the nasal bridge line.

[0042] Step S140: Perform a deformation process on the face in the input image according to the nasal bridge line before adjustment and at least one adjusted nasal bridge line to achieve rhinoplasty.

[0043] In step S110, for example, the facial key points can be some key points with strong characterization ability on the face, including but not limited to key points such as eyes (pupils), eye corners, eyebrow tips, the highest points of cheekbones, nose, mouth, chin, and the outer contour of the face. For example, the facial key points of the face in the input image at least include key points of the nasal contour part or nasal bridge key points. For example, the key points of the nasal contour part can include left and right nasal wing key points, etc. For example, the face in the input image can be a male or female face, and can be a face at any angle including the nose.

[0044] For example, the facial orientation includes the orientation of the face in three-dimensional space. For example, it usually can include the yaw orientation (orientation of rotation around the Y axis, yaw), or the pitch orientation (orientation of rotation around the X axis, pitch), and the roll orientation (orientation of rotation around the Z axis, roll).

[0045] For example, a large number of images (e.g., 10,000 or more) including human faces can be collected in advance as a sample library, and a series of key points such as the corners of the eyes, the corners of the mouth, the wing of the nose, the highest point of the cheekbones, and the outer contour points of the human face can be marked on each image manually or by other methods. Then, the classification model is trained and tested using the images in the sample library through algorithms such as machine learning (e.g., deep learning, or regression algorithms based on local features), so as to obtain algorithm models for face detection, human face key point localization, and human face orientation recognition. The input of this model is an image containing a human face, and the output is the key points of the human face and the human face orientation of this human face image, thereby realizing the localization of human face key points and the recognition of human face orientation. It should be noted that this machine learning algorithm can be implemented by conventional methods in the art and will not be elaborated here. Additionally, it should be noted that the methods for face detection, human face key point localization, and human face orientation recognition can also be implemented by other conventional algorithms in the art, and the embodiments of the present disclosure do not limit this.

[0046] For example, face detection can also be implemented by methods based on templates, methods based on models, or neural network methods, etc. Methods based on templates can include, for example, eigenface methods, linear discriminant analysis methods, singular value decomposition methods, dynamic connection matching methods, etc. Methods based on models can include, for example, hidden Markov models, active shape models, and active appearance models, etc. It should be noted that the above methods can be implemented by conventional algorithms in the art and will not be elaborated here.

[0047] For example, the key point extraction in the human face key point detection process can be implemented by conventional algorithms in the art, and the embodiments of the present disclosure do not limit this.

[0048] For example, the input image can be obtained through an appropriate image acquisition device. The image acquisition device can be a digital camera, the camera of a smart phone, the camera of a tablet computer, the camera of a personal computer, a web camera, a surveillance camera, or other components that can implement the function of image acquisition. The embodiments of the present disclosure do not limit this.

[0049] For example, the input image can be an original image directly captured by an image acquisition device, or an image obtained after preprocessing the original image. For example, before step S110, the image processing method provided by the embodiments of the present disclosure may further include an operation of preprocessing the input image to facilitate detecting a human face in the input image. The image preprocessing operation can eliminate irrelevant information or noise information in the input image, so as to better perform human face detection on the input image. For example, when the input image is a photo, the image preprocessing operation may include processing such as image scaling, compression, or format conversion, color gamut conversion, Gamma correction, image enhancement, or noise reduction filtering of the photo. When the input image is a video, the preprocessing may include extracting key frames of the video, etc.

[0050] For example, face detection, face key point localization, and recognition of face orientation can be implemented by a dedicated face key point detection unit, or can also be implemented by a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities. The processing unit can be a general-purpose processor or a dedicated processor, and can be a processor based on the X86 or ARM architecture, etc.

[0051] In step S120, for example, in one example, when the face key points include a nose bridge key point, the nose bridge key point in the face key points can be extracted, and the nose bridge line 10 of the face can be obtained by fitting based on the nose bridge key point (as shown at B in Figure 2 ). As shown in Figure 2 , A is a schematic diagram of the side of the nose in the human face, and B is a schematic diagram of the nose bridge line fitted based on the face key points. For example, the nose bridge key point can be implemented by the face key point detection algorithm introduced in step S110, which will not be elaborated here. For example, the fitting curve of the nose bridge line can be obtained by the least squares method, and this method can be implemented by conventional methods in the art, which will not be elaborated here. It should be noted that the fitting method for the nose bridge key point can also be implemented by other conventional methods in the art, and the embodiments of the present disclosure do not limit this.

[0052] For example, in another example, the face key points at least include nose shape contour key points (for example, it can include a left nostril key point and a right nostril key point). For example, in this example, the face key points include multiple left nostril key points and right nostril key points, and do not include a nose bridge key point. In this case, for example, the nose bridge key point can be calculated by interpolation of the nose shape contour key points first, and then the nose bridge line of the face can be fitted based on the nose bridge key point.

[0053] Figure 3 shows a flowchart of a method for obtaining a nose bridge line through nose shape contour key points. That is, Figure 3 isFigure 1 Another example flowchart of step S120 shown in the figure. As Figure 3 shown, the method for obtaining the nasal bridge line includes steps S121 to S124. Below, with reference to Figure 3 the method for obtaining the nasal bridge line will be described.

[0054] Step S121: Extract the left nasal wing key point among the key points of the nasal shape contour.

[0055] For example, the left nasal wing key point among the key points of the nasal shape contour can be extracted through the face key point detection algorithm (for example, a classification model trained by a machine learning algorithm) introduced in the above step S110, which will not be elaborated here.

[0056] Step S122: Extract the right nasal wing key point among the key points of the nasal shape contour.

[0057] For example, the right nasal wing key point is symmetrical to the left nasal wing key point, and the right nasal wing key point among the key points of the nasal shape contour can also be extracted through the above face key point detection algorithm, which will not be elaborated here.

[0058] Step S123: Obtain the nasal bridge key point based on the left nasal wing key point and the right nasal wing key point.

[0059] For example, the interpolation calculation formula for obtaining the nasal bridge key point based on the left nasal wing key point and the right nasal wing key point can be expressed as:

[0060] x0 = (x1 + x2) / 2

[0061] where x0 represents the nasal bridge key point, x1 represents the left nasal wing key point, and x2 represents the right nasal wing key point symmetrical to the left nasal wing key point.

[0062] Step S124: Fit the nasal bridge line of the human face based on the nasal bridge key point.

[0063] For example, if all the left nasal wing key points and right nasal wing key points are calculated through the above formula, some nasal bridge key points can be obtained, and then the fitting method (for example, the least squares method) in the above example is used to fit all the nasal bridge key points to obtain the nasal bridge line 10 of the human face.

[0064] For example, the positioning of the nasal bridge line of the human face can be achieved through a dedicated nasal bridge line positioning unit, or can also be achieved through a central processing unit (CPU), an image processor (GPU), a field programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities.

[0065] In step S130, for example, first determine the tip point and the root point of the nose on the nose bridge line, and then rotate the nose bridge line with the tip point and / or the root point as the rotation center to obtain at least one adjusted nose bridge line. For example, the tip point and the root point of the nose are usually the two ends of the nose bridge line, as shown by C in Figure 2 , where P represents the tip point on the nose bridge line, as shown by D in Figure 2 , and Q represents the root point of the nose on the nose bridge line. In this example, the direction and angle of rotation of the nose bridge line are related to the face orientation, and the specific operation process will be introduced in detail in the following examples.

[0066] It should be noted that the rotation center of the nose bridge line can be any appropriate point such as the tip point, the root point, or the center point on the nose bridge line, and the embodiments of the present disclosure are not limited thereto.

[0067] Figure 4 The flowchart of the method for obtaining the adjusted nose bridge line is shown. That is, Figure 4 is Figure 1 a flowchart of an example of step S130 shown in Figure 4 . As shown in Figure 4 , the method for obtaining the adjusted nose bridge line includes step S131 and / or step S132. Below, refer to Figure 4 to describe the method for obtaining the adjusted nose bridge line.

[0068] Step S131: Rotate the nose bridge line with the tip point as the rotation center by a first angle in a plane passing through the nose bridge line and perpendicular to the main plane of the face to obtain an adjusted nose bridge line.

[0069] For example, as shown by C in Figure 2 , rotate the nose bridge line 10 with the tip point P as the rotation center to obtain an adjusted nose bridge line 11. For example, this operation is the operation of padding the root of the nose.

[0070] Step S132: Rotate the nose bridge line with the root point as the rotation center by a second angle in a plane passing through the nose bridge line and perpendicular to the main plane of the face to obtain an adjusted nose bridge line.

[0071] As shown by D in Figure 2 , rotate the nose bridge line 10 with the root point Q as the rotation center to obtain another adjusted nose bridge line 12, and this operation is the operation of lifting the tip of the nose.

[0072] For example, the plane of the main plane of the face can be determined by the pitch orientation (the orientation of rotation around the X axis, pitch) in the three-dimensional orientation of the face. For example, the method for recognizing the three-dimensional orientation of the face can refer to the relevant description in step S110, which will not be elaborated here.

[0073] For example, the first angle and the second angle in the above steps are related to the roll orientation (orientation of rotation about the Z-axis, roll) in the face orientation. Figure 5 A method flowchart for determining the first angle and the second angle is shown. As Figure 5 shown, the method for determining the first angle and the second angle includes steps S133 to S134. Below, with reference to Figure 5 the method for determining the first angle and the second angle in the embodiments of the present disclosure will be described.

[0074] Step S133: Divide the face orientation in a three-dimensional space into multiple intervals, and determine corresponding first regional angles and second regional angles for each of the multiple intervals respectively.

[0075] For example, the face orientation in the three-dimensional space can be divided into 9 intervals according to the roll orientation (orientation of rotation about the Z-axis, roll) in the three-dimensional face orientation (for example, including upper left, left middle, lower left, directly above, middle, directly below, upper right, right middle, lower right). Since the roll orientation (orientation of rotation about the Z-axis, roll) is a rotation about the Z-axis, the angle between the Z-axes of adjacent intervals can be set according to the actual situation, for example, set to 45°. For example, the face orientation in the upper left interval is, relative to the face orientation in the middle interval, for example, 45° to the left and 45° upward, and the face orientation in the lower left interval is, relative to the face orientation in the middle interval, for example, 45° to the left and 45° downward looking, and so on. For example, the face orientation in the middle interval is the direction of looking straight at the screen, that is, the Z-axis in the middle interval is perpendicular to the direction of the screen, and the face orientation of each interval is allowed to fluctuate within a certain range. It should be noted that this embodiment may also include more intervals, and the angle between the Z-axes of adjacent intervals may also be 30°, 60°, etc. The number of intervals and the angle can be determined according to the specific situation, and the embodiments of the present disclosure do not limit this.

[0076] For example, the first region angle and the second region angle corresponding to each of the above-mentioned multiple intervals can be determined by a pre-defined or dynamically adjusted method, or can be determined by other conventional methods in the art. For example, the first region angle and the second region angle of each interval can be different, or a certain angle in two symmetric intervals can be set to be the same. The specific setting method depends on the specific situation, and the embodiments of the present disclosure do not limit this. For example, the pre-defined first region angle and second region angle can be set to default values, which are suitable for the rhinoplasty effects of the general public. The dynamically adjusted method can adjust the rotation angle of the nasal bridge line in real time according to different faces and different face orientations to achieve the most ideal effect. It should be noted that from a visual effect perspective, the differences generated by the pre-defined and dynamically adjusted methods are not significant, and it is difficult for the human eye to perceive. Therefore, the method for determining the first region angle and the second region angle can be selected according to the actual situation, and the embodiments of the present disclosure do not limit this.

[0077] Step S134: Determine the interval to which the face orientation of the face in the input image belongs among the multiple intervals, so that the first angle and the second angle are respectively the first region angle and the second region angle corresponding to the interval to which it belongs.

[0078] For example, when it is determined that the interval to which the face orientation of the face in the input image belongs among the multiple intervals is the upper left interval, the nasal bridge line rotates around the nasal tip point, and the first angle of rotation in the plane passing through the nasal bridge line and perpendicular to the main plane of the face is the first region angle defined (or dynamically adjusted) within the upper left interval; the nasal bridge line rotates around the nasal root point, and the second angle of rotation in the plane passing through the nasal bridge line and perpendicular to the main plane of the face is the second region angle defined (or dynamically adjusted) within the upper left interval.

[0079] For example, the adjusted nasal bridge line can be obtained through a dedicated nasal bridge adjustment unit, or can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities.

[0080] In step S140, the pre-adjusted nasal bridge line can be deformed to the position of the post-adjusted nasal bridge line through operations such as lifting the nasal tip and / or padding the nasal root, thereby achieving the effect of rhinoplasty. For example, conventional methods in the art such as the mesh warping algorithm can be used to achieve the deformation of the nasal bridge line.

[0081] Figure 6 Shows a flowchart of the deformation processing method. That is, Figure 6 For Figure 1 A flowchart of an example of step S140 shown in Figure 6As shown, the method for deformation processing includes steps S141 to S142. Next, reference will be made to Figure 6 to describe the method for deformation processing.

[0082] Step S141: Determine the original control points on the nasal bridge line before adjustment, and respectively determine the target control points corresponding one-to-one to the original control points on at least one adjusted nasal bridge line.

[0083] For example, the original control points can be obtained by uniformly sampling on the nasal bridge line. Denote the original control points as S(i), where i is an integer greater than 0, representing the number of original control points and target control points. Then connect from the original control points to the adjusted nasal bridge line. For example, it can be parallel connection or perpendicular connection. After connection, the endpoints on the adjusted nasal bridge line are the target control points, denoted as T1(i). For example, Figure 7 E in Figure 7 shows the original control points and target control points in the operation of padding the root of the nose; Figure 7 F in

[0084] shows the original control points and target control points in the operation of lifting the tip of the nose. As shown by E and F in

[0085] For example, the input image is uniformly divided into grids to obtain the original grid image. The original control points S(i) on the original grid image are deformed and distorted along the direction from the original control points S(i) to the target control points T1(i) (i.e., the direction of the movement vector) by using the grid distortion algorithm to achieve rhinoplasty. For example, the movement vector is used to move the original control points to the positions of the target control points. It should be noted that the grid distortion algorithm and the calculation of the movement vector in the grid distortion algorithm can be implemented by conventional operations in the art and will not be elaborated here.

[0086] For example, when only the operation of lifting the tip of the nose or only the operation of padding the root of the nose is performed on the input image, there is only one adjusted nasal bridge line. For example, there is only the adjusted nasal bridge line 11 or the adjusted nasal bridge line 12. The original control points S(i) in the original grid image can be directly deformed and distorted along the movement vector by using the grid distortion algorithm to achieve rhinoplasty.

[0087] For example, when both the nose tip lifting operation and the nasal root padding operation are performed on the input image, there are two adjusted nasal bridge lines, for example, including the adjusted nasal bridge line 11 and the adjusted nasal bridge line 12. At this time, the global optimization of the adjusted nasal bridge line 11 and the adjusted nasal bridge line 12 can be performed through the mesh distortion algorithm to fuse the movement vectors of the original control points. For example, at the position where the operation force of lifting the nose tip on the original mesh image is greater than the operation force of padding the nasal root, the distance (i.e., the degree of deformation) that the original control point that needs to perform the nose tip lifting operation moves along the direction of the movement vector can be 80% of the length of the original movement vector, and the distance that the original control point that needs to perform the nasal root padding operation moves along the movement vector is 20% of the length of the original movement vector. The optimization degree of this movement vector can be determined according to specific circumstances, and the embodiments of the present disclosure do not limit this. It should be noted that the operation method for globally optimizing the movement vector in this mesh distortion algorithm can be implemented by using a conventional algorithm in the art, and will not be elaborated here.

[0088] When there are multiple (for example, more than two) adjusted nasal bridge lines, the same method can also be used to achieve rhinoplasty.

[0089] Figure 8 For Figure 6 a flowchart of an example of step S142 shown in. As Figure 8 shown, the method for deformation processing further includes step S1421 and step S1422. Next, refer to Figure 8 to describe the method for deformation processing.

[0090] Step S1421: Perform grid processing on the input image to obtain a grid image.

[0091] For example, performing grid processing on the input image, that is, adding uniformly spaced grids to the input image to obtain a grid image, and this grid image is the original grid image.

[0092] Step S1422: Perform deformation processing on the grid image according to multiple original control points and multiple target control points to obtain the image after rhinoplasty.

[0093] For example, perform deformation distortion on the original control points on the original grid image along the movement vector through the mesh distortion algorithm to obtain the corrected grid image, that is, the image after rhinoplasty, and achieve the rhinoplasty effect.

[0094] For example, the deformation distortion of the image can be realized through a dedicated image deformation unit, or can also be realized through a central processing unit (CPU), an image processor (GPU), a field programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities.

[0095] It should be noted that in the embodiments of the present disclosure, the process of the image processing method may include more or fewer operations, and these operations may be executed sequentially or in parallel. Although the process of the image processing method described above includes multiple operations that appear in a specific order, it should be clearly understood that the order of the multiple operations is not limited. The image processing method described above may be executed once or multiple times according to a predetermined condition.

[0096] The image processing method provided by the embodiments of the present disclosure can estimate the shape of the nose bridge in three-dimensional space by analyzing the key points of the nose bridge or the nose wing part of the face in the input image and in combination with the three-dimensional orientation of the face, and perform rhinoplasty operations such as padding the nasal root and / or lifting the nasal tip on the nose bridge line based on this shape, so as to simulate rhinoplasty surgery in a real environment and achieve the effect of rhinoplasty and beauty enhancement for the face in the input image.

[0097] Figure 9 It is a system flowchart for implementing an image processing method provided by an embodiment of the present disclosure. As Figure 9 shown, the system for implementing the image processing method provided by the embodiments of the present disclosure includes an input unit 11, a face key point detection unit 12, a nose bridge line positioning unit 13, a rhinoplasty unit 14, a control point generation unit 15, an image deformation unit 16, and an output unit 17. For example, each of these units can be implemented by a hardware (such as a circuit) module or a software module, etc.

[0098] For example, the input unit 11 is configured to input an input image including a face. For example, the input image including a face includes multiple key points, such as the corners of a person's eyes, the corners of the mouth, the nose wings, the highest points of the cheekbones, etc. For example, the input image can be obtained by an image acquisition device and transmitted to the face key point detection unit 12. The image acquisition device may include a camera of a smart phone, a camera of a tablet computer, a camera of a personal computer, a digital camera, or a web camera, etc.

[0099] The face key point detection unit 12 receives the image transmitted by the input unit 11, and obtains the face key points and the face orientation of the face in the input image through a classification model trained by, for example, a machine learning algorithm, and sends the obtained information of the face key points and the face orientation to the nose bridge line positioning unit 13. For example, the face key point detection unit 12 can implement step S110, and the specific implementation method can refer to the relevant description of step S110, which will not be elaborated here.

[0100] The nose bridge line positioning unit 13 can determine the nose bridge line of a human face based on the key points of the human face obtained in the human face key point detection unit 12, such as the nose bridge key point or the nose shape contour key points (the left nostril key point and the right nostril key point). For example, the nose bridge line positioning unit 13 can be implemented as step S120, and the specific implementation method can refer to the relevant description of step S120, which will not be elaborated here.

[0101] The nose augmentation unit 14 can determine the interval to which the face orientation of the human face in the input image belongs according to the face orientation obtained in the human face key point detection unit 12, and rotate the nose bridge line of the human face obtained in the nose bridge line positioning unit 13 around the tip of the nose point or the root of the nose point or other appropriate points by a first angle (i.e., the first region angle of the interval to which the human face belongs) or a second angle (i.e., the second region angle of the interval to which the human face belongs) in a plane passing through the nose bridge line and perpendicular to the main plane of the human face to obtain an adjusted nose bridge line. The nose augmentation unit 14 can obtain at least one adjusted nose bridge line after performing the operation of lifting the tip of the nose or padding the root of the nose. For example, the nose augmentation unit 14 can be implemented as step S130, and the specific implementation method can refer to the relevant description of step S130, which will not be elaborated here.

[0102] The control point generation unit 15 is configured to determine the original control points S(i) on the nose bridge line before adjustment, and determine the target control points T1(i) corresponding one-to-one to the original control points S(i) on at least one adjusted nose bridge line respectively. For example, the control point generation unit 15 can be implemented as step S141, and the specific implementation method can refer to the relevant description of step S141, which will not be elaborated here.

[0103] The deformation unit 16 can achieve nose augmentation by adopting a mesh warping algorithm. For example, the deformation unit 16 first performs a meshing process on the input image, that is, adds a uniformly spaced mesh to the input image to obtain an original mesh image, and then performs a deformation process on the original control points S(i) in the original mesh image along the movement vector to obtain an image after nose augmentation, achieving the nose augmentation effect. This unit can implement step S142, and the specific implementation method can refer to the relevant description of step S142, which will not be elaborated here.

[0104] For example, the combination of the control point generation unit 15 and the deformation unit 16 can implement step S140.

[0105] The output unit 17 is configured to output the image after nose augmentation, that is, the image after the deformation unit 16 distorts and deforms the input image.

[0106] It should be noted that, for the sake of clarity and conciseness, the embodiments of the present disclosure do not show all the constituent units of the system for implementing the image processing method. To implement the image processing method, those skilled in the art can provide and set other constituent units not shown according to specific needs, and the embodiments of the present disclosure do not limit this. It should be noted that the above-mentioned various units can be implemented by software, firmware, hardware (such as FPGA), or any combination thereof.

[0107] Figure 10A The following is a schematic block diagram of an image processing apparatus provided by an embodiment of the present disclosure. As Figure 10A shown, the image processing apparatus 100 includes a face key point detection unit 110, a nose bridge line positioning unit 120, a nose bridge adjustment unit 130, and an image deformation unit 140. For example, these units can be implemented by hardware (such as circuits) modules or software modules, etc.

[0108] The face key point detection unit 110 is configured to obtain the face key points and the face orientation of the face in the input image. For example, the face key point detection unit 110 can implement step S110, and the specific implementation method can refer to the relevant description of step S110, which will not be elaborated here.

[0109] The nose bridge line positioning unit 120 is configured to obtain the nose bridge line of the face based on the face key points. For example, the nose bridge line positioning unit 120 can implement step S120, and the specific implementation method can refer to the relevant description of step S120, which will not be elaborated here.

[0110] The nose bridge adjustment unit 130 is configured to obtain at least one adjusted nose bridge line based on the face orientation and the nose bridge line. For example, the nose bridge adjustment unit 130 can implement step S130, and the specific implementation method can refer to the relevant description of step S130, which will not be elaborated here.

[0111] The image deformation unit 140 is configured to perform a deformation process on the face in the input image according to the nose bridge line before adjustment and at least one adjusted nose bridge line to achieve rhinoplasty. For example, the image deformation unit 140 can implement step S140, and the specific implementation method can refer to the relevant description of step S140, which will not be elaborated here.

[0112] For example, the image processing apparatus 100 further includes a control point generation unit (not shown in the figure). The control point generation unit is configured to determine original control points on the nose bridge line before adjustment, and respectively determine target control points corresponding one-to-one to the original control points on at least one adjusted nose bridge line. For example, the control point generation unit can implement step S141, and the specific implementation method can refer to the relevant description of step S141, which will not be elaborated here.

[0113] It should be noted that in the embodiments of the present disclosure, more or fewer circuits or units may be included, and the connection relationships between the respective circuits or units are not limited and may be determined according to actual requirements. The specific composition manners of the respective circuits are not limited and may be constituted by analog devices according to circuit principles, may be constituted by digital chips, or may be constituted in other applicable manners.

[0114] Figure 10B FIG. is a schematic block diagram of another image processing apparatus provided in an embodiment of the present disclosure. As Figure 10B shown, the image processing apparatus 200 includes a processor 210, a memory 220, and one or more computer program modules 221.

[0115] For example, the processor 210 and the memory 220 are connected through a bus system 230. For example, one or more computer program modules 221 are stored in the memory 220. For example, one or more computer program modules 221 include instructions for executing the image processing method provided in any embodiment of the present disclosure. For example, the instructions in one or more computer program modules 221 can be executed by the processor 210. For example, the bus system 230 can be a common serial or parallel communication bus, etc., and the embodiments of the present disclosure do not limit this.

[0116] For example, the processor 210 can be a central processing unit (CPU), an image processing unit (GPU), a field programmable gate array (FPGA), or other forms of processing units having data processing capabilities and / or instruction execution capabilities, can be a general-purpose processor or a dedicated processor, and can control other components in the image processing apparatus 200 to perform desired functions.

[0117] The memory 220 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory, etc. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 210 may run the program instructions to implement the functions (implemented by the processor 210) in the embodiments of the present disclosure and / or other desired functions, such as an image processing method, etc. Various application programs and various data may also be stored in the computer-readable storage medium, such as nose bridge key points, original control points, target control points, motion vectors, and various data used and / or generated by the application programs, etc.

[0118] It should be noted that, for the sake of clarity and conciseness, the embodiments of the present disclosure do not show all the constituent units of the image processing apparatus 200. To implement the necessary functions of the image processing apparatus 200, those skilled in the art can provide and set other constituent units not shown according to specific needs, and the embodiments of the present disclosure do not limit this.

[0119] Regarding the technical effects of the image processing apparatus 100 and the image processing apparatus 200 in different embodiments, reference can be made to the technical effects of the image processing method provided in the embodiments of the present disclosure, which will not be elaborated here.

[0120] An embodiment of the present disclosure further provides a storage medium. For example, the storage medium stores computer-readable instructions non-transitorily, and when the non-transitory computer-readable instructions are executed by a computer (including a processor), the image processing method provided in any embodiment of the present disclosure can be executed.

[0121] For example, the storage medium can be any combination of one or more computer-readable storage media. For example, one computer-readable storage medium contains computer-readable program code for face key point detection, and another computer-readable storage medium contains computer-readable program code for nose bridge line positioning. For example, when the program code is read by a computer, the computer can execute the program code stored in the computer storage medium and execute, for example, the image processing method provided in any embodiment of the present disclosure.

[0122] For example, the storage medium can include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a flash memory, or any combination of the above storage media, and can also be other applicable storage media.

[0123] The following points need to be noted:

[0124] (1) The drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure, and other structures can refer to the general design.

[0125] (2) Without conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0126] The above is only an exemplary implementation manner of the present disclosure, rather than used to limit the protection scope of the present disclosure. The protection scope of the present disclosure is determined by the appended claims.

Claims

1. An image processing method, comprising: obtaining facial key points and a facial orientation of a face in a two-dimensional input image; obtaining a nasal bridge line of the face based on the facial key points; determining a rotation angle and a rotation direction of the nasal bridge line based on the facial orientation; rotating the nasal bridge line by the rotation angle in the rotation direction to obtain at least one adjusted nasal bridge line; and performing a deformation process on the face in the input image according to the nasal bridge line before adjustment and the at least one adjusted nasal bridge line to achieve rhinoplasty; wherein, determining the rotation angle of the nasal bridge line based on the facial orientation includes: dividing the facial orientation in a three-dimensional space into multiple intervals, and respectively determining corresponding regional angles for each of the multiple intervals; judging the interval to which the facial orientation of the face in the two-dimensional input image belongs among the multiple intervals, so that the rotation angle is the regional angle corresponding to the belonging interval.

2. The image processing method according to claim 1, wherein, the facial key points include nasal bridge key points, and obtaining the nasal bridge line of the face based on the facial key points includes: extracting the nasal bridge key points from the facial key points, and fitting the nasal bridge line of the face based on the nasal bridge key points.

3. The image processing method according to claim 1, wherein, the facial key points at least include nasal contour key points, and obtaining the nasal bridge line of the face based on the facial key points includes: extracting a left nasal wing key point from the nasal contour key points; extracting a right nasal wing key point from the nasal contour key points; obtaining nasal bridge key points based on the left nasal wing key point and the right nasal wing key point; fitting the nasal bridge line of the face based on the nasal bridge key points.

4. The image processing method according to claim 3, wherein, obtaining nasal bridge key points based on the left nasal wing key point and the right nasal wing key point includes: x0 = (x1 + x2) / 2 wherein, x0 represents the nasal bridge key point, x1 represents the left nasal wing key point, and x2 represents the right nasal wing key point symmetric to the left nasal wing key point.

5. The image processing method according to any one of claims 1-4, wherein, obtaining at least one adjusted nasal bridge line based on the facial orientation and the nasal bridge line includes: determining a tip point and a root point on the nasal bridge line, and rotating the nasal bridge line with the tip point and / or the root point as a rotation center to obtain the at least one adjusted nasal bridge line.

6. The image processing method according to claim 5, wherein, rotating the nasal bridge line with the tip point and / or the root point as a rotation center to obtain the at least one adjusted nasal bridge line includes: rotating the nasal bridge line with the tip point as a rotation center by a first angle in a plane passing through the nasal bridge line and perpendicular to the main plane of the face to obtain one adjusted nasal bridge line; and / or rotating the nasal bridge line with the root point as a rotation center by a second angle in a plane passing through the nasal bridge line and perpendicular to the main plane of the face to obtain one adjusted nasal bridge line.

7. The image processing method according to claim 6, Wherein: The region angles respectively determined for each of the multiple intervals include a first region angle and a second region angle; the first angle and the second angle are respectively the first region angle and the second region angle corresponding to the interval to which they belong.

8. The image processing method according to claim 7, wherein, The first region angle and the second region angle corresponding to each of the multiple intervals are determined by a method of predefined or dynamic adjustment.

9. The image processing method according to any one of claims 1-4, 6-8, wherein, Performing a deformation process on the face in the two-dimensional input image to achieve nose augmentation according to the nose bridge line before adjustment and the at least one adjusted nose bridge line, includes: Determining original control points on the nose bridge line before adjustment, and respectively determining target control points corresponding one-to-one to the original control points on the at least one adjusted nose bridge line; Performing a deformation process on the face in the two-dimensional input image according to the original control points and the target control points to achieve nose augmentation.

10. The image processing method according to claim 9, wherein, Performing a deformation process on the face in the two-dimensional input image according to the original control points and the target control points to achieve nose augmentation, includes: Performing a grid processing on the two-dimensional input image to obtain a grid image; Performing a deformation process on the grid image according to the multiple original control points and the multiple target control points to obtain an image after nose augmentation.

11. An image processing apparatus, comprising: A face key point detection unit configured to obtain face key points and face orientation of a face in a two-dimensional input image; A nose bridge line positioning unit configured to obtain the nose bridge line of the face based on the face key points; A nose bridge adjustment unit configured to determine a rotation angle and a rotation direction of the nose bridge line based on the face orientation, and rotate the nose bridge line by the rotation angle in the rotation direction to obtain at least one adjusted nose bridge line; and An image deformation unit configured to perform a deformation process on the face in the two-dimensional input image according to the nose bridge line before adjustment and the at least one adjusted nose bridge line to achieve nose augmentation; wherein, determining the rotation angle of the nose bridge line based on the face orientation includes: Dividing the face orientation in a three-dimensional space into multiple intervals, and respectively determining corresponding region angles for each of the multiple intervals; Judging the interval to which the face orientation of the face in the two-dimensional input image belongs among the multiple intervals, so that the rotation angle is the region angle corresponding to the interval to which it belongs.

12. The image processing apparatus according to claim 11, further comprising: A control point generation unit configured to determine original control points on the nose bridge line before adjustment, and respectively determine target control points corresponding one-to-one to the original control points on the at least one adjusted nose bridge line.

13. An image processing apparatus, comprising: A processor; A memory; One or more computer program modules, the one or more computer program modules being stored in the memory and configured to be executed by the processor, the one or more computer program modules including instructions for performing the image processing method according to any one of claims 1-10.

14. A storage medium that non-transitorily stores computer-readable instructions that, when executed by a computer, can execute the instructions of the image processing method according to any one of claims 1-10.

Images