Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention is suitable for the technical field of computers, and provides an image processing method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a face frame of an original face image, the original face image comprising a face in a non-front face state; based on a preset expansion proportion, expanding the face frame to obtain an expansion frame of the original face image; cutting the original face image based on the expansion frame to obtain an expanded face image; the face in the expanded face image is corrected to obtain a corrected face image, and the corrected face image comprises a face in a front face state; and based on the expansion frame and the face frame, cutting the corrected face image to obtain a target face image. According to the method and the device, the face recognition precision can be improved on the premise of ensuring the integrity of the face information in the image.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to an image processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Currently, face recognition technology is widely used in identity authentication, security inspection, access control, payment and other application systems. Face image preprocessing, as an integral part of face recognition applications, usually crops the face image based on the face frame obtained in the face detection step. If the cropped face is not a frontal face, further correction is required before subsequent face recognition operations can be performed.

[0003] In the prior art, after face detection obtains a face frame, the face within the face frame is first cropped, that is, only the face within the face frame is retained, and then face correction is performed. For a face that is not in the front view, there is a situation where the face frame cannot completely frame the face. At this time, if a screenshot is taken according to the face frame, part of the face information will be lost, resulting in incomplete face information in the image, which reduces the accuracy of face recognition. Summary of the invention

[0004] In view of this, the embodiments of the present application provide an image processing method, device, electronic device and computer-readable storage medium, which can improve the accuracy of face recognition while ensuring the integrity of face information in the image.

[0005] A first aspect of an embodiment of the present application provides an image processing method, including:

[0006] Acquire a face frame of an original face image, wherein the original face image includes a face in a non-frontal state;

[0007] Expanding the face frame based on a preset expansion ratio to obtain an expanded frame of the original face image;

[0008] Based on the expansion frame, the original face image is cropped to obtain an expanded face image;

[0009] Correcting the human face in the expanded human face image to obtain a corrected human face image, wherein the corrected human face image includes the human face in a frontal state;

[0010] Based on the expansion frame and the face frame, the corrected face image is cropped to obtain a target face image.

[0011] In an implementation of the first aspect, the expanding the face frame based on a preset expansion ratio to obtain an expanded frame of the original face image includes:

[0012] Based on the preset expansion ratio and the horizontal components of the vertices of the face frame, horizontal expansion is performed to obtain an expanded horizontal component;

[0013] Based on the preset expansion ratio and the vertical components of the vertices of the face frame, vertical expansion is performed to obtain the expanded vertical components;

[0014] An expansion frame of the original face image is determined based on the width of the face frame, the horizontal expansion width, and the vertical expansion width.

[0015] In an implementation of the first aspect, the correcting the face in the expanded face image to obtain the corrected face image includes:

[0016] Determining a face rotation angle based on left eye feature points and right eye feature points in the expanded face image;

[0017] Determining a face center point of the expanded face image based on the face frame and the expanded frame;

[0018] Based on the face rotation angle and the face center point, the expanded face image is rotated to obtain a corrected face image.

[0019] In an implementation of the first aspect, determining the face rotation angle based on the left eye feature points and the right eye feature points in the expanded face image includes:

[0020] Determine a first difference between a vertical component of a left eye feature point and a vertical component of a right eye feature point in the expanded facial image;

[0021] Determine a second difference between a horizontal component of a left eye feature point and a horizontal component of a right eye feature point in the expanded facial image;

[0022] The degree of an angle corresponding to the ratio of the first difference to the second difference is determined as a face rotation angle.

[0023] In an implementation of the first aspect, determining the face center point of the expanded face image based on the face frame and the expanded frame includes:

[0024] determining a third difference between the horizontal components of the vertices of the face frame and the horizontal components of the vertices of the extended frame, and determining a target horizontal component based on the third difference and the width of the face frame;

[0025] determining a fourth difference between the vertical components of the vertices of the face frame and the vertical components of the vertices of the extended frame, and determining a target vertical component based on the fourth difference and the height of the face frame;

[0026] The point determined by the target horizontal component and the target vertical component is used as the face center point of the expanded face image.

[0027] In an implementation of the first aspect, before obtaining the face frame of the original face image, the method further includes:

[0028] Performing face detection on the original face image to obtain a face frame and facial feature points of the original face image;

[0029] Based on the face frame and the facial feature points, a state detection is performed on the face in the original face image to obtain a detection result, and the detection result is used to indicate whether the face is in a frontal state.

[0030] In an implementation of the first aspect, the step of cropping the corrected face image based on the expanded frame and the face frame to obtain a target face image includes:

[0031] Based on the expanded frame and the face frame, determining a restoration frame, wherein the restoration frame is used to mark the position and size of the face frame in the expanded frame;

[0032] Based on the restoration frame, the corrected face image is cropped to obtain a target face image.

[0033] In an implementation of the first aspect, based on vertices of the face frame and vertices of the expansion frame, a first distance between the face frame and the expansion frame in a horizontal direction and a second distance between the face frame and the expansion frame in a vertical direction are determined;

[0034] A restoration frame is determined based on the first distance, the second distance, the height of the face frame, and the width of the face frame.

[0035] A second aspect of an embodiment of the present application provides an image processing device, including:

[0036] An acquisition module, used for acquiring a face frame of an original face image, wherein the original face image includes a face in a non-frontal state;

[0037] An expansion module, used to expand the face frame based on a preset expansion ratio to obtain an expanded frame of the original face image;

[0038] A first cropping module, used for cropping the original face image based on the expansion frame to obtain an expanded face image;

[0039] A correction module, used for correcting the face in the expanded face image to obtain a corrected face image, wherein the corrected face image includes a face in a frontal state;

[0040] The second cropping module is used to crop the corrected face image based on the expansion frame and the face frame to obtain a target face image.

[0041] A third aspect of an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the computer program.

[0042] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0043] In the first aspect of the embodiment of the present application, the face frame in the original face image is expanded outward according to a preset expansion ratio to obtain an expanded frame, and then the original face image is cropped for the first time through the expanded frame, so that the expanded face image obtained can ensure the integrity of the face information in the non-frontal state, and then the non-frontal state of the face in the expanded face image is corrected, and the corrected face image is cropped for the second time to obtain a target face image including the frontal state. In this way, the face recognition accuracy can be improved while ensuring the integrity of the face information in the image.

[0044] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 It is a schematic diagram of the implementation flow of the image processing method provided in the embodiment of the present application;

[0047] Figure 2 is an example diagram of a face frame in an image provided by an embodiment of the present application;

[0048] Figure 3 is an example diagram of an expansion frame provided in an embodiment of the present application;

[0049] Figure 4 is an example diagram of facial feature points provided by an embodiment of the present application;

[0050] Figure 5 is an example diagram of face rotation angles provided in an embodiment of the present application;

[0051] Figure 6 is an example diagram of the processing process of the original person image provided by the embodiment of the present application;

[0052] Figure 7 It is a schematic diagram of the implementation flow of the method for determining the expansion frame provided in an embodiment of the present application;

[0053] Figure 8 It is a schematic diagram of the implementation process of the image correction method provided in the embodiment of the present application;

[0054] Fig. 9 It is a schematic flow chart of a method for determining a face rotation angle provided in an embodiment of the present application;

[0055] Fig.10 It is a schematic diagram of the secondary cropping process of the restoration frame provided in an embodiment of the present application;

[0056] Fig.11 It is a schematic diagram of the implementation flow of the method for determining the restore frame provided in an embodiment of the present application;

[0057] Fig.12 is a schematic diagram of an image processing device provided in an embodiment of the present application;

[0058] Fig.13 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0060] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0061] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0062] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0063] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0064] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0065] In one embodiment, Figure 1 As shown, an image processing method is provided, and the method is applied to an image processing device as an example for explanation. The image processing device may be an electronic device such as a terminal or server that can implement the image processing method, including the following steps S101 to S105.

[0066] Step S101, obtaining a face frame of an original face image, wherein the original face image includes a face in a non-frontal state.

[0067] In the application, the face frame (also known as the face detection frame) is a rectangular frame of the face area defined after the face is recognized from the image. The face frame includes most of the main areas of the face. Whether the face is in the front face state or the back face state, the face frame can accurately select the position and size of the face. The face frame has a clear width and height pixel size, and the size and proportion of the face can be measured according to the size of the pixel points. The front face state means that the height of the two eyes of the face is approximately parallel to the horizontal line. The back face state can be the front face, side face, crooked face, pitch face, etc.

[0068] For example, Figure 2The face frame shown in the figure, (a) is a face frame in a frontal state, which can frame most of the face area. (b) is a face frame in a non-frontal state (side face), where the main part of the face is concentrated on one side, and the face frame cannot completely frame the face. (c) is a face frame in a non-frontal state (crooked face).

[0069] In the application, the image processing device obtains the face frame of the original face image through the face detection model. The face in the original face image applied to the image processing method of the present application is in a non-frontal state. For the frontal face image, the face position can basically be framed by the face frame, and the height of the two eyes is approximately parallel to the horizontal line, so there is no need to use the image processing method of the present application for image preprocessing.

[0070] Step S102: expanding the face frame based on a preset expansion ratio to obtain an expanded frame of the original face image.

[0071] In the application, when cropping according to the actual face frame, part of the face image will be lost in the subsequent processing such as image rotation. Therefore, the image processing device obtains the expansion frame corresponding to the face frame in the original face image. The expansion frame refers to the image frame obtained by expanding the face frame in the image to the surroundings according to the expansion ratio. The expansion ratio refers to the ratio of the length or width of the expansion to the length or width of the original face frame. If the expansion ratio R is set, in the image coordinate system of the original face image, in pixels, the width of the face frame is width1 (that is, the width occupies width1 pixels) and the height is heigth1 (that is, the height occupies heigth1 pixels). One expansion refers to expanding width1*R pixels to the left, width1*R pixels to the right, heigth1*R pixels upward, and heigth1*R pixels downward. In the application, the preset expansion ratio can be determined according to the actual situation. Specifically, since the subsequent image rotation is implemented based on an affine transformation algorithm, the larger the image size, the more pixels, and the more time-consuming it is, so the preset expansion ratio R should not be too large (generally 0.1-0.5 is more appropriate).

[0072] In the application, the expansion operation for the face frame refers to the expansion of the face frame in the vertical direction and the expansion in the horizontal direction in the image coordinate system. In the vertical direction, the image processing device determines the vertical upward expansion length and the vertical downward expansion length according to the expansion ratio, and uses the upward length, the downward length and the length of the face frame in the vertical direction as the length of the expansion frame in the vertical direction. The image processing device determines the horizontal rightward expansion length and the horizontal leftward expansion length according to the expansion ratio, and uses the leftward length, the rightward length and the length of the face frame in the horizontal direction as the length of the expansion frame in the horizontal direction.

[0073] For example, Figure 3 As shown, in the image coordinate system of the original face image, the unit is pixel, that is, one pixel is taken as a unit length, the upper left corner vertex of the original face image is taken as the origin of the image coordinate system (0, 0), the width of the original face image is width, and the height is height. Assuming that the vertex coordinates of the face frame are (a, b), the width is width1, and the height is height1, the vertex coordinates of the obtained expanded frame are (c, d), the width is width2, and the height is height2. If the expansion ratio is R, then in one expansion operation, the length of the face frame expanded to the left is (width1*R), the length expanded to the right is (width1*R), the length expanded upward is (height1*R), and the length expanded downward is (height1*R). The horizontal component c of the expansion frame is the difference between the horizontal component a of the face frame and the length expanded to the left, that is, c=a-width1*R. If c<0, then c=0 is set. The vertical component d of the expansion frame is the difference between the vertical component b of the face frame and the length expanded upward, that is, d=b-height1*R. If d<0, then d=0 is set. The width width2 of the expansion frame is the sum of the length expanded to the left, the length expanded to the right, and the width of the face frame, that is, width2=width1*R+width1+width1*R. The height of the expansion frame height2 is the sum of the length of the face frame expanded upward, the length of the face frame expanded downward, and the height of the face frame, that is, height2 = height1*R + height1 + height1*R. If the length of the left expansion width1*R> (ac), the length of the left expansion is directly taken as (ac), in other cases width1*R = (ac), so width2 = (ac) + width1 + width1*R. If the length of the upward expansion height1*R> (bd), the length of the upward expansion is directly taken as (bd), in other cases height1*R = (bd), so height2 = (bd) + height1 + height1*R. It should be noted that if c+width2>width-1, width2=width-1-c is directly set, and if d+height2>height-1, height2=height-1-d is directly set.

[0074] Step S103: based on the expansion frame, the original face image is cropped to obtain an expanded face image.

[0075] In the application, the image processing device performs the first cropping process on the original face image according to the position of the expansion frame to obtain the expanded face image. Compared with the cropped image based on the face frame, the expanded face image has richer image information for the original face image in a non-frontal state because the face area selected by the expansion frame is larger than the face frame. Compared with the subsequent image rotation process directly based on the cropped image of the face frame, the image loss can be reduced. Using the expanded face image for subsequent image rotation processing can reduce the amount of calculation and improve the efficiency of subsequent face recognition compared to directly using the original face image for subsequent operations.

[0076] Step S104, correcting the human face in the expanded human face image to obtain a corrected human face image, wherein the corrected human face image includes the human face in a frontal state.

[0077] In the application, after the image processing device performs the first cropping based on the expansion frame to obtain the expanded face image, it performs correction processing on the faces in the expanded face image that are not in a frontal state. The essence of the correction processing is to rotate the expanded face image to make the face in a frontal state.

[0078] In the application, after the image processing device detects the face, in addition to obtaining the face frame, it can also identify the position of the calibrated five-point feature points of the face. The five feature points mark the five positions of the key points of the face, namely the left eye, right eye, nose tip, left corner of the mouth and right corner of the mouth. In addition, the face detection method can accurately mark the position of the feature points regardless of whether the face in the image is frontal, sideways, or tilted, and the feature points have accurate pixel coordinates.

[0079] For example, Figure 4 As shown in the figure, the five feature points of the face are the left eye feature point shown by number 0, the right eye feature point shown by number 1, the nose tip feature point shown by number 2, the left corner of the mouth feature point shown by number 3, and the right corner of the mouth feature point shown by number 4. In the figure (a), the facial feature points are in the frontal state, and in the figure (b), the facial feature points are in the non-frontal state.

[0080] In the application, for the non-frontal face in the expanded face image, when the face is tilted left or right, the face is tilted. The face frame will frame a tilted face and mark the positions of five feature points. The angle a formed by the line connecting the eyes and the horizontal line is the rotation angle of the face. Figure 5 As shown, the line connecting the left eye feature point and the right eye feature point forms an angle a with the horizontal line, and the angle is the rotation angle of the face.

[0081] In the application, when the image processing device performs a rotation transformation on the image, it is necessary to specify the rotation center point, that is, the image rotates along the specified rotation center point. In order to ensure that the expanded face image does not move due to rotation, the face center point can be used as the rotation center point. The image processing device determines the face center point in the expanded face image, and then the image rotation transformation can be realized through the preset application interface, and the face image at the level of both eyes is obtained as the corrected face image, that is, the face in the corrected face image is in a frontal state. Among them, the face center point and the rotation angle of the face are input parameters of the preset application interface.

[0082] Step S105 , cropping the corrected face image based on the expanded frame and the face frame to obtain a target face image.

[0083] In the application, the image processing device re-determines the target face frame in the corrected face image based on the expanded frame and the initial face frame. The initial face frame is used to select the face in the original face image, and the face selected by the initial face frame is in a non-frontal state. The target face frame is used to select the face in the corrected face image, and the face selected by the target face frame is in a frontal state. The image processing device crops the corrected face image again based on the target face frame to obtain the target face image, and the face in the target face image is in a frontal state. The target face image includes complete face information in a frontal state. The target face image, as the output of the image preprocessing stage of the face recognition application, can be used as the input of subsequent face recognition and applied to various face recognition scenarios, such as identity authentication, security inspection, access control, payment, etc., to improve the accuracy and efficiency of face recognition.

[0084] like Figure 6As shown in the figure, an example of the change process from the original face image to the target face image is as follows: 1. The image processing device obtains the face frame and the expansion frame in the original person image, executes 2 to perform the first cropping based on the expansion frame to obtain the expanded face image, and then executes 3. Determine the face rotation angle a and the face center point according to the feature points of both eyes, execute 4. Perform image correction (i.e., rotate the face rotation angle a according to the face center point) to obtain the corrected face image, the size of the corrected face image remains unchanged, and then executes 5. Re-determine the new face frame, and perform a second cropping on the corrected face image according to the new face frame to obtain the final target face image. In the entire image preprocessing process, the method of first cropping based on the expansion frame to obtain the expanded face image, and then cropping the corrected face image again after correction can ensure that the face image is not missing or has low loss, thereby improving the accuracy of subsequent face recognition. Compared with rotating and then cropping the entire original image, the method of cropping the image twice is more efficient. Because the rotation transformation algorithm is time-consuming and not conducive to large-pixel image processing, the method of expanding first and then cropping is adopted to reduce the size of the original image, ensuring the integrity of the face image data while avoiding the rotation of the entire image, thereby improving the efficiency of the program.

[0085] In this embodiment, the face frame in the original face image is expanded outward by a certain area according to a preset expansion ratio to obtain an expanded frame, and then the original face image is cropped for the first time by using the expanded frame. The expanded face image thus obtained can not only reduce the amount of calculation, but also ensure the integrity of the face information in the non-frontal state, and then the non-frontal state of the face in the expanded face image is corrected, and the corrected face image is cropped for the second time to obtain a target face image including the frontal state. In this way, the integrity of the face data can be ensured and the face recognition accuracy can be improved while reducing the amount of calculation.

[0086] In one embodiment, Figure 7 As shown, the implementation process of step S102 includes the following steps S201 to S203. Among them:

[0087] Step S201 : performing horizontal expansion based on the preset expansion ratio and the horizontal components of the vertices of the face frame to obtain the expanded horizontal components.

[0088] In the application, the image processing device obtains the vertices of the initial face frame in the image coordinate system of the original face image, and horizontally expands the initial face frame based on a preset expansion ratio and the horizontal components of the vertices to obtain an expanded horizontal component.

[0089] like Figure 3As shown, in the image coordinate system of the original face image (with the upper left vertex of the original face image as the origin), the unit is pixel, the vertex coordinates of the face frame are (a, b), the width is width1, the height is height1, the preset expansion ratio is R, and the length of the left expansion is width1*R. The value of the horizontal component c of the vertex coordinates of the expansion frame is determined to be c=a-width1*R. When c<0, set c=0.

[0090] Step S202: Perform vertical expansion based on the preset expansion ratio and the vertical components of the vertices of the face frame to obtain expanded vertical components.

[0091] In the application, the image processing device vertically expands the initial face frame based on a preset expansion ratio and the vertical component of the vertex to obtain an expanded vertical component.

[0092] Continuing with the above example, the frame is expanded vertically upward by height1*R pixels, that is, the length of the upward expansion is height1*R. The value of the vertical component d of the vertex coordinates of the expansion frame is determined to be d=b-height1*R. When d<0, d=0 is set.

[0093] Step S203, determining an expansion frame of the original face image based on the width of the face frame, the horizontal expansion width, and the vertical expansion width.

[0094] In the application, since the horizontal expansion of the face frame includes horizontal expansion to the left and horizontal expansion to the right, and the vertical expansion includes vertical expansion upward and vertical expansion downward, the width of the horizontal expansion is equal to the sum of the width of the horizontal expansion to the left and the width of the horizontal expansion to the right, and the width of the vertical expansion is equal to the sum of the width of the vertical expansion upward and the width of the vertical expansion downward. The width of the face frame includes the horizontal width (i.e., the width) and the vertical width (i.e., the height). The image processing device uses the sum of the horizontal expansion width and the horizontal width of the face frame as the horizontal width of the expansion frame, and uses the sum of the vertical expansion width and the vertical width of the face frame as the height of the expansion frame (i.e., the vertical height).

[0095] Continuing with the above example, Figure 3As shown, the horizontal width of the expansion frame is width2 = (ac) + width1 + width1 * R. When c + width2 > width-1, width2 = width-1-c is set, (ac) is the width of horizontal expansion to the left, and width1 * R is the width of horizontal expansion to the right. The vertical width of the expansion frame is height2 = (bd) + height1 + height1 * R. When d + height2 > height-1, height2 = height-1-d is set, (bd) is the width of vertical expansion upward, and height1 * R is the width of vertical expansion downward. Finally, the coordinates of the expansion frame in the image coordinate system are e_rect (c, d, width2, height2).

[0096] In this embodiment, the method of determining the expansion frame by presetting the expansion ratio can improve the accuracy of the expansion frame and ensure the integrity of the face information as much as possible.

[0097] In one embodiment, Figure 8 As shown, the implementation process of step S104 includes the following steps S301 to S303. Among them:

[0098] Step S301, determining the face rotation angle based on the left eye feature points and the right eye feature points in the expanded face image.

[0099] In the application, the face in the expanded face image is in a non-frontal state, and the image processing device corrects the non-frontal face according to the acquired left eye feature points and right eye feature points. Specifically, the image processing device determines the angle between the line connecting the left eye feature points and the right eye feature points and the horizontal line. The angle is the face rotation angle required for subsequent face state correction, such as Figure 6 The angle a shown in . It can be solved by the tangent function and the inverse tangent function in the trigonometric function relationship.

[0100] Step S302: determining a face center point of the expanded face image based on the face frame and the expanded frame.

[0101] In applications, image rotation is usually performed along a specified center point. In the process of correcting the expanded face image, the face center point can be used as the center point of image rotation. The face center point is determined based on the face frame and the expanded frame.

[0102] Step S303: performing a rotation transformation on the expanded face image based on the face rotation angle and the face center point to obtain a corrected face image.

[0103] In the application, the image processing device obtains a preset application program interface to implement the rotation change of the expanded face image, and obtains a corrected face image, in which the face in the corrected face image is in a frontal state. Among them, the face rotation angle and the face center point are input parameters of the preset application program interface. The preset application program interface is a function of realizing image rotation transformation based on an affine transformation algorithm.

[0104] In this embodiment, on the one hand, the face state is estimated by detecting and using the position of the eyes, so as to accurately determine the rotation angle of the face in the image. On the other hand, by combining the information of the face frame and the extended frame, the center position of the face can be determined more accurately. In this way, the accuracy of the face state in the corrected face image can be improved.

[0105] In one embodiment, the specific implementation process of the image processing device determining the face rotation angle is as follows: determining a first difference between a vertical component of a left eye feature point and a vertical component of a right eye feature point in the expanded face image; determining a second difference between a horizontal component of a left eye feature point and a horizontal component of a right eye feature point in the expanded face image; and determining the degree of an angle corresponding to the ratio of the first difference to the second difference as the face rotation angle.

[0106] In the application, the image processing device obtains the left eye feature points and the right eye feature points in the expanded face image. Based on the tangent function in the trigonometric function, the tangent value of the face rotation angle is determined, the image processing device obtains the first difference between the vertical component of the left eye feature point and the vertical component of the right eye feature point, and simultaneously obtains the second difference between the horizontal component of the left eye feature point and the horizontal component of the right eye feature point, and uses the ratio of the first difference to the second difference as the tangent value of the face rotation angle, and determines the face rotation angle corresponding to the tangent value through the inverse tangent function arctan.

[0107] Exemplarily, the left eye feature point p0, the right eye feature point p1, the face rotation angle a, the tangent value tan(a) = (p1.y-p0.y) / (p1.x-p0.x), where (p1.y-p0.y) is the first difference between the vertical component of the left eye feature point and the vertical component of the right eye feature point, and (p1.x-p0.x) is the second difference between the horizontal component of the left eye feature point and the horizontal component of the right eye feature point. The face rotation angle a = arctan((p1.y-p0.y) / (p1.x-p0.x)) is determined by the inverse tangent function.

[0108] In this embodiment, the method of determining the face rotation angle by using the feature points of both eyes can not only improve the accuracy of determining the face rotation angle, but also reduce the calculation complexity.

[0109] In one embodiment, Fig. 9As shown, the implementation process of step S302 includes the following steps S401 to S403. Among them:

[0110] Step S401, determining a third difference between the horizontal component of the vertices of the face frame and the horizontal component of the vertices of the extended frame, and determining a target horizontal component based on the third difference and the width of the face frame.

[0111] In the application, the image processing device determines the center point of the face according to the face frame and the extended frame, and subsequently performs image rotation transformation with the center point of the face. The center point of the face includes a target horizontal component and a target vertical component in the image coordinate system. The image processing device determines a third difference between the horizontal component of the vertex of the face frame and the horizontal component of the vertex of the extended frame, and uses the sum of half the width of the face frame and the third difference as the target horizontal component in the image coordinate system.

[0112] Step S402, determining a fourth difference between the vertical component of the vertices of the face frame and the vertical component of the vertices of the extended frame, and determining a target vertical component based on the fourth difference and the height of the face frame.

[0113] In the application, the image processing device determines a fourth difference between the vertical components of the vertices of the face frame and the vertical components of the vertices of the extended frame, and uses the sum of half the height of the face frame and the fourth difference as the target treatment component.

[0114] Step S403: taking the point determined by the target horizontal component and the target vertical component as the face center point of the expanded face image.

[0115] In the application, the image processing device uses the point determined by the target horizontal component and the target vertical component in the image coordinate system as the center point of the face.

[0116] For example, Figure 3 In the image coordinate system, the face frame o_rect (a, b, width1, height1), the expansion frame e_rect (c, d, width2, height2), the width of the original face image is width, the height is height, the image processing device calculates the distance between the face frame and the expansion frame in the horizontal direction (x-axis direction) as x=ac, that is, the third difference between the horizontal component of the vertex of the face frame and the horizontal component of the vertex of the expansion frame. At the same time, the distance between the face frame and the expansion frame in the vertical direction (y-axis direction) is calculated as y=bd, that is, the fourth difference between the vertical component of the vertex of the face frame and the vertical component of the vertex of the expansion frame. The coordinates of the center point of the face p(x0, y0) are obtained, x0=(ac)+width / 2, y0=(bd)+height / 2.

[0117] In this embodiment, the method of determining the face center point based on the expansion frame and the face frame can provide accuracy of the face center point.

[0118] In one embodiment, before obtaining the face frame of the original face image, the image processing device will first perform face detection on the original face image, specifically including: performing face detection on the original face image to obtain the face frame and facial feature points of the original face image; based on the face frame and facial feature points, performing state detection on the face in the original face image to obtain a detection result, and the detection result is used to characterize whether the face is in a frontal face state.

[0119] In the application, the image processing device can first use the trained artificial intelligence-based face detection model (such as the retinaface model, etc.) to detect the face from the original face image and output the face frame in the original face image (such as the face frame). Figure 2 As shown) and facial landmarks (as Figure 4 The image processing device uses the face frame and facial feature points to detect whether the face in the original face image is in a frontal state, so as to use the face for subsequent face recognition.

[0120] In this embodiment, by detecting whether the human face is in a frontal state, a suitable image preprocessing operation is provided for the original face image, thereby improving the image processing efficiency.

[0121] In one embodiment, Fig.10 The implementation process of step S105 includes the following steps S501 to S502. Among them:

[0122] Step S501: determining a restoration frame based on the expanded frame and the face frame, wherein the restoration frame is used to mark the position and size of the face frame in the expanded frame.

[0123] In the application, the restoration frame is actually the face frame in the corrected face image. The restoration frame represents the position coordinates of the initial face frame in the expanded frame. The corrected face image is obtained by cropping and correcting the original face image through the expanded frame. The initial face frame is relative to the original face image, and the restoration frame is relative to the corrected face image. Compared with the initial face frame, the restoration frame has the same size, but its position in the image coordinate system has changed.

[0124] Step S502: cropping the corrected face image based on the restoration frame to obtain a target face image.

[0125] In the application, after the image processing device determines the restoration frame, the corrected face image is cropped again based on the restoration frame to obtain the target face image.

[0126] In this embodiment, secondary cropping of the original face image is achieved through the restoration frame, thereby improving the accuracy of face recognition while minimizing image loss.

[0127] In one embodiment, Fig.11 As shown, the process of determining the restore frame includes steps S601 to S602. Among them:

[0128] Step S601, based on the vertices of the face frame and the vertices of the extended frame, determining a first distance between the face frame and the extended frame in the horizontal direction and a second distance between the face frame and the extended frame in the vertical direction.

[0129] In the application, the image processing device uses the difference between the horizontal component of the vertices of the extended frame and the horizontal component of the vertices of the face frame as the first distance between the face frame and the extended frame in the horizontal direction. At the same time, the difference between the vertical component of the vertices of the extended frame and the vertical component of the vertices of the face frame is used as the second distance between the face frame and the extended frame in the vertical direction.

[0130] For example, Figure 3 In the image coordinate system, the face frame o_rect (a, b, width1, height1), the expansion frame e_rect (c, d, width2, height2), the width of the original face image is width, the height is height, the first distance is ac, and the second distance is bd.

[0131] Step S602: determining a restoration frame based on the first distance, the second distance, the height of the face frame, and the width of the face frame.

[0132] In the application, the image processing device uses the first distance as the horizontal component of the vertices of the restoration frame in the image coordinate system, and uses the second distance as the vertical component of the vertices of the restoration frame in the image coordinate system. Since the restoration frame is the same size as the initial face frame, the width of the restoration frame is the width of the initial face frame, and the height of the restoration frame is the height of the initial face frame.

[0133] Continuing with the above example, Figure 3 In the image coordinate system, the restored box is r_rect(x, y, width3, height3), where x=ac, y=bd, width3=width1, height3=height1.

[0134] In this embodiment, the restoration frame is determined by using the face frame and the expansion frame, so that the restoration frame can be accurately located in the image.

[0135] The present application also provides an image processing device for executing the steps in the above-mentioned image processing method embodiment. Fig.12 As shown, the image processing device 800 provided in the embodiment of the present application includes:

[0136] The acquisition module 810 is used to acquire a face frame of an original face image, where the original face image includes a face in a non-frontal state.

[0137] The expansion module 820 is used to expand the face frame based on a preset expansion ratio to obtain an expanded frame of the original face image.

[0138] The first cropping module 830 is used to crop the original face image based on the expansion frame to obtain an expanded face image.

[0139] The correction module 840 is used to correct the face in the expanded face image to obtain a corrected face image, wherein the corrected face image includes a face in a frontal state.

[0140] The second cropping module 850 is used to crop the corrected face image based on the expanded frame and the face frame to obtain a target face image.

[0141] In one embodiment, the expansion module is further used to perform horizontal expansion based on the preset expansion ratio and the horizontal component of the vertices of the face frame to obtain the expanded horizontal component; perform vertical expansion based on the preset expansion ratio and the vertical component of the vertices of the face frame to obtain the expanded vertical component; and determine the expansion frame of the original face image based on the width of the face frame, the horizontal expansion width, and the vertical expansion width.

[0142] In one embodiment, the correction module is also used to determine the face rotation angle based on the left eye feature points and the right eye feature points in the expanded face image; determine the face center point of the expanded face image based on the face frame and the expanded frame; and perform a rotation transformation on the expanded face image based on the face rotation angle and the face center point to obtain a corrected face image.

[0143] In one embodiment, the correction module is also used to determine a first difference between a vertical component of a left eye feature point and a vertical component of a right eye feature point in the expanded facial image; determine a second difference between a horizontal component of a left eye feature point and a horizontal component of a right eye feature point in the expanded facial image; and determine the degree of an angle corresponding to the ratio of the first difference to the second difference as a facial rotation angle.

[0144] In one embodiment, the correction module is further used to determine a third difference between a horizontal component of the vertices of the face frame and a horizontal component of the vertices of the extended frame, and determine a target horizontal component based on the third difference and the width of the face frame; determine a fourth difference between a vertical component of the vertices of the face frame and a vertical component of the vertices of the extended frame, and determine a target vertical component based on the fourth difference and the height of the face frame; and use a point determined by the target horizontal component and the target vertical component as a face center point of the extended face image.

[0145] In one embodiment, the acquisition module is also used to perform face detection on the original face image to obtain a face frame and facial feature points of the original face image; based on the face frame and facial feature points, a state detection is performed on the face in the original face image to obtain a detection result, and the detection result is used to characterize whether the face is in a frontal face state.

[0146] In one embodiment, the second cropping module is further used to determine a restoration frame based on the expansion frame and the face frame, wherein the restoration frame is used to mark the position and size of the face frame in the expansion frame; based on the restoration frame, the corrected face image is cropped to obtain a target face image.

[0147] In one embodiment, the second cropping module is further used to determine a first distance between the face frame and the expanded frame in the horizontal direction and a second distance between the face frame and the expanded frame in the vertical direction based on the vertices of the face frame and the vertices of the expanded frame; and determine a restoration frame based on the first distance, the second distance, the height of the face frame, and the width of the face frame.

[0148] Fig.13 The following is a schematic diagram of the structure of an electronic device for implementing the image processing method of the present application according to an embodiment of the present application. Fig.13 As shown, the electronic device 900 of this embodiment includes: at least one processor 910 ( Fig.13 Only one is shown in the figure) a processor, a memory 920, and a computer program 930 stored in the memory 920 and executable on the at least one processor 910, wherein the processor 910 implements the steps in any of the above-mentioned image processing method embodiments when executing the computer program 930.

[0149] The electronic device 900 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device 900 may include, but is not limited to, a processor 910 and a memory 920. Those skilled in the art will appreciate that Fig.13This is merely an example of the electronic device 900 and does not constitute a limitation on the electronic device 900 . The electronic device 900 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0150] The processor 910 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0151] In some embodiments, the memory 920 may be an internal storage unit of the electronic device 900, such as a hard disk or memory of the electronic device 900. In other embodiments, the memory 920 may also be an external storage device of the electronic device 900, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 900. Further, the memory 920 may also include both an internal storage unit of the electronic device 900 and an external storage device. The memory 920 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 920 may also be used to temporarily store data that has been output or is to be output.

[0152] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0153] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0154] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0155] An embodiment of the present application provides a computer program product. When the computer program product runs on an image processing device, the electronic device can implement the steps in the above-mentioned method embodiments when the computer program product is executed.

[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0157] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0158] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0159] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0160] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a face frame of an original face image, wherein the original face image includes a face in a non-frontal state; Expanding the face frame based on a preset expansion ratio to obtain an expanded frame of the original face image; Based on the expansion frame, the original face image is cropped to obtain an expanded face image; Correcting the human face in the expanded human face image to obtain a corrected human face image, wherein the corrected human face image includes the human face in a frontal state; Based on the expansion frame and the face frame, the corrected face image is cropped to obtain a target face image.

2. The image processing method according to claim 1, characterized in that: The step of expanding the face frame based on a preset expansion ratio to obtain an expanded frame of the original face image includes: Based on the preset expansion ratio and the horizontal components of the vertices of the face frame, horizontal expansion is performed to obtain an expanded horizontal component; Performing vertical expansion based on the preset expansion ratio and the vertical components of the vertices of the face frame to obtain the expanded vertical components; An expansion frame of the original face image is determined based on the width of the face frame, the horizontal expansion width, and the vertical expansion width.

3. The image processing method according to claim 1, characterized in that: The step of correcting the face in the expanded face image to obtain a corrected face image includes: Determining a face rotation angle based on left eye feature points and right eye feature points in the expanded face image; Determining a face center point of the expanded face image based on the face frame and the expanded frame; Based on the face rotation angle and the face center point, the expanded face image is rotated to obtain a corrected face image.

4. The image processing method according to claim 3, characterized in that: The step of determining the face rotation angle based on the left eye feature points and the right eye feature points in the expanded face image includes: Determine a first difference between a vertical component of a left eye feature point and a vertical component of a right eye feature point in the expanded facial image; Determine a second difference between a horizontal component of a left eye feature point and a horizontal component of a right eye feature point in the expanded facial image; The degree of an angle corresponding to the ratio of the first difference to the second difference is determined as a face rotation angle.

5. The image processing method according to claim 3, characterized in that: The determining the face center point of the expanded face image based on the face frame and the expanded frame includes: determining a third difference between the horizontal components of the vertices of the face frame and the horizontal components of the vertices of the extended frame, and determining a target horizontal component based on the third difference and the width of the face frame; determining a fourth difference between the vertical components of the vertices of the face frame and the vertical components of the vertices of the extended frame, and determining a target vertical component based on the fourth difference and the height of the face frame; The point determined by the target horizontal component and the target vertical component is used as the face center point of the expanded face image.

6. The image processing method according to claim 1, characterized in that: Before obtaining the face frame of the original face image, the method further includes: Performing face detection on the original face image to obtain a face frame and facial feature points of the original face image; Based on the face frame and the facial feature points, a state detection is performed on the face in the original face image to obtain a detection result, and the detection result is used to indicate whether the face is in a frontal state.

7. The image processing method according to claim 1, characterized in that: The step of cropping the corrected face image based on the expanded frame and the face frame to obtain a target face image includes: Based on the expanded frame and the face frame, determining a restoration frame, wherein the restoration frame is used to mark the position and size of the face frame in the expanded frame; Based on the restoration frame, the corrected face image is cropped to obtain a target face image.

8. The image processing method according to claim 7, characterized in that: The determining of the restoration frame based on the expanded frame and the face frame includes: Determine, based on the vertices of the face frame and the vertices of the extended frame, a first distance between the face frame and the extended frame in the horizontal direction, and a second distance between the face frame and the extended frame in the vertical direction; A restoration frame is determined based on the first distance, the second distance, the height of the face frame, and the width of the face frame.

9. An image processing device, characterized in that: include: An acquisition module, used for acquiring a face frame of an original face image, wherein the original face image includes a face in a non-frontal state; An expansion module, used to expand the face frame based on a preset expansion ratio to obtain an expanded frame of the original face image; A first cropping module, used for cropping the original face image based on the expansion frame to obtain an expanded face image; A correction module, used for correcting the face in the expanded face image to obtain a corrected face image, wherein the corrected face image includes a face in a frontal state; The second cropping module is used to crop the corrected face image based on the expansion frame and the face frame to obtain a target face image.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image processing method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 8 is implemented.

Images