Image recognition method, device, and computer storage medium

Abstract

The application provides an image recognition method, device and computer storage medium. The image recognition method comprises the following steps: acquiring a left-eye image and a right-eye image of a same object; acquiring at least one first face detection frame of the left-eye image and at least one second face detection frame of the right-eye image; determining a candidate face according to a coincidence of the at least one first face detection frame and the at least one second face detection frame; inputting the candidate face into at least one detection model to generate a comprehensive score of the candidate face; and determining the candidate face as an optimal face of the object in response to the comprehensive score being higher than a preset score threshold. The image recognition method improves the accuracy of face recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to an image recognition method, device and computer storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, face recognition technology has been widely used in security monitoring, identity verification, intelligent terminal unlocking and other fields, and has become one of the core technologies to improve the intelligence and convenience of the scene.

[0003] In actual application process, when collecting face image, the face quality in the image is unstable, such as too dark or too strong light leading to abnormal face exposure, shooting angle deviation causing face feature shielding, motion blur causing face detail loss, which is difficult to guarantee the accuracy of face recognition.

[0004] Therefore, the related art has the problem that the face quality in the face image is not high, resulting in low face recognition accuracy. SUMMARY

[0005] To solve the above technical problems, the present application provides an image recognition method, device and computer storage medium.

[0006] To solve the above technical problems, the present application provides an image recognition method, which comprises:

[0007] Obtaining a left eye image and a right eye image of the same object;

[0008] Obtaining at least one first face detection frame of the left eye image and at least one second face detection frame of the right eye image;

[0009] According to the coincidence of the at least one first face detection frame and the at least one second face detection frame, determining a candidate face;

[0010] Inputting the candidate face into at least one detection model to generate a comprehensive score of the candidate face;

[0011] In response to the comprehensive score being higher than a preset score threshold, determining the candidate face as the optimal face of the object.

[0012] Wherein, after obtaining the at least one first face detection frame of the left eye image and the at least one second face detection frame of the right eye image, the image recognition method further comprises:

[0013] Obtaining the detection frame area of each face detection frame;

[0014] Eliminating the face detection frame whose detection frame area exceeds the face area range.

[0015] The image recognition method further comprises the following steps after the at least one first face detection frame of the left eye image and the at least one second face detection frame of the right eye image are obtained:

[0016] The position of each face detection frame and the image boundary where each face detection frame is located are obtained.

[0017] Face detection frames whose distance between the position and the image boundary is less than a preset boundary threshold are removed.

[0018] The image recognition method further comprises the following steps after the candidate face is determined according to the coincidence of the at least one first face detection frame and the at least one second face detection frame:

[0019] First face key points of the candidate face in the left eye image are extracted.

[0020] A first region detection frame of the candidate face is divided according to the first face key points.

[0021] Second face key points of the candidate face in the right eye image are extracted.

[0022] A second region detection frame of the candidate face is divided according to the second face key points.

[0023] Whether the coincidence of the first region detection frame and the second region detection frame and the coincidence of the first face detection frame and the second face detection frame meet a depth difference relationship is determined.

[0024] If not, the candidate face is removed.

[0025] The region detection frame comprises a nose tip detection frame, a mouth detection frame, a left eye detection frame, and / or a right eye detection frame.

[0026] The depth difference relationship is that the intersection over union of the nose tip detection frame is the smallest, the intersection over union of the face detection frame is the largest, and the difference between the intersection over union of the mouth detection frame and the intersection over union of the left eye detection frame and the intersection over union of the right eye detection frame is less than a preset difference threshold.

[0027] The candidate face is input into at least one detection model to generate a comprehensive score of the candidate face, which comprises the following steps:

[0028] The candidate face is sequentially or in parallel input into several detection models to obtain detection results output by each detection model.

[0029] When the detection results of all detection models are passed, a comprehensive score of the candidate face is calculated according to the detection scores of all detection results.

[0030] The candidate face is sequentially or in parallel input into the plurality of detection models, and detection results output by each detection model are obtained, including:

[0031] The candidate face is sequentially input into the plurality of detection models, and when the detection result output by a current detection model is passed, the candidate face is continuously input into a next detection model for detection.

[0032] When the detection result output by the current detection model is not passed, the candidate face is rejected.

[0033] When the detection results of all the detection models are passed, a comprehensive score of the candidate face is calculated according to detection scores of all the detection results, including:

[0034] When the detection results of all the detection models are passed, an attitude angle output by a face angle model, a quality score output by a face quality model, an occlusion prediction value output by a face occlusion model, and an expression prediction value output by a face expression model are obtained.

[0035] An angle detection score is determined according to an average value of the absolute values of the attitude angles.

[0036] A quality detection score is determined according to the quality score.

[0037] An occlusion detection score is determined according to a sum of the occlusion prediction values of all the face regions.

[0038] An expression detection score is determined according to the maximum expression prediction value.

[0039] The comprehensive score of the candidate face is calculated according to the angle detection score, the quality detection score, the occlusion detection score, and the expression detection score.

[0040] To solve the above technical problem, the present application further provides an image recognition device, which comprises a memory and a processor coupled with the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to realize the image recognition method as described above.

[0041] To solve the above technical problem, the present application further provides a computer storage medium, which is used to store program data, and the program data is used to realize the image recognition method as described above when executed by a computer.

[0042] Compared with the prior art, the beneficial effects of the present application are: by processing the left-eye image and the right-eye image containing the same object, determining the candidate face through the coincidence of the face detection frame in the left-eye image and the right-eye image, scoring the candidate face through at least one detection model, and determining the candidate face as the optimal face when the comprehensive score is higher than the preset score threshold, the optimal face is selected before face recognition in the image, the face with low quality is removed, the face with the optimal quality is selected, and the subsequent face recognition is performed based on the optimal face, thereby improving the accuracy of face recognition. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. Among them:

[0044] Figure 1 is a flowchart of an embodiment of the image recognition method provided by the present application;

[0045] Figure 2 is a schematic diagram of the left-eye image and the right-eye image in an embodiment provided by the present application;

[0046] Figure 3 is a schematic diagram of the detection frame in the left-eye image and the right-eye image provided by the present application;

[0047] Figure 4 is a schematic diagram of the region detection frame provided by the present application;

[0048] Figure 5 is a flowchart of an image processing method in another embodiment provided by the present application;

[0049] Figure 6 is a structural schematic diagram of an embodiment of the image recognition device provided by the present application;

[0050] Figure 7 is a structural schematic diagram of an embodiment of the image recognition device provided by the present application;

[0051] Figure 8 is a structural schematic diagram of an embodiment of the computer storage medium provided by the present application. DETAILED DESCRIPTION

[0052] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative effort are within the scope of the present application.

[0053] The terms "first", "second", "third", "fourth" and the like (if any) in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in other than the order illustrated or described herein. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices. "Several" refers to a positive integer greater than or equal to 1.

[0054] The image recognition method of the present application is applied to an image recognition device. The image recognition device of the present application can be a server, a terminal device, or a system comprising a server and a terminal device. Accordingly, each part of the image recognition device, such as each unit, sub-unit, module, and sub-module, can be provided in the server, the terminal device, or both.

[0055] Further, the server described above can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster comprising multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules for providing a distributed server, or as a single software or software module, without specific limitation here.

[0056] For details, please refer to Figure 1 , Figure 1 is a flowchart of an embodiment of the image recognition method provided by the present application, as shown in Figure 1 , the specific steps are as follows:

[0057] Step S11: Obtain left-eye and right-eye images of the same object.

[0058] In the embodiments of the present application, the left-eye image and the right-eye image are a pair of images simulating the difference in the visual angle of human eyes, and the left-eye image is captured by simulating the visual angle of the left eye of a human being using a binocular camera or a double-lens device, and the right-eye image is captured by simulating the visual angle of the right eye of a human being.

[0059] The left-eye image and the right-eye image are obtained by photographing the same object from different visual angles, and refer to Figure 2 , Figure 2 are schematic diagrams of the left-eye image and the right-eye image in an embodiment provided by the present application, as shown in Figure 2 Due to the spacing between the visual angles, the horizontal positions of the same object in the left-eye image and the right-eye image are slightly different, and the object is slightly more to the right in the right-eye image than in the left-eye image.

[0060] Optionally, after the left-eye image and the right-eye image are obtained, epipolar rectification is performed on the left-eye image and the right-eye image, so that the projection points of the same object in the left-eye image and the right-eye image are located on the same horizontal scanning line.

[0061] Step S12: obtaining at least one first face detection box of the left-eye image and at least one second face detection box of the right-eye image.

[0062] In the embodiments of the present application, face detection is performed on the left-eye image and the right-eye image respectively, the face detection result in the left-eye image is represented by a first face detection box, and the face detection result in the right-eye image is represented by a second face detection box.

[0063] Optionally, after the at least one first face detection box of the left-eye image and the at least one second face detection box of the right-eye image are obtained, the area of each face detection box is obtained; the face detection box whose area exceeds the range of the area of a face is removed; and / or the position of each face detection box and the image boundary where each face detection box is located are obtained; the face detection box whose distance from the position to the image boundary is less than a preset boundary threshold is removed.

[0064] In the embodiments of the present application, the face size in the too-small face detection box is small, and the face features are not obvious, or the too-small face detection box is formed by mistakenly detecting a small object in the image as a face; the too-large face detection box may not accurately focus on the core area of the face, and a large amount of background is contained in the face detection box.

[0065] Face recognition based on the faces in the too-large and too-small face detection boxes will both lead to a decrease in the accuracy of face recognition, and in the embodiments of the present application, the too-large and too-small face detection boxes are filtered out to select a face with better quality for face recognition, thereby improving the accuracy of face recognition.

[0066] The face area range is a reasonable face area range of a preset face detection frame. For any face detection frame in the left-eye image and the right-eye image, if the area of the face detection frame is greater than the maximum value of the face area range or less than the minimum value of the face area range, the area of the face detection frame exceeds the face area range, and the face detection frame is removed.

[0067] In addition, a face detection frame that is too close to the image boundary is prone to cause a local face to be cropped in a subsequent cropping process, resulting in the face recognition being unable to extract complete face information. Therefore, in the present application, the face detection frame that is too close to the image boundary is removed to improve the accuracy of face recognition.

[0068] Specifically, whether a face detection frame is too close to the image boundary is determined by the distance between the face detection frame and the image boundary.

[0069] For each face detection frame in the left-eye image and the right-eye image, the face detection frame position is recorded as (x1, y1, x2, y2), where (x1, y1) is the top-left corner coordinate of the face detection frame, and (x2, y2) is the top-right corner coordinate of the face detection frame. The width of the image in which the face detection frame is located is W, and the height of the image is H, which correspond to the image boundary of the image in which the face detection frame is located.

[0070] The distance between the face detection frame position and the image boundary includes the distance between the face detection frame and the left image boundary, the distance between the face detection frame and the right image boundary, the distance between the face detection frame and the top image boundary, and the distance between the face detection frame and the bottom image boundary.

[0071] The left boundary distance is represented as d_left=x1, the right boundary distance is represented as d_right=W-x2, the top boundary distance is represented as d_top=y1, and the bottom boundary distance is represented as d_bottom=H-y2.

[0072] When at least one of the left boundary distance, the right boundary distance, the top boundary distance, and the bottom boundary distance is less than a preset boundary threshold, the corresponding face detection frame is too close to the image boundary, and the face detection frame is removed.

[0073] Step S13: determining a candidate face according to the overlap of the at least one first face detection frame and the at least one second face detection frame.

[0074] In the embodiment of the present application, in a binocular scene, the face of the same object forms a face detection frame in the left-eye image and a face detection frame in the right-eye image. However, there may be multiple redundant detection frames. Even if there are redundant detection frames, the two detection frames with the highest overlap in the left-eye image and the right-eye image are most likely to be the detection frame combination of the face of the object.

[0075] Therefore, according to the overlap condition of the at least one first face detection frame in the left eye image and the at least one second face detection frame in the right eye image, the candidate face of the object is determined.

[0076] Specifically, the intersection-over-union between the at least one first face detection frame and the at least one second face detection frame is determined, that is, for each first face detection frame, the intersection-over-union with all second face detection frames is calculated respectively, and the face in the face detection frame group with the maximum intersection-over-union (one first face detection frame and one second face detection frame) is determined as the candidate face.

[0077] Please refer to Figure 3 , Figure 3 is a schematic diagram of the detection frame in the left eye image and the right eye image provided by the present application, as shown in Figure 3 the left eye image includes two first face detection frames (detection frame 1 and detection frame 2), and the right eye image includes two second face detection frames (detection frame 3 and detection frame 4), the intersection-over-union of detection frame 1 and detection frame 3, the intersection-over-union of detection frame 1 and detection frame 4, the intersection-over-union of detection frame 2 and detection frame 3, and the intersection-over-union of detection frame 2 and detection frame 4 are calculated, according to the calculation result, the intersection-over-union of detection frame 1 and detection frame 3 is the largest, and the face in detection frame 1 and detection frame 3 is determined as the candidate face.

[0078] In an optional embodiment, after the candidate face is determined, the first face key point of the candidate face in the left eye image is extracted; the first region detection frame of the candidate face is divided according to the first face key point; the second face key point of the candidate face in the right eye image is extracted; the second region detection frame of the candidate face is divided according to the second face key point; whether the overlap condition of the first region detection frame and the second region detection frame and whether the overlap condition of the first face detection frame and the second face detection frame meet the depth difference relationship are judged; if not, the candidate face is removed.

[0079] The region detection frame can include at least one of a tip of the nose detection frame, a mouth detection frame, a left eye detection frame and a right eye detection frame.

[0080] The depth difference relationship is that the intersection-over-union of the tip of the nose detection frame is the smallest, the intersection-over-union of the face detection frame is the largest, and the difference between the intersection-over-union of the mouth detection frame and the intersection-over-union of the left eye detection frame and the intersection-over-union of the right eye detection frame is less than a preset difference threshold.

[0081] In the embodiments of the present application, after determining the candidate face based on the face detection frame, and before scoring the candidate face, the candidate face is preliminarily evaluated. If the preliminary evaluation is passed, the candidate face is input into at least one detection model for scoring. If the preliminary evaluation is not passed, the candidate face is removed, and the image is re-acquired to determine the candidate face.

[0082] The preliminary evaluation of the candidate face includes: extracting key points of the candidate face in the left eye image and the right eye image, i.e., first face key points and second face key points.

[0083] Based on the first face key points, a region detection frame is divided in the left eye image to obtain a first region detection frame. Based on the second face key points, a region detection frame is divided in the left eye image to obtain a second region detection frame.

[0084] Please refer to Figure 4 , Figure 4 is a schematic diagram of the region detection frame provided by the present application, as shown in Figure 4 , the region detection frame in the left eye image and the right eye image is divided into a nose tip detection frame, a mouth detection frame, a left eye detection frame, and / or a right eye detection frame, i.e., the nose tip detection frame, the mouth detection frame, the left eye detection frame, and / or the right eye detection frame of the candidate face are divided in the left eye image, and the nose tip detection frame, the mouth detection frame, the left eye detection frame, and / or the right eye detection frame of the candidate face are divided in the right eye image.

[0085] Specifically, the coincidence of the first region detection frame and the second region detection frame includes: the intersection over union IOU nose of the nose tip detection frame in the first region detection frame and the nose tip detection frame in the second region detection frame, the intersection over union IOU mouth of the mouth detection frame in the first region detection frame and the mouth detection frame in the second region detection frame, the intersection over union IOU left_eye of the left eye detection frame in the first region detection frame and the left eye detection frame in the second region detection frame, and the intersection over union IOU right_eye of the right eye detection frame in the first region detection frame and the right eye detection frame in the second region detection frame.

[0086] The coincidence of the first face detection frame and the second face detection frame is represented by the intersection over union IOU face between the face detection frame of the candidate face in the left eye image and the face detection frame of the candidate face in the right eye image.

[0087] According to the principle of binocular stereo vision, and the depth difference relationship between the face, the nose tip, the mouth, and the eyes, the intersection over union of each region is positively correlated with the area size and the distance from the camera, and thus has the following mathematical relationship:

[0088] IOUnose <IOU mouth ≈IOU left_eye ≈IOU right_eye <IOU face

[0089] comparing the sizes of the multiple intersection over union, when the intersection over union is the smallest, the intersection over union is the largest, and the difference between the intersection over union and the intersection over union is less than a preset difference threshold, the depth difference relationship is met, at this time, the face is selected through preliminary evaluation, otherwise, the depth difference relationship is not met, that is, the candidate face does not pass the preliminary evaluation, and the candidate face may be a face placed in the binocular camera through a static photo or an electronic screen, which is recognized by the binocular camera, and the candidate face is removed. nose face mouth≈ left_eye right_eye

[0090] Step S14: inputting the candidate face into at least one detection model to generate a comprehensive score of the candidate face.

[0091] Step S15: in response to the comprehensive score being higher than a preset score threshold, determining the candidate face as an optimal face of the object.

[0092] In the embodiments of the present application, after the candidate face is determined, the candidate face is scored by at least one detection model to obtain a comprehensive score of the candidate face, and if the comprehensive score is higher than a preset score threshold, the candidate face is determined as an optimal face of the object, and the face recognition can be performed through the optimal face.

[0093] Through the above embodiments, by processing the left eye image and the right eye image containing the same object, the candidate face is determined through the coincidence of the face detection frame in the left eye image and the right eye image, the candidate face is scored by at least one detection model, and when the comprehensive score is higher than a preset score threshold, the candidate face is determined as an optimal face, so as to realize the optimization of the face in the image before the face recognition, remove the face with low quality, select the face with the optimal quality, and perform the subsequent face recognition based on the optimal face, thereby improving the accuracy of the face recognition.

[0094] In an optional embodiment, the inputting the candidate face into at least one detection model to generate a comprehensive score of the candidate face comprises: sequentially or in parallel inputting the candidate face into a plurality of detection models to obtain a detection result output by each detection model; and when the detection results of all the detection models are passed, calculating the comprehensive score of the candidate face according to the detection scores of all the detection results.

[0095] ​​​​​In the embodiments of the present application, several detection models are provided, each of which detects the candidate face and outputs a detection result, the detection result including whether the detection is passed and a detection score when the detection is passed, and when the detection results of all the detection models are passed, a comprehensive score of the candidate face is determined based on the detection scores in all the detection results, for example, the comprehensive score is obtained by weighting and fusing the several detection scores.

[0096] The candidate face can be input into the several detection models in sequence or in parallel.

[0097] If the detection result of at least one of the several detection models is not passed, the candidate face is rejected. Based on this, in order to reduce calculation redundancy, the candidate face is preferably input into the several detection models in sequence, the candidate face is input into one of the detection models, and the detection result is obtained, then when the detection result is passed, the candidate face is input into the next detection model for detection, if the detection result is not passed, the candidate face is rejected and no longer detected by the detection model, and the above steps are repeated until all the detection models complete the detection or the candidate face is rejected. That is, the candidate face is input into the several detection models in sequence, when the detection result output by the current detection model is passed, the candidate face is continuously input into the next detection model for detection; when the detection result output by the current detection model is not passed, the candidate face is rejected.

[0098] In an optional embodiment, when the detection results of all the detection models are passed, the comprehensive score of the candidate face is calculated according to the detection scores of all the detection results, including: when the detection results of all the detection models are passed, the pose angle output by the face angle model, the quality score output by the face quality model, the occlusion prediction value output by the face occlusion model, and the expression prediction value output by the face expression model are obtained; the angle detection score is determined according to the average value of the absolute value of the pose angle; the quality detection score is determined according to the quality score; the occlusion detection score is determined according to the sum of the occlusion prediction values of all the face regions; the expression detection score is determined according to the maximum expression prediction value; and the comprehensive score of the candidate face is calculated according to the angle detection score, the quality detection score, the occlusion detection score, and the expression detection score.

[0099] In the embodiments of the present application, the several detection models include a face angle model, a face quality model, a face occlusion model, and a face expression model.

[0100] The candidate face is input into the face angle model trained in advance to obtain the pose angle (including the pitch angle, the yaw angle, and the roll angle) of the candidate face, when the pose angle is greater than an angle threshold, the detection result is not passed, otherwise, the detection result is passed and the pose angle is output.

[0101] The candidate face is input into a pre-trained face quality model to obtain a quality score of the candidate face. When the quality score is less than a quality threshold, the detection result is not passed, otherwise, the detection result is passed, and the quality score is output. The face quality model scores the candidate face by comprehensively considering factors such as illumination and blurring.

[0102] The candidate face is input into a pre-trained face occlusion model. The face occlusion model divides the face into 8 regions, namely forehead, left eye, right eye, nose, left cheek, right cheek, mouth, and chin. The occlusion of each region is predicted, and each region corresponds to an occlusion prediction value. When the occlusion prediction values of all regions are less than or equal to the corresponding occlusion threshold, the detection result is passed, and the occlusion prediction values of all regions are output, otherwise, the detection result is not passed.

[0103] The candidate face is input into a pre-trained face expression model. The face expression model divides the face expression into 7 types, namely anger, disgust, fear, happiness, normal, sadness, and surprise. The expression prediction value of each type of expression corresponding to the candidate face is determined, and the expression prediction values of all expressions are integrated into 1. The maximum expression prediction value is determined. When the expression corresponding to the maximum expression prediction value is normal, happy, or sad, the detection result is passed, and the maximum expression prediction value is output. When the expression corresponding to the maximum expression prediction value is anger, disgust, fear, or surprise, and the maximum expression prediction value is less than or equal to an expression threshold, the detection result is passed, and the maximum expression prediction value is output, otherwise, the detection result is not passed.

[0104] The average value of the absolute values of the attitude angles output by the face angle model is used to determine the angle detection score, which is specifically represented as:

[0105]

[0106] Wherein, represents the angle detection score, pitch represents the pitch angle in the attitude angle, yaw represents the yaw angle in the attitude angle, and roll represents the roll angle in the attitude angle.

[0107] The quality detection score is determined according to the quality score output by the face quality model, which is specifically represented as:

[0108]

[0109] Wherein, represents the quality detection score, and Q represents the quality score.

[0110] The occlusion detection score is determined according to the sum of the occlusion prediction values of all face regions output by the face occlusion model, that is, the occlusion prediction values are integrated to determine the occlusion detection score.

[0111] The maximum expression prediction value output by the face expression model is determined as the expression detection score.

[0112] The comprehensive score of the candidate face is calculated according to the angle detection score, the quality detection score, the occlusion detection score, and the expression detection score, and is specifically represented as:

[0113]

[0114] wherein S is the comprehensive score, is the angle detection score, is the quality detection score, is the occlusion detection score, is the expression detection score, and A, B, C, and D are coefficients.

[0115] Referring to Figure 5 , Figure 5 is a flowchart of an image processing method in another embodiment provided by the present application, as shown in Figure 5 , the method comprises:

[0116] The right-eye image is acquired by a right-eye camera, and the left-eye image is acquired by a left-eye camera; the left-eye image and the right-eye image are subjected to polar correction; face detection is performed in the left-eye image and the right-eye image; the face detection boxes that are too large and the face detection boxes that are too small in the left-eye image and the right-eye image are eliminated; and the face detection boxes that are too close to the image boundary in the left-eye image and the right-eye image are eliminated.

[0117] The intersection-over-union of each pair of the remaining face detection boxes of the left-eye image and the remaining face detection boxes of the right-eye image is calculated; and the face in the face detection box with the maximum intersection-over-union is selected as the candidate face.

[0118] The IOU nose , the IOU mouth , the IOU left_eye , the IOU right_eye , and the IOU face are calculated, i.e., the intersection-over-union between the region detection box of the candidate face in the left-eye image and the region detection box of the candidate face in the right-eye image is calculated, including the intersection-over-union IOU nose of the nose detection box, the intersection-over-union IOU mouth of the mouth detection box, the intersection-over-union IOU left_eye of the left eye detection box, and the intersection-over-union IOU right_eye of the right eye detection box; the intersection-over-union IOU face。

[0119] The IOU nose , the IOU mouth , the IOUleft_eye , IOU right_eyey and IOU face whether the depth difference relationship is met, if not, the candidate face is removed, if yes, the pose angle of the candidate face is predicted according to the face angle model; whether the pose angle is greater than the angle threshold is judged, if yes, the candidate face is removed, if not, the quality score of the candidate face is predicted according to the face quality model; whether the quality score is less than the quality threshold is judged, if yes, the candidate face is removed, if not, the occlusion situation of the candidate face in multiple regions is predicted to obtain an occlusion prediction value according to the face occlusion model; whether the occlusion prediction value of all regions is greater than the occlusion threshold is judged, if yes, the candidate face is removed, if not, the expression prediction value of each kind of expression corresponding to the candidate face is predicted according to the face expression model; whether the expression corresponding to the maximum expression prediction value is anger, disgust, fear or surprise is judged, if not, the optimal face is determined from the candidate face for face recognition, if yes, whether the maximum expression prediction value is greater than the expression threshold is judged; if not, the candidate face is removed, if yes, the comprehensive score is calculated for secondary optimization; whether the comprehensive score is greater than the preset score threshold is judged, if yes, the candidate face is removed, if not, the optimal face is determined from the candidate face for face recognition.

[0120] It should be noted that the step of removing the candidate face is not shown in Figure 5 .

[0121] Those skilled in the art can understand that the writing order of each step in the above-mentioned method of the specific embodiment does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0122] To implement the above-mentioned image recognition method, the present application further provides an image recognition device. For details, please refer to Figure 6 , Figure 6 is a structural schematic diagram of an embodiment of the image recognition device provided by the present application.

[0123] The image recognition device 500 of the present embodiment comprises:

[0124] A first acquisition module 51 is configured to acquire a left eye image and a right eye image of the same object.

[0125] A second acquisition module 52 is configured to acquire at least one first face detection frame of the left eye image and at least one second face detection frame of the right eye image.

[0126] A first determination module 53 is configured to determine a candidate face according to the coincidence of the at least one first face detection frame and the at least one second face detection frame.

[0127] The generating module 54 is configured to input the candidate face into at least one detection model to generate a comprehensive score of the candidate face.

[0128] The second determining module 55 is configured to determine the candidate face as the optimal face of the object in response to the comprehensive score being higher than a preset score threshold.

[0129] To implement the image recognition method, the present application further provides an image recognition device, please refer to Figure 7 , Figure 7 is a structural schematic diagram of an embodiment of the image recognition device provided by the present application.

[0130] The image recognition device 400 of the embodiment comprises a processor 41, a memory 42, an input / output device 43 and a bus 44.

[0131] The processor 41, the memory 42 and the input / output device 43 are connected to the bus 44 respectively, the memory 42 stores program data, and the processor 41 is configured to execute the program data to implement the image recognition method described in the above embodiments.

[0132] In the embodiment of the present application, the processor 41 can also be referred to as a CPU (Central Processing Unit). The processor 41 can be an integrated circuit chip with signal processing capability. The processor 41 can also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor 41 can also be any conventional processor.

[0133] The present application further provides a computer storage medium, please continue to refer to Figure 8 , Figure 8 is a structural schematic diagram of an embodiment of the computer storage medium provided by the present application. The computer storage medium 600 stores a computer program 61. When the computer program 61 is executed by a processor, the computer program 61 is used to implement the image recognition method of the above embodiments.

[0134] The embodiments of the present application are realized in the form of software function units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the whole or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0135] The above description is only the embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. An image recognition method, characterized in that, The image recognition method includes: Obtain the left and right eye images for the same object; Obtain at least one first face detection bounding box in the left eye image and at least one second face detection bounding box in the right eye image; Candidate faces are determined based on the overlap between the at least one first face detection box and the at least one second face detection box; The candidate face is input into at least one detection model to generate a comprehensive score for the candidate face; In response to the overall score being higher than a preset score threshold, the candidate face is determined as the optimal face of the object; The step of inputting the candidate face into at least one detection model to generate a comprehensive score for the candidate face includes: The candidate faces are input into several detection models sequentially or in parallel, and the detection results output by each detection model are obtained. When all detection models pass the test, the comprehensive score of the candidate face is calculated based on the detection scores of all detection results; The step of inputting the candidate faces sequentially or in parallel into several detection models and obtaining the detection results output by each detection model includes: The candidate face is sequentially input into the plurality of detection models. When the detection result output by the current detection model is "pass", the candidate face is input into the next detection model for detection. If the current detection model outputs a "fail" result, the candidate face is removed.

2. The image recognition method according to claim 1, characterized in that, After acquiring at least one first face detection bounding box of the left eye image and at least one second face detection bounding box of the right eye image, the image recognition method further includes: Obtain the area of ​​the detection box for each face detection box; Remove face detection frames whose area exceeds the area of ​​the face.

3. The image recognition method according to claim 1 or 2, characterized in that, After acquiring at least one first face detection bounding box of the left eye image and at least one second face detection bounding box of the right eye image, the image recognition method further includes: Obtain the position of each face detection box and the image boundary where each face detection box is located; Face detection boxes whose distance from the image boundary is less than a preset boundary threshold are removed.

4. The image recognition method according to claim 1, characterized in that, After determining the candidate face based on the overlap between the at least one first face detection box and the at least one second face detection box, the image recognition method further includes: Extract the first facial landmark of the candidate face in the left eye image; The candidate's face is divided into a first region detection box based on the first facial key points; Extract the candidate face from the second facial landmarks in the right eye image; The candidate's face is divided into a second region detection box based on the second facial key points; Determine the overlap between the first region detection box and the second region detection box, and whether the overlap between the first face detection box and the second face detection box conforms to the depth difference relationship; If not, remove the candidate face; The region detection box includes a nose tip detection box, a mouth detection box, a left eye detection box, and / or a right eye detection box.

5. The image recognition method according to claim 4, characterized in that, The depth difference relationship is as follows: the intersection-union ratio of the nose tip detection box is the smallest, the intersection-union ratio of the face detection box is the largest, and the difference between the intersection-union ratio of the mouth detection box, the intersection-union ratio of the left eye detection box, and the intersection-union ratio of the right eye detection box is less than a preset difference threshold.

6. The image recognition method according to claim 1, characterized in that, When all detection models pass the test, a comprehensive score for the candidate face is calculated based on the detection scores of all detection results, including: When all detection models pass the test, obtain the pose angle output by the face angle model, the quality score output by the face quality model, the occlusion prediction value output by the face occlusion model, and the expression prediction value output by the face expression model. An angle detection score is determined based on the average value of the absolute values ​​of the attitude angles. A quality inspection score is determined based on the quality score. The occlusion detection score is determined based on the sum of the occlusion prediction values ​​for all face regions. The expression detection score is determined based on the maximum predicted expression value; The comprehensive score of the candidate's face is calculated based on the angle detection score, the quality detection score, the occlusion detection score, and the expression detection score.

7. An image recognition device, characterized in that, The image recognition device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the image recognition method as described in any one of claims 1 to 6.

8. A computer storage medium, characterized in that, The computer storage medium is used to store program data, which, when executed by the computer, is used to implement the image recognition method as described in any one of claims 1 to 6.

Images