A visible light face image acquisition method and system for border inspection integrated machine

Abstract

The present invention relates to the field of image processing technology, and in particular to a visible light face image acquisition method and system for a border inspection integrated machine. The method comprises: acquiring multiple frames of first face images acquired when a user's face is facing a visible light camera in the border inspection integrated machine; determining a reference frontal face image based on a first key feature point in each frame of the first face image; acquiring multiple frames of second face images acquired when the user's face is rotated in a horizontal direction and a vertical direction respectively; aligning the second face image to the reference frontal face image based on a first key feature point in the reference frontal face image and a second key feature point in the second face image to obtain multiple frames of reference frontal face images; fusing and calibrating the reference frontal face image based on the multiple frames of reference frontal face images to obtain a visible light face image. The present invention can reduce the interference of visible light on the face image and improve the acquisition quality of the face image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a visible light face image acquisition method and system for a border inspection integrated machine. Background Art

[0002] The multifunctional all-in-one machine at the entry-exit border inspection station is referred to as the border inspection machine. It collects facial images under visible light and automatically compares the collected facial images with the facial images in the database to identify whether the traveler is consistent with the documents he holds.

[0003] However, due to the illumination variations of visible light, different lighting conditions (such as strong light, shadow, and backlight) may affect the visibility of facial features, resulting in recognition difficulties and affecting the performance and accuracy of face recognition.

[0004] Therefore, it is necessary to improve the face images collected under visible light to reduce the interference of visible light on the face images and improve the collection quality of face images. Summary of the invention

[0005] The present invention aims to provide a visible light facial image acquisition method and system for a border inspection integrated machine, which can reduce the interference of visible light on facial images and improve the acquisition quality of facial images.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides a visible light face image acquisition method for a border inspection integrated machine, the method comprising the following steps:

[0008] S100, obtaining a plurality of frames of first facial images acquired when the user's face is facing the visible light camera in the border inspection integrated machine;

[0009] S200, identifying first key feature points in each frame of the first facial image, and determining a reference frontal facial image based on the first key feature points in each frame of the first facial image;

[0010] S300, acquiring a plurality of frames of second facial images acquired when the user's face is rotated in the horizontal direction and the vertical direction respectively;

[0011] S400, determining second key feature points in a plurality of frames of the second facial image, aligning the second facial image to the reference frontal facial image based on first key feature points in the reference frontal facial image and second key feature points in the second facial image, to obtain a plurality of frames of reference frontal facial images;

[0012] S500, fusing and calibrating the benchmark frontal face image based on multiple frames of reference frontal face images to obtain a visible light face image.

[0013] In a second aspect, an embodiment of the present invention provides a visible light face image acquisition system for a border inspection integrated machine, the system comprising:

[0014] at least one processor;

[0015] at least one memory for storing at least one program;

[0016] When the at least one program is executed by the at least one processor, the at least one processor implements the visible light facial image acquisition method for the border inspection integrated machine as described in any one of the first aspects.

[0017] The beneficial effects of the present invention are as follows: the present invention discloses a visible light face image acquisition method and system for a border inspection integrated machine, which obtains a first face image of a user when the face is facing the visible light camera in the border inspection integrated machine, determines a reference frontal face image based on the first key feature points in multiple frames of the first face image, and uses the second key feature points in multiple frames of the second face image as a reference to fill in the missing first key feature points in the reference frontal face image. Then, feature point completion and image calibration are performed on the reference frontal face image, and the obtained visible light face image contains complete and clear key feature points; thereby reducing the interference of visible light on the face image and improving the acquisition quality of the face image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 It is a flow chart of a visible light face image acquisition method used in a border inspection integrated machine in an embodiment of the present invention;

[0020] Figure 2 It is a structural schematic diagram of a visible light face image acquisition system for a border inspection integrated machine in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0022] Since the key feature points of the face image collected on site are generally compared with the face image in the ID card during identity authentication, it is necessary to obtain the user's frontal face image, which contains complete and clear key feature points. The border inspection machine is set up indoors. Due to the interference of factors such as indoor lighting, the original face image collected by the visible light camera in the border inspection machine may lack some key feature points, and the clarity of the key feature points in the original face image; therefore, it is necessary to process the original face image collected by the visible light camera into a visible light face image, which contains complete and clear key feature points.

[0023] Based on this, the present invention provides a visible light facial image acquisition method and system for a border inspection integrated machine, aiming to reduce the interference of visible light on facial images and improve the acquisition quality of facial images by optimizing facial images acquired by a visible light camera.

[0024] See also Figure 1 , Figure 1 The present invention provides a flow chart of a visible light face image acquisition method for a border inspection integrated machine, wherein the method comprises the following steps:

[0025] S100, obtaining a plurality of frames of first facial images acquired when the user's face is facing the visible light camera in the border inspection integrated machine;

[0026] Specifically, when the user performs identity authentication at the border inspection integrated machine, the user is reminded to face the face as directly as possible to the visible light camera in the border inspection integrated machine, and then multiple frames of first facial images are acquired through the visible light camera.

[0027] S200, identifying first key feature points in each frame of the first facial image, and determining a reference frontal facial image based on the first key feature points in each frame of the first facial image;

[0028] Specifically, since the user's face may be tilted or shaken to a certain extent, by processing multiple frames of the first face image, a frontal face image that is close to the user's face facing the visible light camera can be obtained as a reference frontal face image; in addition, due to interference from visible light, multiple frames of the first face image may have missing key feature points, and it is necessary to first fill in the missing key feature points in the subsequent process, and then improve each key feature point to meet the clarity requirements. The reference frontal face image obtained in this step based on multiple frames of the first face image can be used as a reference for the face image for subsequent identity verification.

[0029] S300, acquiring a plurality of frames of second facial images acquired when the user's face is rotated in the horizontal direction and the vertical direction respectively;

[0030] Specifically, in order to ensure that the key feature points in the visible light facial image reach the set quantity threshold and have good clarity, in this embodiment, the user is reminded to face the face as directly as possible to the visible light camera in the border inspection integrated machine, and then rotate in the horizontal and vertical directions respectively to obtain a sufficient number of key feature points to ensure that the key feature points in the visible light facial image obtained by subsequent processing are not less than the quantity threshold; the user's face is imaged by the visible light camera to obtain multiple frames of second facial images; and the missing first key feature points in the reference frontal facial image are complemented by the second key feature points in the multiple frames of second facial images.

[0031] S400, determining second key feature points in a plurality of frames of the second facial image, aligning the second facial image to the reference frontal facial image based on first key feature points in the reference frontal facial image and second key feature points in the second facial image, to obtain a plurality of frames of reference frontal facial images;

[0032] Specifically, a facial key feature point detection technology is used to identify and locate the second key feature points in the second facial image, and a feature point matching algorithm is used to find feature points corresponding to the reference frontal facial image in the second facial image. The second facial image is transformed using the corresponding feature points, such as rotation, scaling, and translation, to achieve optimal alignment with the reference frontal facial image, and the aligned second facial image is used as the reference frontal facial image.

[0033] S500, fusing and calibrating the reference frontal face image based on multiple frames of reference frontal face images to obtain a visible light face image; wherein the key feature points in the visible light face image are not less than a set quantity threshold.

[0034] Specifically, the missing key feature points in the reference frontal face image are inferred based on the reference frontal face image, and then the feature points of the reference frontal face image are completed and the image is calibrated, so that the visible light face image obtained contains complete and clear key feature points. The number threshold is pre-set according to the accuracy requirements of face matching. For example, when using a 5-point model for face matching, the number threshold can be set to 5. The key feature points of the 5-point model usually include the two eye centers, the nose tip, and the two corners of the mouth; or the number threshold can be set to 68, and the face matching is more accurately performed using a 68-point model. The key feature points of the 68-point model include points around the eyebrows, eyes, nose, mouth, and chin.

[0035] In some preferred embodiments, in S200, the step of identifying the first key feature point in each frame of the first facial image, and determining the reference frontal facial image based on the first key feature point in each frame of the first facial image, includes:

[0036] S210, for each frame of the first face image, identifying a face detection frame in the first face image and a first key feature point inside the face detection frame; wherein the face detection frame is the minimum circumscribed rectangle of the face area in the first face image;

[0037] S220, determining a nose region and an eye region based on the first key feature point inside the face detection frame, and then generating a nose vector and an eye vector;

[0038] Specifically, the nose area refers to the area where the nose is located, and the eye area refers to the area where the two eyes are located; it should be noted that the direction of indoor lighting should be adjusted in advance to avoid direct light to the visible light camera and human eyes, causing serious interference with visible light, and ensure that when the user faces the visible light camera, there are a certain number of first key feature points in the nose area and the eye area in the first facial image taken.

[0039] S230, determining an orthogonal point between a nose vector and two eye vectors in a face detection frame, and determining a distance from the orthogonal point to two side boundaries of the face detection frame based on the orthogonal point and a boundary of the face detection frame;

[0040] S240, determining a width and a length of the face detection frame based on the boundary of the face detection frame, and determining a reference degree of the first face image where the face detection frame is located based on the width and the length of the face detection frame and the distance from the orthogonal point to the boundary of both sides of the face detection frame;

[0041] Specifically, the reference degree represents the degree of angle deviation of the user's face toward the visible light camera when the first facial image is captured; the higher the reference degree, the smaller the angle deviation of the user's face toward the visible light camera.

[0042] S250, selecting a first face image having the most first key feature points from the plurality of first face images with the highest reference degree as a first preferred image;

[0043] Specifically, firstly, sort the multiple frames of first face images in descending order according to the reference degree, and select the top n frames of first face images, for example, 6≤n≤N-2, 9≤N≤100 can be set, and N is the total number of first face images; compare the number of first key feature points in the n frames of first face images, and select the first face image with the most first key feature points as the first preferred image; the first key feature points in the first preferred image are used as the original key feature points, which can retain the most facial features and serve as the high-quality face image applied to face matching.

[0044] S260, determining whether the reference degree of the first preferred image is the largest, if so, executing S270; if not, executing S280;

[0045] S270, using the first preferred image as a reference frontal face image;

[0046] S280: Take a first facial image whose reference degree is greater than that of the first preferred image as a first candidate image, and align the first preferred image based on the first candidate image to obtain a reference frontal facial image.

[0047] In some embodiments, the reference degree is calculated by the following formula:

[0048]

[0049] Among them, i and j are the numbers of the face detection frames, R i It represents the reference degree of the first face image where the i-th face detection frame is located, d1i and d2i represent the distances from the orthogonal point in the i-th face detection frame to the boundaries on both sides of the face detection frame respectively; maxDj represents the maximum value among the widths of N face detection frames, and maxLj represents the maximum value among the lengths of N face detection frames; Di represents the width of the i-th face detection frame, Li represents the length of the i-th face detection frame, and N represents the total number of face detection frames; λ1, λ2, and λ3 are all weight coefficients between 0 and 1.

[0050] In this embodiment, the larger the width of the face detection frame, the more the face is facing the visible light camera in the horizontal direction; the larger the length of the face detection frame, the more the face is facing the visible light camera in the vertical direction; if the values ​​of the face in the horizontal and vertical directions are close to 1, and the distance deviation from the orthogonal point to the boundaries of the face detection frame on both sides is large, it means that the face may be tilted; λ1, λ2, and λ3 are used to perform weighted processing on these three situations according to actual conditions, and the obtained benchmark can comprehensively reflect the standard degree of the face facing the visible light camera, thereby selecting a first face image with better quality.

[0051] In some embodiments, in S280, the step of aligning the first preferred image based on the first candidate image to obtain a reference frontal face image includes:

[0052] S281, determining a difference in the number of first key feature points between the first candidate image and the first preferred image;

[0053] Specifically, the number of first key feature points in the first candidate image and the number of first key feature points in the first preferred image are determined respectively, and the difference between the two numbers is the number difference.

[0054] S282, sorting the plurality of frames of the first candidate images in descending order of the quantity difference, and selecting the plurality of frames of the first candidate images with the highest sorting as the second candidate images;

[0055] Specifically, if the quantity difference is small, the first candidate image and the first preferred image may be relatively close, and a good collimation effect cannot be guaranteed; the larger the quantity difference is, the more the first candidate image deviates from the first preferred image, and the better the collimation effect on the first preferred image is.

[0056] S283, for each frame of the second candidate image, matching the first key feature points common to the first preferred image and the second candidate image by a feature point matching algorithm as the first common feature points of the first preferred image and the second candidate image;

[0057] S284, aligning the first preferred image to each frame of the second candidate image based on the first common feature point, to obtain multiple frames of second preferred images;

[0058] Specifically, for each frame of the second candidate image, the first preferred image is transformed once, and the first preferred image is transformed to obtain multiple frames of second preferred images, and each frame of the second preferred image is aligned one-to-one with each frame of the second candidate image.

[0059] S285, obtaining each key feature point in each frame of the second preferred image, determining the first common feature point corresponding to each key feature point in each frame of the second preferred image, corresponding each key feature point in each frame of the second preferred image according to the first common feature point, calculating the position variance of the corresponding key feature point in each frame of the second preferred image, and retaining the key feature points whose position variance is less than the variance threshold as the retained feature points;

[0060] Specifically, after aligning the first preferred image to the second alternative image, the first common feature point in the first preferred image is transformed into a key feature point in the second preferred image. Based on the first common feature point, each frame of the second preferred image obtains a corresponding key feature point. After the key feature points in each frame of the second preferred image are matched according to the first common feature point, multiple groups of key feature points are obtained, and each group of key feature points corresponds to a first common feature point. After calculating the position variance of each group of key feature points, the key feature points with position deviation are excluded, and the key feature points with close positions are retained, so as to fit a retained feature point with a relatively accurate position.

[0061] S286, determining the second preferred image where each retained feature point is located, normalizing the second candidate image corresponding to the second preferred image according to the reference degree, and using the obtained normalized value as the weight of the retained feature point in the second preferred image;

[0062] Specifically, after determining the second preferred image where each retained feature point is located, all retained feature points in the second preferred image are assigned the same weight; the weight is derived from the second alternative image corresponding to the second preferred image, firstly, the second alternative images corresponding to all the second preferred images are normalized, and the reference degrees corresponding to all the second alternative images are normalized to obtain the normalized value corresponding to each second alternative image, and then the second preferred image is determined, and then corresponding to the retained feature points in the second preferred image, the weights of the retained feature points in the second preferred image are assigned to the normalized values ​​corresponding to the second alternative image.

[0063] S287, after each retained feature point is matched to the first common feature point, multiple retained feature points corresponding to the same first common feature point are summed according to corresponding weights to obtain multiple reference points;

[0064] Specifically, for multiple retained feature points corresponding to the same first common feature point, the position of each retained feature point is multiplied by the corresponding weight and then summed to obtain a reference point, thereby obtaining multiple reference points.

[0065] S288, replacing the first common feature point in the first preferred image with the corresponding reference point to obtain a reference frontal face image.

[0066] In some embodiments, in S400, determining the second key feature points in the plurality of frames of the second facial image, aligning the second facial image to the reference frontal facial image based on the first key feature points in the reference frontal facial image and the second key feature points in the second facial image, and obtaining the plurality of frames of reference frontal facial images, includes:

[0067] S410, recording the second facial image acquired when the user's face is rotated in the horizontal direction as a horizontal image, and recording the second facial image acquired when the user's face is rotated in the vertical direction as a vertical image;

[0068] S420, extracting a valid image from the acquired second face image, wherein the brightness mean of each row of pixels in the valid image is within a brightness threshold range; wherein the valid image includes a first valid image extracted from the horizontal image and a second valid image extracted from the vertical image;

[0069] Specifically, a first valid image is extracted from the acquired horizontal image, and a second valid image is extracted from the acquired vertical image; wherein the brightness mean of each column of pixels in the first valid image is within a brightness threshold range, and the brightness mean of each row of pixels in the second valid image is within a brightness threshold range;

[0070] For each frame of horizontal image, after converting the horizontal image into a first binary image, the first binary image is scanned column by column, the brightness mean of each column of pixels is calculated, a column of pixels whose brightness mean is within the brightness threshold range is taken as a valid pixel column, and it is determined whether there are multiple consecutive valid pixel columns whose continuous number of columns exceeds the column number threshold in the first binary image. If so, multiple consecutive valid pixel columns are cut out from the first binary image to obtain a first valid image. Similarly, for each frame of vertical image, after converting the vertical image into a second binary image, the second binary image is scanned row by row, the brightness mean of each row of pixels is calculated, a row of pixels whose brightness mean is within the brightness threshold range is taken as a valid pixel row, and it is determined whether there are multiple consecutive valid pixel rows whose continuous number of rows exceeds the row number threshold in the second binary image. If so, multiple consecutive valid pixel rows are cut out from the second binary image to obtain a second valid image.

[0071] S430, determining effective key feature points in the effective images of the plurality of frames, matching the effective key feature points in the effective images with the reference points in the reference frontal face image by a feature point matching algorithm, and obtaining effective common feature points of the effective images and the reference frontal face image;

[0072] S440, aligning the effective image to the reference frontal face image based on the effective common feature points to obtain multiple frames of reference frontal face images; wherein the reference frontal face images include a first reference frontal face image corresponding to the first effective image, and a second reference frontal face image corresponding to the second effective image.

[0073] Specifically, determine the first effective key feature points in multiple frames of the first effective image, match the first effective key feature points in the first effective image with the reference points in the reference frontal face image through a feature point matching algorithm, and obtain the second common feature points of the first effective image and the reference frontal face image; align the first effective image to the reference frontal face image based on the second common feature points to obtain a first reference frontal face image; and determine the second effective key feature points in multiple frames of the second effective image, match the second effective key feature points in the second effective image with the reference points in the reference frontal face image through a feature point matching algorithm to obtain the third common feature points of the second effective image and the reference frontal face image; align the second effective image to the reference frontal face image based on the third common feature points to obtain a second reference frontal face image; use the first reference frontal face image and the second reference frontal face image as reference frontal face images to obtain multiple frames of reference frontal face images.

[0074] In this step, the first effective image is aligned to the reference frontal face image to obtain a first reference frontal face image; the second effective image is aligned to the reference frontal face image to obtain a second reference frontal face image;

[0075] In some embodiments, in S500, the fusing and calibrating the reference frontal face image based on multiple frames of reference frontal face images to obtain a visible light face image includes:

[0076] S510, determining key feature reference points in the reference frontal face image;

[0077] Specifically, determining a first key feature reference point in the first reference frontal face image and a second key feature reference point in the second reference frontal face image;

[0078] S520, making one-to-one correspondence between key feature reference points in the reference frontal face image and effective key feature points in the effective image, taking key feature reference points in the reference frontal face image corresponding to effective common feature points in the effective image as reference reference points, and taking remaining key feature reference points in the reference frontal face image as reference missing points; wherein the reference reference points include a first reference reference point in the first reference frontal face image and a second reference reference point in the second reference frontal face image; and the reference missing points include a first reference missing point in the first reference frontal face image and a second reference missing point in the second reference frontal face image;

[0079] Specifically, a first key feature reference point in the first reference frontal face image is matched with a first effective key feature point in the first effective image one by one; a first key feature reference point in the first reference frontal face image corresponding to a second common feature point in the first effective image is determined as a first reference datum point;

[0080] The second key feature reference point in the second reference frontal face image is matched one by one with the second effective key feature point in the second effective image; the second key feature reference point in the second reference frontal face image corresponding to the third common feature point in the second effective image is determined as the second reference reference point.

[0081] S530, for each frame of the reference frontal face image, determine a reference reference point in the reference frontal face image and a corresponding reference point in the reference frontal face image, and determine a deviation of the reference frontal face image based on a positional deviation between the reference reference point and the corresponding reference point; wherein the deviation includes a first deviation and a second deviation, the first deviation being determined based on a positional deviation between the first reference reference point and the corresponding reference point in a horizontal direction, and the second deviation being determined based on a positional deviation between the second reference reference point and the corresponding reference point in a vertical direction;

[0082] Specifically, for each frame of the first reference frontal face image, a first reference reference point in the first reference frontal face image and a corresponding reference point in the reference frontal face image are determined, and a first deviation of the first reference frontal face image is determined based on a positional deviation in a horizontal direction between the first reference reference point and the corresponding reference point;

[0083] For each frame of the second reference frontal facial image, determine the second reference reference point in the second reference frontal facial image and the corresponding reference point in the reference frontal facial image, and determine the second deviation of the second reference frontal facial image based on the position deviation in the vertical direction between the second reference reference point and the corresponding reference point.

[0084] S540, selecting a first reference frontal face image whose first deviation is less than a first deviation threshold, to obtain a plurality of frames of first filling images; selecting a second reference frontal face image whose second deviation is less than a second deviation threshold, to obtain a plurality of frames of second filling images;

[0085] S550, generating a plurality of fitting reference points based on multiple frames of the first filled-in images and multiple frames of the second filled-in images, and filling the plurality of fitting reference points into corresponding positions in the reference frontal face image until the number of reference points and fitting reference points in the reference frontal face image reaches a set threshold number, thereby obtaining a visible light facial image.

[0086] It should be noted that, in this embodiment, if the screened first infill image and the second infill image cannot generate enough fitting reference points, the reference frontal face image is recaptured until the sum of the number of reference points and fitting reference points reaches a set threshold value.

[0087] In some embodiments, in S530, determining the deviation of the reference frontal face image based on the position deviation between the reference reference point and the corresponding reference point includes:

[0088] The first deviation is calculated by the following formula:

[0089]

[0090] Among them, Cx represents the first deviation, x i represents the horizontal coordinate of the i-th reference point, x 0i represents the horizontal coordinate of the reference point corresponding to the i-th reference point, N represents the total number of reference points, x th Indicates the horizontal axis deviation threshold;

[0091] The second deviation is calculated by the following formula:

[0092]

[0093] Among them, Cy represents the first deviation, y i Indicates the ordinate of the i-th reference point, y 0i Indicates the ordinate of the datum point corresponding to the i-th reference datum point, y th Indicates the vertical axis deviation threshold.

[0094] In this embodiment, |x i -x 0i | represents the difference between the horizontal coordinates of the reference point and the corresponding reference point, |y i -y 0i | represents the difference between the horizontal coordinates of the reference reference point and the corresponding reference point, e is the base of the exponential function, and the coordinate difference exceeding the coordinate deviation threshold is amplified by the exponential function to highlight the reference reference points that deviate too much from the reference points. The obtained deviation can reflect the overall deviation of the N reference reference points in the x-axis direction (i.e., horizontal direction) and the y-axis direction (i.e., vertical direction). The smaller the deviation, the closer the reference reference point and the reference point, and the more matched the reference frontal face image and the reference frontal face image, which facilitates the subsequent screening of reference frontal face images whose deviation meets the requirements.

[0095] In some embodiments, in S550, the generating of a plurality of fitting reference points based on a plurality of frames of the first padded image and a plurality of frames of the second padded image, and filling the plurality of fitting reference points into corresponding positions in the reference frontal face image until the number of reference points and fitting reference points in the reference frontal face image reaches a set number threshold, thereby obtaining a visible light face image, includes:

[0096] S551, selecting a frame of first filling image in order of first deviation from small to large, using the selected first filling image as the first main image, selecting the second filling image with the smallest second deviation as the second main image, and determining the vertical deviation between the first main image and the second main image;

[0097] Specifically, the binocular vectors of the first main image and the binocular vectors of the second main image are determined respectively; the difference between the binocular vectors of the first main image and the binocular vectors of the second main image on the vertical axis is used as a vertical deviation; and the vertical coordinate of the first center point is updated based on the vertical deviation.

[0098] S552, taking a first filling image with a first deviation greater than that of the first main image as a first auxiliary image, taking a first reference missing point in the first main image as a first center point, grouping first reference missing points in each frame of the first auxiliary image that are within a horizontal coordinate deviation threshold of the first center point into one group, and determining a maximum pixel value among pixel values ​​of each first reference missing point in the group;

[0099] S553, keeping the horizontal coordinate of the first center point unchanged, subtracting the vertical deviation from the vertical coordinate of the first center point to obtain an updated vertical coordinate of the first center point, and updating the pixel value of the first center point to the maximum pixel value to obtain an updated first center point;

[0100] Specifically, after the screening process in the above steps, the horizontal coordinate of the first center point and the vertical coordinate of the second main image are accurate enough. By updating the vertical coordinate of the first center point based on the vertical deviation, an accurate vertical coordinate can be obtained; updating the pixel value of the first center point to the maximum pixel value can obtain good clarity, thereby subsequently improving the quality of the baseline frontal face image.

[0101] S554, filling the updated first center point as a reference point into a corresponding position in the reference front face image to obtain a filled reference front face image;

[0102] S555, determine whether the reference points in the filled reference frontal face image reach the set quantity threshold; if not, keep the reference points in the filled reference frontal face image unchanged and execute S551; if so, use the filled reference frontal face image as the visible light face image.

[0103] It should be noted that after the updated first center point is filled into the benchmark frontal face image, the benchmark point in the benchmark frontal face image remains unchanged. In the subsequent iteration process, it is only necessary to fill the new benchmark point into the benchmark frontal face image, so as to gradually obtain more benchmark points until the benchmark points in the benchmark frontal face image reach the set number threshold; since the filled image captures the complete face image in both the horizontal and vertical directions and has sufficient key feature points; it is possible to complete the key feature points of the benchmark frontal face image based on the first filled image and the second filled image, thereby improving the accuracy and application scope of face image processing.

[0104] In addition, refer to Figure 2 An embodiment of the present invention further provides a visible light face image acquisition system for a border inspection integrated machine, the system comprising:

[0105] at least one processor;

[0106] at least one memory for storing at least one program;

[0107] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0108] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0109] In addition, an embodiment of the present invention also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by a processor or controller, for example, by a processor in the above-mentioned electronic device embodiment, so that the above-mentioned processor can execute the visible light facial image acquisition method for the border inspection integrated machine in the above-mentioned embodiment.

[0110] Similarly, the contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0111] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CDROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0112] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present disclosure. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A visible light face image acquisition method for a border inspection integrated machine, characterized in that: The method comprises the following steps: S100, obtaining a plurality of frames of first facial images acquired when the user's face is facing the visible light camera in the border inspection integrated machine; S200, identifying first key feature points in each frame of the first facial image, and determining a reference frontal facial image based on the first key feature points in each frame of the first facial image; S300, acquiring a plurality of frames of second facial images acquired when the user's face is rotated in the horizontal direction and the vertical direction respectively; S400, determining second key feature points in a plurality of frames of the second facial image, aligning the second facial image to the reference frontal facial image based on first key feature points in the reference frontal facial image and second key feature points in the second facial image, to obtain a plurality of frames of reference frontal facial images; S500, fusing and calibrating the reference frontal face image based on multiple frames of reference frontal face images to obtain a visible light face image; In S200, the step of identifying the first key feature points in each frame of the first facial image, and determining the reference frontal facial image based on the first key feature points in each frame of the first facial image, includes: S210, for each frame of the first face image, identifying a face detection frame in the first face image and a first key feature point inside the face detection frame; wherein the face detection frame is the minimum circumscribed rectangle of the face area in the first face image; S220, determining a nose region and an eye region based on the first key feature point inside the face detection frame, and then generating a nose vector and an eye vector; S230, determining an orthogonal point between a nose vector and two eye vectors in a face detection frame, and determining a distance from the orthogonal point to two side boundaries of the face detection frame based on the orthogonal point and a boundary of the face detection frame; S240, determining a width and a length of the face detection frame based on the boundary of the face detection frame, and determining a reference degree of the first face image where the face detection frame is located based on the width and the length of the face detection frame and the distance from the orthogonal point to the boundary of both sides of the face detection frame; S250, selecting a first face image having the most first key feature points from the plurality of first face images with the highest reference degree as a first preferred image; S260, determining whether the reference degree of the first preferred image is the largest, if so, executing S270; if not, executing S280; S270, using the first preferred image as a reference frontal face image; S280: Take a first facial image whose reference degree is greater than that of the first preferred image as a first candidate image, and align the first preferred image based on the first candidate image to obtain a reference frontal facial image.

2. The visible light face image acquisition method for a border inspection integrated machine according to claim 1, characterized in that: The reference degree is calculated by the following formula: Among them, i and j are the numbers of the face detection frames, R i It represents the reference degree of the first face image where the i-th face detection frame is located, d1i and d2i represent the distances from the orthogonal point in the i-th face detection frame to the boundaries on both sides of the face detection frame respectively; maxDj represents the maximum value among the widths of N face detection frames, and maxLj represents the maximum value among the lengths of N face detection frames; Di represents the width of the i-th face detection frame, Li represents the length of the i-th face detection frame, and N represents the total number of face detection frames; λ1, λ2, and λ3 are all weight coefficients between 0 and 1.

3. The visible light face image acquisition method for a border inspection integrated machine according to claim 1, characterized in that: In S280, the step of aligning the first preferred image based on the first candidate image to obtain a reference frontal face image includes: S281, determining a difference in the number of first key feature points between the first candidate image and the first preferred image; S282, sorting the plurality of frames of the first candidate images in descending order of the quantity difference, and selecting the plurality of frames of the first candidate images with the highest sorting as the second candidate images; S283, for each frame of the second candidate image, matching the first key feature points common to the first preferred image and the second candidate image by a feature point matching algorithm as the first common feature points of the first preferred image and the second candidate image; S284, aligning the first preferred image to each frame of the second candidate image based on the first common feature point, to obtain multiple frames of second preferred images; S285, obtaining each key feature point in each frame of the second preferred image, determining the first common feature point corresponding to each key feature point in each frame of the second preferred image, corresponding each key feature point in each frame of the second preferred image according to the first common feature point, calculating the position variance of the corresponding key feature point in each frame of the second preferred image, and retaining the key feature points whose position variance is less than the variance threshold as the retained feature points; S286, determining the second preferred image where each retained feature point is located, normalizing the second candidate image corresponding to the second preferred image according to the reference degree, and using the obtained normalized value as the weight of the retained feature point in the second preferred image; S287, after each retained feature point is matched to the first common feature point, multiple retained feature points corresponding to the same first common feature point are summed according to corresponding weights to obtain multiple reference points; S288, replacing the first common feature point in the first preferred image with the corresponding reference point to obtain a reference frontal face image.

4. The visible light face image acquisition method for a border inspection integrated machine according to claim 3, characterized in that: In S400, the determining of the second key feature points in the plurality of frames of the second facial image, aligning the second facial image to the reference frontal facial image based on the first key feature points in the reference frontal facial image and the second key feature points in the second facial image, and obtaining the plurality of frames of reference frontal facial images includes: S410, recording the second facial image acquired when the user's face is rotated in the horizontal direction as a horizontal image, and recording the second facial image acquired when the user's face is rotated in the vertical direction as a vertical image; S420, extracting a valid image from the acquired second face image, wherein the brightness mean of each row of pixels in the valid image is within a brightness threshold range; wherein the valid image includes a first valid image extracted from the horizontal image and a second valid image extracted from the vertical image; S430, determining effective key feature points in the effective images of the plurality of frames, matching the effective key feature points in the effective images with the reference points in the reference frontal face image by a feature point matching algorithm, and obtaining effective common feature points of the effective images and the reference frontal face image; S440, aligning the effective image to the reference frontal face image based on the effective common feature points to obtain multiple frames of reference frontal face images; wherein the reference frontal face images include a first reference frontal face image corresponding to the first effective image, and a second reference frontal face image corresponding to the second effective image.

5. The visible light face image acquisition method for a border inspection integrated machine according to claim 4, characterized in that: In S500, the step of fusing and calibrating the reference frontal face image based on multiple frames of reference frontal face images to obtain a visible light face image includes: S510, determining key feature reference points in the reference frontal face image; S520, making one-to-one correspondence between key feature reference points in the reference frontal face image and effective key feature points in the effective image, taking key feature reference points in the reference frontal face image corresponding to effective common feature points in the effective image as reference reference points, and taking remaining key feature reference points in the reference frontal face image as reference missing points; wherein the reference reference points include a first reference reference point in the first reference frontal face image and a second reference reference point in the second reference frontal face image; and the reference missing points include a first reference missing point in the first reference frontal face image and a second reference missing point in the second reference frontal face image; S530, for each frame of the reference frontal face image, determine a reference reference point in the reference frontal face image and a corresponding reference point in the reference frontal face image, and determine a deviation of the reference frontal face image based on a positional deviation between the reference reference point and the corresponding reference point; wherein the deviation includes a first deviation and a second deviation, the first deviation being determined based on a positional deviation between the first reference reference point and the corresponding reference point in a horizontal direction, and the second deviation being determined based on a positional deviation between the second reference reference point and the corresponding reference point in a vertical direction; S540, selecting a first reference frontal face image whose first deviation is less than a first deviation threshold, to obtain a plurality of frames of first filling images; selecting a second reference frontal face image whose second deviation is less than a second deviation threshold, to obtain a plurality of frames of second filling images; S550, generating a plurality of fitting reference points based on multiple frames of the first filled-in images and multiple frames of the second filled-in images, and filling the plurality of fitting reference points into corresponding positions in the reference frontal face image until the number of reference points and fitting reference points in the reference frontal face image reaches a set threshold number, thereby obtaining a visible light facial image.

6. A visible light face image acquisition system for a border inspection integrated machine, characterized in that: The system comprises: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the visible light facial image acquisition method for the border inspection integrated machine as described in any one of claims 1 to 5.

Images