Image correction method and device and face tracking recognition system

Abstract

The invention provides an image correction method, device and equipment, a medium, a program product and a face tracking recognition system, relates to the technical field of biological recognition, and can be applied to the field of financial science and technology or other fields, the image correction method comprises the following steps: obtaining a first continuous image frame sequence of a head rotation action executed by a user; multiple frames of adjacent images in the first continuous image frame sequence are input into the optical flow acceleration prediction model, a predicted acceleration diagram output by the optical flow acceleration prediction model is obtained, and the predicted acceleration diagram represents the acceleration of pixel points in the multiple frames of adjacent images; determining a pixel point set with the acceleration value within a preset range from the predicted acceleration diagram; fitting the pixel point set to obtain a fitting straight line; calculating an included angle between the fitting straight line and the reference axis of the image coordinate system to obtain a head deviation angle; and according to the head deviation angle, performing rotation correction on each frame of image in the first continuous image frame sequence to obtain a head rotation action image frame sequence with a regular attitude.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of biometric identification and the technical field of image processing, and more particularly to an image correction method, device, equipment, medium and program product, and a face tracking and identification system. BACKGROUND

[0002] In the prior art, image correction, especially pose correction before face recognition, relies on the detection and positioning of facial key points (such as eyes, nose, and corners of the mouth) to estimate the head pose by calculating the relative position relationship of these feature points, and then to rotate or perform affine transformation on the image to achieve correction. However, this kind of method has obvious limitations: on the one hand, it highly depends on the integrity and visibility of the facial features, and when the user wears sunglasses, a mask, or is in a profile, occlusion, or other situations, the key point detection is easily failed, resulting in correction failure; on the other hand, the method based on static features is sensitive to light changes, image quality, and user cooperation, and is not robust enough in real scenes. In addition, the existing methods usually require complex feature extraction and high-precision positioning, and the computational overhead is large, which is difficult to efficiently run in embedded devices or real-time systems. Therefore, there is an urgent need for a technical solution that does not rely on facial appearance features and can estimate head pose and correct images based on motion information, to improve the applicability and accuracy of face recognition systems in complex environments. SUMMARY

[0003] In view of the above problems, the present application provides an image correction method, device, equipment, medium and program product, and a face tracking and identification system.

[0004] According to a first aspect of the present application, an image correction method is provided, comprising: acquiring a first sequence of continuous image frames of a user performing a head turning action; inputting a plurality of adjacent images in the first sequence of continuous image frames into a trained optical flow acceleration prediction model to obtain a predicted acceleration map output by the optical flow acceleration prediction model, the predicted acceleration map representing acceleration information of pixel points determined based on the plurality of adjacent images; determining a set of pixel points with acceleration values within a preset numerical range from the predicted acceleration map; performing linear fitting on the set of pixel points to obtain a fitting straight line; calculating an included angle between the fitting straight line and a reference axis of an image coordinate system to obtain a head offset angle; and performing rotation correction on each image in the first sequence of continuous image frames according to the head offset angle to obtain a sequence of head turning action images with an upright pose.

[0005] According to an embodiment of the present application, the multi-frame adjacent images include adjacent first, second and third frame images, and the optical flow acceleration prediction model includes a first optical flow prediction branch, a second optical flow prediction branch and an acceleration prediction branch; wherein the first prediction optical flow map is output after the first frame image and the second frame image are processed by the first optical flow prediction branch; the second prediction optical flow map is output after the second frame image and the third frame image are processed by the second optical flow prediction branch; and the prediction acceleration map is output after the first prediction optical flow map and the second prediction optical flow map are fused by the acceleration prediction branch.

[0006] According to an embodiment of the present application, the first optical flow prediction branch and the second optical flow prediction branch are of the same structure, the first optical flow prediction branch includes a first image splicing module, a first encoder and a first decoder, and the second optical flow prediction branch includes a second image splicing module, a second encoder and a second decoder; wherein the first frame image and the second frame image are spliced by the first image splicing module; the image features are extracted from the spliced images by the first encoder; the image features are decoded into the first prediction optical flow map by the first decoder; the second frame image and the third frame image are spliced by the second image splicing module; the image features are extracted from the spliced images by the second encoder; and the image features are decoded into the second prediction optical flow map by the second decoder.

[0007] According to an embodiment of the present application, the output of the prediction acceleration map after the first prediction optical flow map and the second prediction optical flow map are fused by the acceleration prediction branch includes: the first prediction optical flow map and the second prediction optical flow map are spliced by an optical flow map splicing module to obtain a spliced optical flow map; the fusion features are extracted from the spliced optical flow map by a third encoder; and the fusion features are decoded into the prediction acceleration map by a third decoder; wherein the acceleration prediction branch includes the optical flow map splicing module, the third encoder and the third decoder.

[0008] According to an embodiment of the present application, the optical flow acceleration prediction model is trained by a knowledge distillation method, and the training process includes: providing a first teacher model and a second teacher model, the first teacher model and the second teacher model being optical flow prediction networks for predicting optical flow from two image frames; obtaining a training data set, the training data set including a sequence of continuous sample image frames; extracting adjacent first, second and third sample image frames from the sequence of continuous sample image frames each time to obtain a group of model inputs; inputting the first sample image frame and the second sample image frame into the first teacher model to obtain a first teacher optical flow map; inputting the second sample image frame and the third sample image frame into the second teacher model to obtain a second teacher optical flow map; inputting the first sample image frame and the second sample image frame into a first optical flow prediction branch to obtain a first training optical flow map; inputting the second sample image frame and the third sample image frame into a second optical flow prediction branch to obtain a second training optical flow map; calculating a first distillation loss between the first teacher optical flow map and the first training optical flow map; calculating a second distillation loss between the second teacher optical flow map and the second training optical flow map; and constructing a total loss function by combining at least the first distillation loss and the second distillation loss to optimize parameters of the optical flow acceleration prediction model.

[0009] According to an embodiment of the present application, constructing the total loss function by combining at least the first distillation loss and the second distillation loss includes: calculating an acceleration loss between a predicted acceleration map output by the optical flow acceleration prediction model and an acceleration ground truth map; and constructing the total loss function by weighted summing the first distillation loss, the second distillation loss and the acceleration loss.

[0010] According to an embodiment of the present application, when the first optical flow ground truth map and the second optical flow ground truth map exist, constructing the total loss function by combining at least the first distillation loss and the second distillation loss further includes: calculating a first optical flow loss between the first predicted optical flow map and the first optical flow ground truth map; calculating a second optical flow loss between the second predicted optical flow map and the second optical flow ground truth map; and constructing the total loss function by weighted summing the first distillation loss, the second distillation loss, the acceleration loss, the first optical flow loss and the second optical flow loss.

[0011] According to an embodiment of the present application, the step of obtaining the first sequence of continuous image frames of the user performing the head turning action includes: guiding the user to sequentially perform a plurality of verification actions containing at least one head turning action; obtaining a complete continuous video stream containing the plurality of verification actions; and according to the execution order of the verification actions, cutting a sequence of continuous image frames containing the head turning action from the complete continuous video stream as the first sequence of continuous image frames.

[0012] According to an embodiment of the present application, the image correction method further comprises: according to the execution sequence of the verification actions, from the complete continuous video stream, intercepting a second continuous image frame sequence containing each verification action in the plurality of verification actions except for the intercepted head rotation action; and performing rotation correction on the second continuous image frame sequence according to the head offset angle.

[0013] According to an embodiment of the present application, the image correction method further comprises: after obtaining the head rotation action image frame sequence with correct posture, instructing the user to perform at least one other verification action; obtaining a third continuous image frame sequence of the user performing the at least one other verification action; and performing rotation correction on the third continuous image frame sequence of the other verification action according to the head offset angle.

[0014] According to an embodiment of the present application, the step of obtaining the continuous image frame sequence of the user performing the head rotation action comprises: obtaining a scene video stream through a fixed first camera; controlling an adjustable second camera to track and align the user's face based on the scene video stream; guiding the user to perform the head rotation action; and obtaining the continuous image frame sequence of the user performing the head rotation action through the second camera.

[0015] According to an embodiment of the present application, the image correction method further comprises: generating a control instruction of the second camera according to the head offset angle; adjusting the shooting angle of the second camera according to the control instruction; obtaining an image frame sequence of the user performing at least one other verification action after adjusting the shooting angle; and performing face recognition based on the head rotation action image frame sequence with correct posture and the newly obtained image frame sequence of the other verification action.

[0016] A second aspect of the present application provides an image correction device, comprising: a first continuous image frame sequence acquisition module configured to obtain a first continuous image frame sequence of a user performing a head rotation action; a predicted acceleration map acquisition module configured to input a plurality of adjacent images in the first continuous image frame sequence into a trained optical flow acceleration prediction model to obtain a predicted acceleration map output by the optical flow acceleration prediction model, the predicted acceleration map representing acceleration information of a pixel point determined based on the plurality of adjacent images; a pixel point set acquisition module configured to determine a pixel point set with an acceleration value within a preset value range from the predicted acceleration map; a fitting module configured to perform linear fitting on the pixel point set to obtain a fitting straight line; a head offset angle calculation module configured to calculate an included angle between the fitting straight line and a reference axis of an image coordinate system to obtain a head offset angle; and a rotation correction module configured to perform rotation correction on each image in the first continuous image frame sequence according to the head offset angle to obtain a head rotation action image frame sequence with correct posture.

[0017] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method.

[0018] The fourth aspect of the present application also provides a computer-readable storage medium having stored thereon a computer program or instructions, which, when executed by a processor, implement the steps of the method.

[0019] The fifth aspect of the present application also provides a computer program product comprising a computer program or instructions, which, when executed by a processor, implement the steps of the method.

[0020] The sixth aspect of the present application also provides a face tracking and recognition system, a fixed first camera for acquiring a scene video stream; an adjustable second camera for tracking a user's face based on the video stream of the first camera and collecting a sequence of image frames of the user performing a head turning action and other verification actions; a local computing platform in communication connection with the first camera and the second camera, the local computing platform being configured to implement the steps of the method.

[0021] The image correction method of the embodiments of the present application analyzes the sequence of continuous image frames in the user's head turning process, extracts the acceleration information of the pixel points by using the trained optical flow acceleration prediction model, further fits a straight line representing the head rotation axis and calculates the included angle between the straight line and the reference axis, and finally rotates and corrects the image according to the angle. This method significantly reduces the dependence of the image correction process on the visibility and integrity of the five features, and can robustly estimate the head posture even when part of the user's face is obscured or in a profile state, to some extent, improving the adaptability and robustness of the face recognition system in real complex scenes. At the same time, this scheme makes full use of dynamic motion information rather than static appearance features, reduces the dependence on high-precision face key point detection, reduces the computational complexity, and is more suitable for deployment and operation in embedded devices or real-time systems, thereby providing a more efficient and reliable image preprocessing means for face recognition applications. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other objects, features and advantages of the present application will become more apparent from the following description of the embodiments of the present application taken with reference to the accompanying drawings, in which:

[0023] Figure 1 The application scenario of the image correction method, device, equipment, medium and program product according to the embodiments of the present application is schematically shown;

[0024] Figure 2 The flowchart of the image correction method according to the embodiments of the present application is schematically shown.

[0025] Figure 3 The structure and working flowchart of a classical optical flow neural network model according to an embodiment of the present application are schematically shown.

[0026] Figure 4 The structure and working flowchart of an optical flow acceleration prediction model according to an embodiment of the present application are schematically shown.

[0027] Figure 5 The training mode schematic diagram of the optical flow acceleration prediction model according to an embodiment of the present application is schematically shown.

[0028] Figure 6 The flowchart of the first embodiment of acquiring the first continuous image frame sequence of the head turning action performed by the user according to an embodiment of the present application is schematically shown.

[0029] Figure 7 The flowchart of acquiring and correcting the third continuous image frame sequence according to an embodiment of the present application is schematically shown.

[0030] Figure 8 The flowchart of the second embodiment of acquiring the first continuous image frame sequence of the head turning action performed by the user according to an embodiment of the present application is schematically shown.

[0031] Figure 9 The flowchart of controlling the second camera to acquire and correct the other verification action image frame sequence according to an embodiment of the present application is schematically shown.

[0032] Figure 10 The structure block diagram of the image correction device according to an embodiment of the present application is schematically shown; and

[0033] Figure 11 The block diagram of the electronic device suitable for implementing the image correction method according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0034] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present application. In the following detailed description of the embodiments of the present application, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it would be apparent to those skilled in the art that the embodiments of the present application can be practiced without these specific details. In other instances, well-known structures and techniques have not been described in detail in order to avoid obscuring aspects of the present application.

[0035] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the terms "comprises", "comprising", "includes", "including" and the like are specifically intended to be open-ended and to mean that other features, steps, operations, and / or components can be added.

[0036] All terms used herein including technical and scientific terms have the meanings commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein are defined as having a meaning that is consistent with the context of the specification in which the terms are utilized and the terms should not be interpreted in an idealized or overly formal sense.

[0037] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted that the meaning of the expression is at least one of A, B, and C, etc. (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.).

[0038] In the technical solutions of the present application, the user information (including but not limited to user personal information, user image information, user equipment information, such as location information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved are information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal.

[0039] In the scenario of using personal information for automated decision-making, the method, device and system provided by the embodiments of the present application all provide corresponding operation portal for the user to choose to agree or refuse the automated decision-making result; if the user chooses to refuse, the expert decision-making process is entered. The expression "automated decision-making" here refers to the activity of automatically analyzing, evaluating the behavior habits, interests and hobbies or economic, health, credit status of individuals, etc. by computer programs and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by personnel who are engaged in a certain field of work, have specialized experience, knowledge and skills and reach a certain professional level.

[0040] Figure 1 An application scenario diagram of the image correction method, apparatus, device, medium and program product according to the embodiments of the present application is schematically shown.

[0041] As Figure 1As shown, the application scenario 100 according to this embodiment can include an automatic teller machine 101, a network 102 and a server 103. The network 102 is a medium for providing a communication link between the automatic teller machine 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0042] The user can use the automatic teller machine 101 to interact with the server 103 through the network 102 to receive or send messages, etc.

[0043] The server 103 can be a server providing various services, such as a background management server (only as an example) providing support for service requests of the user using the automatic teller machine 101. The background management server can analyze and process received user requests, etc., and feed back the processing results to the automatic teller machine 101.

[0044] It should be noted that the image correction method provided by the embodiment of the present application can be generally executed by the server 103. Correspondingly, the image correction apparatus provided by the embodiment of the present application can be generally arranged in the server 103. The image correction method provided by the embodiment of the present application can also be executed by a server or a server cluster different from the server 103 and capable of communicating with the automatic teller machine 101 and / or the server 103. Correspondingly, the image correction apparatus provided by the embodiment of the present application can also be arranged in a server or a server cluster different from the server 103 and capable of communicating with the automatic teller machine 101 and / or the server 103.

[0045] It should be understood that the number of terminal devices, networks and servers in the application scenario 100 is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers. Figure 1

[0046] The image correction method according to the embodiment of the present application will be described in detail below based on the application scenario described above, by Figure 1 Figures 2-9

[0047] Figure 2 An illustrative flowchart of the image correction method according to the embodiment of the present application is shown.

[0048] As shown, the image correction method of this embodiment includes operations S210-S260. Figure 2

[0049] In operation S210, a first continuous image frame sequence of a user performing a head turning action is acquired.

[0050] ​​​​In the embodiments of the present application, a continuous video stream of a user performing a specific head movement is captured by an image acquisition device, the head turning movement refers to the rotational movement of the user's head around an axis in a three-dimensional space, including but not limited to left and right turning of the head in the horizontal direction, nodding or tilting the head in the vertical direction, and other angle rotating movements, which have a clear motion trajectory and a sufficient amplitude to produce effective image sequence changes. The first continuous image frame sequence is composed of multiple image frames arranged in time sequence, which is usually extracted from the video stream at a fixed sampling rate to ensure the continuity in time and the integrity of the motion information. In actual application, the user can be guided to perform such head turning movements alone or as one of a series of multi-step verification movements, and then obtain the first continuous image frame sequence meeting the processing requirements. The sequence is the basic data for subsequent optical flow acceleration prediction and posture analysis, and its quality and continuity directly affect the model processing effect and the final correction accuracy.

[0051] In operation S220, multiple adjacent images in the first continuous image frame sequence are input into the trained optical flow acceleration prediction model to obtain a predicted acceleration map output by the optical flow acceleration prediction model, the predicted acceleration map representing acceleration information of the pixel points determined based on the multiple adjacent images.

[0052] In the embodiments of the present application, the optical flow acceleration prediction model is based on a deep learning architecture, which derives the acceleration distribution from the motion information by analyzing the time sequence relationship and pixel intensity change between continuous image frames. The basic principle is to use the displacement rate of pixel points between adjacent frames to represent the motion acceleration, in which the optical flow information provides the instantaneous velocity field, and the difference between the continuous velocity fields reflects the acceleration characteristics. The value of each pixel point represents the acceleration size and direction of the position in the image sequence, which provides important kinematic basis for subsequent head posture analysis. It should be noted that the acceleration value in the predicted acceleration map corresponds to the pixel point acceleration at the intermediate time frame. For example, for three frames of input, the acceleration distribution at the second frame can be obtained by the difference between the first frame to the second frame and the second frame to the third frame two velocity fields.

[0053] Regarding the application process of the structure of the optical flow acceleration prediction model, in the embodiments of the present application:

[0054] The multi-frame adjacent images include adjacent first, second and third frame images, and the optical flow acceleration prediction model includes a first optical flow prediction branch, a second optical flow prediction branch and an acceleration prediction branch; wherein the first prediction optical flow map is output after the first optical flow prediction branch processes the first and second frame images; the second prediction optical flow map is output after the second optical flow prediction branch processes the second and third frame images; and the prediction acceleration map is output after the acceleration prediction branch fuses the first and second prediction optical flow maps.

[0055] The first and second optical flow prediction branches have the same structure, the first optical flow prediction branch includes a first image splicing module, a first encoder and a first decoder, and the second optical flow prediction branch includes a second image splicing module, a second encoder and a second decoder; wherein the first and second frame images are spliced by the first image splicing module; the image features are extracted from the spliced images by the first encoder; the image features are decoded into the first prediction optical flow map by the first decoder; the second and third frame images are spliced by the second image splicing module; the image features are extracted from the spliced images by the second encoder; and the image features are decoded into the second prediction optical flow map by the second decoder.

[0056] The acceleration prediction branch includes an optical flow map splicing module, a third encoder and a third decoder.

[0057] Figure 3 The structure and workflow diagram of the classical optical flow neural network model according to the embodiment of the application are schematically shown.

[0058] Figure 4 The structure and workflow diagram of the optical flow acceleration prediction model according to the embodiment of the application are schematically shown.

[0059] In the embodiment of the application, the optical flow acceleration prediction model processes three continuous input images by using a specially designed multi-branch architecture, which contains two optical flow prediction branches (the first and second optical flow prediction branches) with the same structure and an acceleration prediction branch. The two optical flow prediction branches are based on the basic paradigm of the classical optical flow neural network: taking two adjacent frame images as input, extracting and fusing features to output an optical flow map. The application makes lightweight adaptation to the feature fusion link on the basis of maintaining the core logic of “two-frame input→optical flow map output”, so as to reduce the computational complexity and meet the requirements of embedded deployment.

[0060] Specifically, the first optical flow prediction branch is used to process the first frame image and the second frame image: firstly, the two frame images are image-stitched by the first image stitching module to form a stitched image containing double-frame information; then the stitched image is input into the first encoder for multi-level feature extraction, the first encoder can adopt a convolutional neural network with a residual block and obtain deep feature representation through stepwise down-sampling; finally, the features are up-sampled and reconstructed by the first decoder, the first decoder can adopt an inverse convolutional network with a residual block to gradually restore the spatial resolution, thereby outputting the first predicted optical flow map. The above processing procedure of "image stitching + encoding and decoding" can be regarded as a simplified implementation of the "feature extraction and correlation fusion" calculation procedure in some classical methods, so as to reduce the calculation amount and resource consumption caused by correlation calculation within an acceptable accuracy range.

[0061] Similarly, the second optical flow prediction branch processes the second frame image and the third frame image in a symmetrical structure: after stitching by the second image stitching module, the features are extracted by the second encoder, and the second predicted optical flow map is output by the second decoder. The two optical flow prediction branches can be implemented synchronously or in parallel, and do not depend on the output results of each other to improve the overall calculation efficiency.

[0062] Subsequently, the acceleration prediction branch fuses the first predicted optical flow map and the second predicted optical flow map: the two predicted optical flow maps are stitched by the optical flow map stitching module to obtain a fusion input; the fusion input is sent into the third encoder to extract fusion features for representing the acceleration information corresponding to the difference between adjacent velocity fields; and the fusion features are up-sampled and reconstructed by the third decoder to output the predicted acceleration map corresponding to the input image.

[0063] Figure 5 The training method of the optical flow acceleration prediction model according to the embodiment of the application is schematically shown.

[0064] In the embodiments of the present application, the optical flow acceleration prediction model is trained by a knowledge distillation method, and the training process includes: providing a first teacher model and a second teacher model, the first teacher model and the second teacher model being optical flow prediction networks for predicting optical flow from two frames of images; obtaining a training data set, the training data set including a sequence of continuous sample image frames; extracting adjacent first, second and third sample image frames from the sequence of continuous sample image frames each time to obtain a group of model inputs; inputting the first sample image frame and the second sample image frame into the first teacher model to obtain a first teacher optical flow map; inputting the second sample image frame and the third sample image frame into the second teacher model to obtain a second teacher optical flow map; inputting the first sample image frame and the second sample image frame into a first optical flow prediction branch to obtain a first training optical flow map; inputting the second sample image frame and the third sample image frame into a second optical flow prediction branch to obtain a second training optical flow map; calculating a first distillation loss between the first teacher optical flow map and the first training optical flow map; calculating a second distillation loss between the second teacher optical flow map and the second training optical flow map; and constructing a total loss function by at least combining the first distillation loss and the second distillation loss to optimize parameters of the optical flow acceleration prediction model.

[0065] Constructing the total loss function by at least combining the first distillation loss and the second distillation loss includes: calculating an acceleration loss between a predicted acceleration map output by the optical flow acceleration prediction model and an acceleration ground truth map; and constructing the total loss function based on weighted summation of the first distillation loss, the second distillation loss and the acceleration loss.

[0066] When the first optical flow ground truth map and the second optical flow ground truth map exist, constructing the total loss function by at least combining the first distillation loss and the second distillation loss further includes: calculating a first optical flow loss between the first predicted optical flow map and the first optical flow ground truth map; calculating a second optical flow loss between the second predicted optical flow map and the second optical flow ground truth map; and constructing the total loss function based on weighted summation of the first distillation loss, the second distillation loss, the acceleration loss, the first optical flow loss and the second optical flow loss.

[0067] In the embodiments of the present application, the optical flow acceleration prediction model is trained by a knowledge distillation method, and the training process fully utilizes the supervision information of the pre-trained teacher model and relies on high-quality ground truth data. Specifically, during training, a first teacher model and a second teacher model are provided, both of which are optical flow prediction networks based on a classic architecture and are respectively dedicated to processing two frames of input images and outputting a predicted optical flow map.

[0068] During the training process, adjacent first, second and third sample image frames are extracted from a training data set including a sequence of continuous sample image frames each time as a group of model inputs.

[0069] The first frame sample image and the second frame sample image are input into the first teacher model to obtain a first teacher optical flow map as a supervision signal; similarly, the second frame sample image and the third frame sample image are input into the second teacher model to obtain a second teacher optical flow map, wherein the first teacher model and the second teacher model are both the aforementioned classical optical flow neural network model, which has been optimized by constructing a loss between a predicted optical flow map and a real optical flow map and performing error back propagation, and has mature and reliable optical flow prediction capability.

[0070] The same input sample is sent into the optical flow acceleration prediction model to be trained: the first frame sample image and the second frame sample image are input into the first optical flow prediction branch to obtain a first training optical flow map; and the second frame sample image and the third frame sample image are input into the second optical flow prediction branch to obtain a second training optical flow map.

[0071] A first distillation loss L ts1 between the first teacher optical flow map and the first training optical flow map is calculated, and a second distillation loss L ts2 between the second teacher optical flow map and the second training optical flow map is calculated, so that the student model (i.e., the optical flow acceleration prediction model) can learn the optical flow prediction capability from the teacher model.

[0072] In the embodiments of the present application, the training process also needs to calculate an acceleration loss L add between a predicted acceleration map output by the optical flow acceleration prediction model and an acceleration ground truth map. The optical flow ground truth map and the acceleration ground truth map are usually obtained by computer graphics methods: a three-dimensional face model is constructed in a virtual environment and a head rotation process is simulated, a continuous image sequence is generated by means of a rendering engine, an accurate optical flow ground truth map is calculated based on known motion parameters, and an accurate acceleration ground truth map is generated by calculating the rate of change of the continuous optical flow field.

[0073] In the embodiments of the present application, in the case of complete training data, when there are a first optical flow ground truth map and a second optical flow ground truth map, a first optical flow loss L rs1 between the first training optical flow map and the first optical flow ground truth map is further calculated, and a second optical flow loss L rs2 between the second training optical flow map and the second optical flow ground truth map is further calculated. These losses are collectively weighted and summed with the distillation loss and the acceleration loss to construct a more comprehensive total loss function.

[0074] This multi-level loss function design ensures that the model can learn abstract features from the teacher network and maintain the ability to learn real labeled data, thereby improving the prediction accuracy while keeping the model lightweight. By using the accurate ground truth provided by the synthetic data, the model can learn more accurate optical flow and acceleration prediction capability, providing a reliable foundation for subsequent head pose analysis.

[0075] In operation S230, a set of pixel points with acceleration values within a preset value range is determined from the predicted acceleration map.

[0076] In operation S240, a straight line is fitted to the set of pixel points.

[0077] In the embodiments of the present application, a set of pixel points with acceleration values within a preset value range is first selected from a predicted acceleration map; the predicted acceleration map is used to represent acceleration vector information of the pixel points; the acceleration value is a scalar determined by the acceleration vector. The preset value range is a threshold range containing zero value; in an ideal case, the acceleration value is zero, and in actual application, the threshold can be adaptively selected and adjusted. Subsequently, the set is fitted with a straight line to obtain a key feature representing the head posture. The physical principle is based on kinematic analysis: when the user performs a head rotation action at a relatively constant speed, the tangential velocity of each point on the head surface remains basically constant, but in the two-dimensional imaging plane of the camera, these velocity vectors show different change rules due to the projection geometry. Specifically, the projection component of the tangential velocity of a point on the head surface on the image plane changes with time, and the change rate is the observed acceleration; when the point rotates closest to the camera position, the velocity direction is parallel to the imaging plane, and the projection velocity reaches a maximum value, and at this time the acceleration is exactly zero, which is consistent with the mathematical principle that the derivative of the velocity extreme point is zero in one-dimensional motion. These acceleration zero points essentially mark the area with the maximum instantaneous velocity in the head rotation process, and their spatial distribution represents the projection of the instantaneous axis of head rotation on the image. In the processing, the system extracts these feature points from the acceleration map by setting a reasonable threshold range, and then uses a robust straight line fitting algorithm, such as the random sample consensus (RANSAC) algorithm, the least squares method, etc., to fit the data set which may contain noise and outliers, to obtain a fitting straight line that best represents the trend of zero point distribution. The direction of the final straight line directly reflects the inclination angle of the actual rotation axis of the head, providing a geometric basis for subsequent head posture correction.

[0078] In operation S250, an angle between the fitted straight line and a reference axis of the image coordinate system is calculated to obtain a head offset angle.

[0079] In the embodiments of the present application, the fitted straight line represents the projection of the instantaneous rotation axis of the head on the two-dimensional image plane. No matter whether the user performs a nodding or a lateral head turning action, the actual rotation axis of the head in the three-dimensional space can be tilted relative to the ground coordinate system due to the initial misalignment. The angle between the fitted straight line and the image reference axis directly quantifies the degree of such tilt, i.e., the "tilted head" angle. For example, when the user performs a nodding action, the head should ideally rotate around a horizontal axis; if there is an angle between the fitted straight line and the image horizontal axis at this time, it means that the user's head has tilted sideways while rotating. Similarly, when turning the head left and right, the angle between the fitted straight line and the vertical axis also reflects the degree of tilted head in the vertical direction. The system accurately obtains an absolute offset for measuring whether the head is "straight" by calculating the angle between the fitted straight line and the selected image reference axis (horizontal axis or vertical axis). This angle is the fundamental basis for subsequent image rotation correction and correction of the face to the upright state.

[0080] In operation S260, according to the head offset angle, the rotation correction is performed on each frame image in the first continuous image frame sequence to obtain a head turning action image frame sequence with a straight posture.

[0081] In the embodiments of the present application, each frame image is inversely rotated and compensated according to the calculated head offset angle with the image center as the rotation center. This correction process ensures that the posture of the head in the image is always upright, eliminating the posture deviation caused by the tilt of the user's head. After rotation correction, the image sequence has a standardized distribution of facial features, improving the accuracy and reliability of subsequent dynamic feature analysis, and providing high-quality input data for dynamic biometric recognition.

[0082] Figure 6 The flowchart of the first embodiment for obtaining the first continuous image frame sequence of the user performing the head turning action according to the embodiments of the present application is schematically shown.

[0083] In the embodiments of the present application, the step of obtaining the first continuous image frame sequence of the user performing the head turning action includes operations S601-S603:

[0084] In operation S601, the user is guided to perform a plurality of verification actions containing at least one head turning action in sequence.

[0085] In operation S602, a complete continuous video stream containing a plurality of verification actions is obtained.

[0086] In the embodiments of the present application, the user is guided to perform a plurality of verification actions including at least one head rotation action (which can include nodding, shaking, opening the mouth, blinking, and other biometric feature verification actions) in a single coherent action through voice prompts or screen instructions. During this period, a complete continuous video stream covering all verification actions is continuously acquired by the image acquisition device.

[0087] In operation S603, according to the execution order of the verification actions, a sequence of continuous image frames containing the head rotation action is extracted from the complete continuous video stream as a first sequence of continuous image frames.

[0088] In the embodiments of the present application, based on the complete continuous video stream acquired in operation S602, according to the preset action execution order and the timestamp information in the video stream, the specific segment containing only the head rotation action is accurately extracted from the complete video stream as the first sequence of continuous image frames for subsequent calculation of the head offset angle. This "one-time acquisition and sequential extraction" method ensures the coherence and completeness of data acquisition, while reducing the operational complexity of repeated user cooperation.

[0089] Further, according to the execution order of the verification actions, a second sequence of continuous image frames containing each of the verification actions other than the extracted head rotation action in the plurality of verification actions is extracted from the complete continuous video stream; and the second sequence of continuous image frames is corrected by rotation according to the head offset angle.

[0090] In the embodiments of the present application, the step of acquiring the sequence of continuous image frames of the user performing the head rotation action can be implemented in various ways.

[0091] Also based on the same complete continuous video stream in step S602, according to the execution order of the verification actions, the segments containing other verification actions such as opening the mouth and blinking are extracted as a second sequence of continuous image frames. In the case where the head offset posture of the user in the same coherent action is stable within a short period of time, the system can directly reuse the head offset angle calculated based on the first sequence of continuous image frames to correct each second sequence of continuous image frames, thereby obtaining a sequence of user head action correction images with various postures. This processing method can improve the processing efficiency of the system, avoid the waste of computing power and time resources caused by repeated calculation, and ensure that each dynamic feature image is corrected under a unified reference.

[0092] These feature extraction and analysis methods based on optical flow can improve the matching success rate and anti-interference ability of face recognition to some extent, and provide consistent and reliable image data basis for subsequent face recognition application expansion.

[0093] Figure 7A flowchart of acquiring and correcting a third continuous image frame sequence according to an embodiment of the present application is shown schematically. It includes operation S701 to operation S703.

[0094] In operation S701, after obtaining the posture-correct head rotation action image frame sequence, the user is instructed to perform at least one other verification action.

[0095] In operation S702, a third continuous image frame sequence of the user performing at least one other verification action is acquired.

[0096] In operation S703, the third continuous image frame sequence of the other verification action is corrected by rotation according to the head offset angle.

[0097] After obtaining the posture-correct head rotation action image frame sequence through the foregoing operation S210 to operation S260, the present scheme can instruct the user to perform other types of verification actions in stages after the first head rotation action collection and correction are completed according to subsequent application requirements. Specifically, the system sends a new action instruction to the user through voice prompts or screen text, such as guiding the user to complete the mouth opening, blinking, and left-right head shaking dynamic biometric actions, and synchronously acquires a third continuous image frame sequence of these new actions through an image acquisition device. It should be noted that the third continuous image frame sequence is not from the video stream of the foregoing head rotation action, but an image sequence independently collected for the new action after the head offset angle calculation is completed.

[0098] Based on the head offset angle calculated through the head rotation action in advance, the system can directly correct the newly acquired third continuous image frame sequence by rotation. The core basis is that the head offset posture of the user in the same scene has short-term stability, and the correction accuracy can be guaranteed without re-guiding the head rotation action. This staged data collection and processing method not only guarantees the accuracy and stability of the head offset angle calculation through an independent head rotation sequence, but also avoids the resource overhead and potential error caused by repeated angle calculation for new actions; at the same time, through the step-by-step guidance of “first completing the core correction, and then supplementing the action as needed”, a multi-form and posture-unified high-quality image sequence is ultimately provided for further application.

[0099] In the embodiments of the present application, the implementation of operation S601 to operation S603 of one-time collection of a complete video stream and the implementation of operation S701 to operation S703 of staged collection of a head rotation sequence and a subsequent new action sequence ultimately obtain posture-unified image frame sequences containing head rotation, mouth opening, blinking, and other actions, which can be used as high-quality input for face recognition.

[0100] The corrected action sequences can ensure that different dynamic features are presented in the same image coordinate system, effectively solve the feature misalignment problem caused by attitude deviation, and provide accurate data support for a face recognition algorithm. For example, the face recognition can be performed by using an optical flow algorithm. For head rotation actions, the optical flow algorithm can track the pixel motion trajectory of the face contour and key feature points, restore the angle and rate change of the head rotation. For local actions such as mouth opening and eye blinking, the optical flow algorithm can capture the subtle pixel displacement of the regions such as the corners of the mouth and the eyelids, and extract unique dynamic biometric features.

[0101] In the manner of combining the corrected sequence with the optical flow algorithm, the recognition model can efficiently integrate multi-dimensional dynamic features and static facial features for comparison, thereby improving the accuracy and efficiency of face recognition, effectively identifying static synthetic faces, and improving the robustness and security of face recognition in financial scenarios such as self-service cash machines.

[0102] Figure 8 A flowchart of a second embodiment of acquiring a first continuous image frame sequence of a user performing a head rotation action according to an embodiment of the present application is schematically shown. It includes operations S801 to S804.

[0103] In operation S801, a scene video stream is acquired by a fixed first camera.

[0104] In an embodiment of the present application, the fixed first camera corresponds to a passive camera in a dual-camera architecture, which continuously captures a scene video stream at a fixed angle. The core role is to provide a global field of view containing the approximate position of the user and environmental information in real time, laying a foundation for subsequent face positioning and tracking. The video stream is directly transmitted to a local computing platform as the initial data source for face detection and position prediction.

[0105] In operation S802, based on the scene video stream, a second adjustable camera is controlled to track and align with the user's face.

[0106] In an embodiment of the present application, based on the scene video stream output by the first camera, a face detection and positioning algorithm is used to identify the user's face region and motion trajectory, and then a corresponding control signal is generated to drive the second adjustable camera to adjust its pitch and yaw angles in real time, so as to accurately track the user's face and always align with the core region of the face, ensuring the stability of the acquisition angle.

[0107] In operation S803, the user is guided to perform a head rotation action.

[0108] In operation S804, a first continuous image frame sequence of the user performing a head rotation action is acquired by the second camera.

[0109] In the embodiments of the present application, after the second camera is aligned with the face, the system guides the user to complete the actions of nodding, turning the head, etc., and the second camera with variable focal length capability captures a clear sequence of continuous image frames, which can be used for subsequent calculation of the offset angle, and provides high-quality input for image correction and dynamic recognition.

[0110] Figure 9 The flowchart of controlling the second camera to acquire and correct the sequence of image frames of other verification actions is schematically shown according to the embodiments of the present application. After obtaining the sequence of image frames of the head turning action with correct posture through the foregoing operations, the present scheme can perform operations S901-S904 according to subsequent application requirements:

[0111] In operation S901, a control instruction of the second camera is generated according to the head offset angle.

[0112] In the embodiments of the present application, after the head offset angle is calculated, a control instruction of the corresponding second camera is generated according to the angle parameter, and the head offset angle is compensated by rotation.

[0113] In operation S902, the shooting angle of the second camera is adjusted according to the control instruction.

[0114] In the embodiments of the present application, the adjustable second camera adjusts the shooting angle according to the head offset angle, which directly eliminates the shooting deviation caused by the head offset from the hardware level.

[0115] In operation S903, after adjusting the shooting angle, a sequence of image frames of the user performing at least one other verification action is acquired.

[0116] After the angle calibration is completed, the system guides the user to perform other verification actions (such as dynamic biometric actions of opening the mouth, blinking, etc.), and the second camera captures the corresponding sequence of image frames, which has a correct posture in the original image. Compared with the image correction by only rotating the image in the software layer, the image details and clarity can be preserved to a greater extent.

[0117] In operation S904, face recognition is performed based on the sequence of image frames of the head turning action with correct posture and the newly acquired sequence of image frames of other verification actions.

[0118] The two types of sequences are integrated as input data to complete the comparison and recognition in the dynamic face recognition algorithm based on optical flow.

[0119] By the hardware cooperative architecture of fixed scene positioning of the first camera and accurate tracking, angle adaptation of the adjustable second camera, combined with the whole process processing of "software calculation offset angle-hardware adjustment shooting angle-accurate collection of multi-motion sequence", the image data quality and feature consistency input to the face recognition model can be improved to a certain extent, the recognition error is reduced, and the accuracy, stability and anti-interference ability of face recognition are enhanced, especially suitable for scenes with high recognition accuracy requirements.

[0120] Figure 10 A structural block diagram of an image correction device according to an embodiment of the present application is schematically shown.

[0121] As Figure 10 shown, the image correction device 1000 of this embodiment includes a first continuous image frame sequence acquisition module 1010, a predicted acceleration map acquisition module 1020, a pixel point set acquisition module 1030, a fitting module 1040, a head offset angle calculation module 1050, and a rotation correction module 1060.

[0122] The first continuous image frame sequence acquisition module 1010 is configured to acquire a first continuous image frame sequence of a user performing a head turning action. In an embodiment, the first continuous image frame sequence acquisition module 1010 can be configured to perform the operation S210 described above, and details are not repeated here.

[0123] The predicted acceleration map acquisition module 1020 is configured to input a plurality of adjacent images in the first continuous image frame sequence into a trained optical flow acceleration prediction model to obtain a predicted acceleration map output by the optical flow acceleration prediction model, the predicted acceleration map representing acceleration information of pixel points determined based on the plurality of adjacent images. In an embodiment, the predicted acceleration map acquisition module 1020 can be configured to perform the operation S220 described above, and details are not repeated here.

[0124] The pixel point set acquisition module 1030 is configured to determine a pixel point set with an acceleration value within a preset value range from the predicted acceleration map. In an embodiment, the pixel point set acquisition module 1030 can be configured to perform the operation S230 described above, and details are not repeated here.

[0125] The fitting module 1040 is configured to perform linear fitting on the pixel point set to obtain a fitting straight line. In an embodiment, the fitting module 1040 can be configured to perform the operation S240 described above, and details are not repeated here.

[0126] The head offset angle calculation module 1050 is configured to calculate an included angle between the fitting straight line and a reference axis of an image coordinate system to obtain a head offset angle. In an embodiment, the head offset angle calculation module 1050 can be configured to perform the operation S250 described above, and details are not repeated here.

[0127] The rotation correction module 1060 is configured to perform rotation correction on each frame image in the first continuous image frame sequence according to the head offset angle, so as to obtain a sequence of posture-orthogonal head rotation action images. In an embodiment, the rotation correction module 1060 can be configured to perform the operation S260 described above, and details are not repeated here.

[0128] According to embodiments of the present application, any one or more of the first continuous image frame sequence acquisition module 1010, the predicted acceleration map acquisition module 1020, the pixel point set acquisition module 1030, the fitting module 1040, the head offset angle calculation module 1050 and the rotation correction module 1060 can be combined in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of other modules, and implemented in one module. According to embodiments of the present application, at least one of the first continuous image frame sequence acquisition module 1010, the predicted acceleration map acquisition module 1020, the pixel point set acquisition module 1030, the fitting module 1040, the head offset angle calculation module 1050 and the rotation correction module 1060 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system in package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or any one of software, hardware and firmware or any appropriate combination of several of them. Alternatively, at least one of the first continuous image frame sequence acquisition module 1010, the predicted acceleration map acquisition module 1020, the pixel point set acquisition module 1030, the fitting module 1040, the head offset angle calculation module 1050 and the rotation correction module 1060 can be at least partially implemented as a computer program module which can perform corresponding functions when it is run.

[0129] Figure 11 The block diagram of the electronic device suitable for implementing the image correction method according to embodiments of the present application is schematically shown.

[0130] As Figure 11As shown, the electronic device 1100 according to an embodiment of the present application includes a processor 1101 which can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 1102 or a program loaded into a random access memory (RAM) 1103 from a storage section 1108. The processor 1101 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chip set, and / or a dedicated microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 1101 can also include an on-board memory for cache use. The processor 1101 can include a single processing unit or multiple processing units to perform the various actions of the method processes according to embodiments of the present application.

[0131] In the RAM 1103, various programs and data required for the operation of the electronic device 1100 are stored. The processor 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. The processor 1101 performs various operations of the method processes according to embodiments of the present application by executing the programs in the ROM 1102 and / or the RAM 1103. Note that the programs can also be stored in one or more memories other than the ROM 1102 and the RAM 1103. The processor 1101 can also perform various operations of the method processes according to embodiments of the present application by executing the programs stored in the one or more memories.

[0132] According to embodiments of the present application, the electronic device 1100 can also include an input / output (I / O) interface 1105 which is also connected to the bus 1104. The electronic device 1100 can also include one or more of the following components connected to the input / output (I / O) interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the input / output (I / O) interface 1105 as necessary. A removable recording medium 1111 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1110 as necessary, so that a computer program read therefrom is installed into the storage section 1108 as necessary.

[0133] The application further provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist independently without being assembled into the device / apparatus / system. The computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the application.

[0134] According to the embodiments of the application, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In this application, a computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. For example, in the embodiments of the application, a computer readable storage medium can include one or more of the above-described ROM 1102 and / or RAM 1103 and / or one or more memories other than the ROM 1102 and the RAM 1103.

[0135] The embodiments of the application also include a computer program product, which includes a computer program containing program codes for executing the method shown in the flow chart. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the image correction method provided by the embodiments of the application.

[0136] The above functions defined in the system / apparatus of the embodiments of the application are performed when the computer program is executed by the processor 1101. According to the embodiments of the application, the above-described system, apparatus, module, unit, etc. can be implemented by computer program modules.

[0137] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium, and be downloaded and installed through the communication part 1109, and / or installed from the detachable medium 1111. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0138] In such embodiments, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the detachable medium 1111. When the computer program is executed by the processor 1101, the above-described functions defined in the system of the embodiments of the present application are performed. According to the embodiments of the present application, the system, device, apparatus, module, unit, and the like described above can be implemented by the computer program modules.

[0139] According to the embodiments of the present application, the program code for executing the computer program provided by the embodiments of the present application can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language, or similar programming language. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet through an Internet service provider).

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a part of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the accompanying drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the involved functions. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0141] The execution of each step in the above method embodiments depends on a specific hardware architecture and computing platform. Based on the same inventive concept, the present application also provides a face tracking and recognition system, which comprises: a fixedly arranged first camera for acquiring a scene video stream; an adjustable second camera for tracking a user's face based on the video stream of the first camera and collecting a high-definition image frame sequence of the user performing a head rotation action and other verification actions; and a local computing platform in communication connection with the first camera and the second camera, the local computing platform being configured to implement any of the above method embodiments.

[0142] The hardware composition of the system and the functional cooperation of each component will be described in detail below:

[0143] In the implementation of the present application, on the basis of the above method embodiments, the present application also provides a complete system implementation. The system realizes the entire process of image correction and dynamic feature recognition through the cooperation of specific hardware architecture and software function. The system comprises a fixedly arranged first camera for continuously acquiring a scene video stream, which usually adopts a wide-angle lens to ensure coverage of the entire operation area; an adjustable second camera equipped with a controllable gimbal and a zoom lens, which can track and align the user's face in real time based on the video stream analysis result of the first camera, and collect a high-definition image frame sequence of the user performing a head rotation action and other verification actions; and a local computing platform in communication connection with the two cameras through a Controller Area Network (CAN) bus, and in communication with a remote computing platform through an industrial Ethernet protocol. The local computing platform is equipped with an embedded neural network processor and is configured to execute all steps of the above image correction method: including receiving a video stream and dividing it into a continuous image frame sequence, calculating a predicted acceleration map through an optical flow acceleration prediction model, fitting a straight line according to the acceleration zero point and calculating a head offset angle, and finally rotating and correcting the image sequence according to the angle. The corrected image data is transmitted to the remote computing platform through the network for final face recognition comparison. Through the division of labor and cooperation of the two cameras and the power distribution of the local and remote platforms, the entire system realizes the full-process automatic processing from face tracking, posture correction to dynamic feature recognition, and to some extent improves the accuracy and reliability of the face recognition system in complex scenes.

[0144] Those skilled in the art can understand that the features described in the various embodiments of the present application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present application. In particular, the features described in the various embodiments of the present application can be combined and / or combined in various ways without departing from the spirit and teachings of the present application. All these combinations and / or combinations fall within the scope of the present application.

Claims

1. An image correction method characterized by, The method comprises the following steps: obtaining a first continuous image frame sequence of a user performing a head rotation action; inputting a plurality of adjacent images in the first continuous image frame sequence into a trained optical flow acceleration prediction model to obtain a predicted acceleration map output by the optical flow acceleration prediction model, the predicted acceleration map representing acceleration information of pixel points determined based on the plurality of adjacent images; determining a pixel point set with acceleration values within a preset numerical range from the predicted acceleration map; performing straight line fitting on the pixel point set to obtain a fitted straight line; calculating an included angle between the fitted straight line and a reference axis of an image coordinate system to obtain a head offset angle; performing rotation correction on each image in the first continuous image frame sequence according to the head offset angle to obtain an image frame sequence of the head rotation action with an upright posture.

2. The method of claim 1, wherein, The plurality of adjacent images comprise a first frame image, a second frame image and a third frame image, and the optical flow acceleration prediction model comprises a first optical flow prediction branch, a second optical flow prediction branch and an acceleration prediction branch; wherein the first optical flow prediction branch processes the first frame image and the second frame image to output a first predicted optical flow map; the second optical flow prediction branch processes the second frame image and the third frame image to output a second predicted optical flow map; the acceleration prediction branch fuses the first predicted optical flow map and the second predicted optical flow map to output the predicted acceleration map.

3. The method of claim 2, wherein, The first optical flow prediction branch and the second optical flow prediction branch have the same structure, the first optical flow prediction branch comprises a first image splicing module, a first encoder and a first decoder, and the second optical flow prediction branch comprises a second image splicing module, a second encoder and a second decoder; wherein the first image splicing module splices the first frame image and the second frame image; the first encoder extracts image features from the spliced image; the first decoder decodes the image features into the first predicted optical flow map; the second image splicing module splices the second frame image and the third frame image; the second encoder extracts image features from the spliced image; the second decoder decodes the image features into the second predicted optical flow map.

4. The method of claim 2, wherein, The acceleration prediction branch fuses the first predicted optical flow map and the second predicted optical flow map to output the predicted acceleration map, comprising: an optical flow map splicing module splices the first predicted optical flow map and the second predicted optical flow map to obtain a spliced optical flow map; a third encoder extracts fusion features from the spliced optical flow map; and a third decoder decodes the fusion features into the predicted acceleration map; wherein the acceleration prediction branch comprises an optical flow map splicing module, a third encoder and a third decoder.

5. The method according to any one of claims 2 to 4, characterized in that, The optical flow acceleration prediction model is trained by a knowledge distillation method, and the training process comprises: providing a first teacher model and a second teacher model, the first teacher model and the second teacher model being optical flow prediction networks for predicting optical flow using two frame images; obtaining a training data set, the training data set comprising a continuous sample image frame sequence; extracting, each time, adjacent first frame sample image, second frame sample image and third frame sample image from the continuous sample image frame sequence, to obtain a group of model inputs; inputting the first frame sample image and the second frame sample image into the first teacher model to obtain a first teacher optical flow map; inputting the second frame sample image and the third frame sample image into the second teacher model to obtain a second teacher optical flow map; inputting the first frame sample image and the second frame sample image into the first optical flow prediction branch to obtain a first training optical flow map; inputting the second frame sample image and the third frame sample image into the second optical flow prediction branch to obtain a second training optical flow map; calculating a first distillation loss between the first teacher optical flow map and the first training optical flow map; calculating a second distillation loss between the second teacher optical flow map and the second training optical flow map; constructing a total loss function by at least combining the first distillation loss and the second distillation loss to optimize the parameters of the optical flow acceleration prediction model.

6. The method of claim 5, wherein, The constructing of the total loss function by at least combining the first distillation loss and the second distillation loss includes: calculating an acceleration loss between a predicted acceleration map output by the optical flow acceleration prediction model and an acceleration ground truth map; constructing the total loss function based on weighted sum of the first distillation loss, the second distillation loss and the acceleration loss.

7. The method of claim 6, wherein, When there are first optical flow ground truth map and second optical flow ground truth map, the constructing of the total loss function by at least combining the first distillation loss and the second distillation loss further includes: calculating a first optical flow loss between the first predicted optical flow map and the first optical flow ground truth map; calculating a second optical flow loss between the second predicted optical flow map and the second optical flow ground truth map; constructing the total loss function based on weighted sum of the first distillation loss, the second distillation loss, the acceleration loss, the first optical flow loss and the second optical flow loss.

8. The method of claim 1, wherein, The step of obtaining the first continuous image frame sequence of the user performing the head rotation action includes: guiding the user to sequentially perform a plurality of verification actions containing at least one head rotation action; obtaining a complete continuous video stream containing the plurality of verification actions; According to the execution order of the verification actions, the continuous image frame sequence containing the head rotation action is intercepted from the complete continuous video stream as the first continuous image frame sequence.

9. The method of claim 8, wherein, The method further includes: According to the execution order of the verification actions, the second continuous image frame sequence containing each verification action in the plurality of verification actions except the intercepted head rotation action is intercepted from the complete continuous video stream; According to the head offset angle, the second continuous image frame sequence is rotated and corrected.

10. The method of claim 1, wherein, The method further includes: After the head rotation action image frame sequence with correct posture is obtained, instructing the user to perform at least one other verification action; obtaining a third continuous image frame sequence of the user performing at least one other verification action; According to the head offset angle, the third continuous image frame sequence of the other verification action is rotated and corrected.

11. The method of claim 1, wherein, The step of obtaining the first continuous image frame sequence of the user performing the head rotation action includes: acquire a scene video stream through a fixed first camera; control an adjustable second camera to track and align with a user's face based on the scene video stream; guide the user to perform the head turning action; acquire a first continuous image frame sequence of the user performing the head turning action through the second camera.

12. The method of claim 11, wherein, The method further comprises: generating a control instruction of the second camera according to the head offset angle; adjusting a shooting angle of the second camera according to the control instruction; after adjusting the shooting angle, acquiring an image frame sequence of the user performing at least one other verification action; performing face recognition based on the posture-corrected head turning action image frame sequence and the newly-acquired other verification action image frame sequence.

13. An image correction apparatus characterized by comprising: The device comprises: a first continuous image frame sequence acquisition module for acquiring a first continuous image frame sequence of the user performing a head turning action; a predicted acceleration map acquisition module for inputting multiple adjacent images in the first continuous image frame sequence into a trained optical flow acceleration prediction model to obtain a predicted acceleration map output by the optical flow acceleration prediction model, the predicted acceleration map representing acceleration information of pixel points determined based on the multiple adjacent images; a pixel point set acquisition module for determining a pixel point set with acceleration values within a preset value range from the predicted acceleration map; a fitting module for performing linear fitting on the pixel point set to obtain a fitting straight line; a head offset angle calculation module for calculating an included angle between the fitting straight line and a reference axis of an image coordinate system to obtain a head offset angle; a rotation correction module for performing rotation correction on each image in the first continuous image frame sequence according to the head offset angle to obtain a posture-corrected head turning action image frame sequence. 14.An electronic device comprising: one or more processors; a memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-12.

15. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the steps of the method according to any one of claims 1-12.

16. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the steps of the method according to any one of claims 1-12.

17. A face tracking and recognition system, characterized by, The system comprises: a fixedly arranged first camera for acquiring a scene video stream; an adjustable second camera for tracking a user's face based on the video stream of the first camera and collecting image frame sequences of the user performing a head turning action and other verification actions; a local computing platform in communication connection with the first camera and the second camera, the local computing platform being configured to execute the method according to any one of claims 1-12.

Images