Facial feature analysis system based on multi-modal fusion and dynamic visualization

Abstract

The invention relates to the field of face recognition, is used for solving the problems that a face feature analysis system cannot perform visual output, face feature items are single, parameter fusion cannot be automatically realized, and the feature capture effect is reduced, and particularly relates to a multi-modal fusion and dynamic visualization-based face feature analysis system. Comprising a multi-parameter acquisition module, an information fusion analysis module, a dynamic analysis module, a feedback correction module and a visualization generation module. According to the method, visual, depth and infrared image data are fused through a multi-modal sensor matrix, three-dimensional accurate reconstruction of facial features can be realized, so that the problem of feature information loss caused by single-modal acquisition in a traditional means is improved, and through continuous acquisition of the face and real-time interactive verification of three-dimensional modeling, the real-time reconstruction of the facial features is realized. The limitation that micro expressions and physiological changes are difficult to capture in static modeling in a traditional means is improved, modeling errors are automatically corrected through a feedback mechanism, and the problem that manual intervention efficiency is low in the traditional means is solved.

Description

Technical Field

[0001] The present invention relates to the field of facial recognition, and in particular to a facial feature analysis system based on multimodal fusion and dynamic visualization. Background Art

[0002] Facial recognition software falls under a broader category of technologies called biometrics. A facial recognition system, centered around human face recognition technology, is an emerging biometric technology. Human faces are popular because they are non-replicable, easy to collect, and don't require the cooperation of the person being photographed. Facial recognition systems have a wide range of applications, including eliminating voting fraud, identity verification for cash withdrawals, and computer security.

[0003] The basic process used by facial recognition software to capture and compare images is as follows: When the system is connected to a video surveillance system, the recognition software searches for facial information in the camera's field of view. It uses a multi-scale algorithm to search for facial images at low resolution. After detecting a head-like shape, it switches to a high-resolution search to determine the head's position, size, and posture. The normalization process can be performed regardless of the head's position or distance from the camera. Light has no effect on the normalization process. The system converts the facial data into a unique code. The encoding makes it easier to compare newly captured facial data with stored facial data. The newly captured facial data is compared with the stored data and connected to at least one stored facial image. The core of the facial recognition system is the local feature analysis algorithm. In the existing technology, when analyzing local features, a single feature analysis method is generally used. Therefore, it is impossible to comprehensively compare features in multiple directions, thereby reducing the accuracy of feature capture and comparison. At the same time, in order to ensure the efficiency of comparison, the existing technology generally records the face image in a character encoding manner, resulting in the lack of visual output of the recorded face image, which affects the scope of application of face recognition in some specific aspects. In response to the above technical problems, this application proposes a solution. Summary of the Invention

[0004] In the present invention, by fusing visual, depth and infrared image data through a multimodal sensor matrix, accurate three-dimensional reconstruction of facial features can be achieved, thereby improving the problem of feature information loss caused by single-modality acquisition in traditional means, and through the real-time interactive verification of continuous facial acquisition and three-dimensional modeling, the limitation of static modeling in traditional means that it is difficult to capture micro-expressions and physiological changes is improved, and the automatic correction of modeling errors and intuitive presentation of results are achieved through a feedback mechanism, thereby improving the problem of low efficiency of manual intervention in traditional means, solving the problems that the facial feature analysis system cannot perform visual output and the facial feature items are single, parameter fusion cannot be automatically realized, and the feature capture effect is reduced, and a facial feature analysis system based on multimodal fusion and dynamic visualization is proposed.

[0005] The purpose of the present invention can be achieved through the following technical solutions: A facial feature analysis system based on multimodal fusion and dynamic visualization includes a multi-parameter acquisition module, an information fusion analysis module, a dynamic analysis module, a feedback correction module, and a visualization generation module. The multi-parameter acquisition module collects multi-faceted facial features through a sensor matrix to obtain facial visual images, facial depth images, and facial infrared images. The information fusion analysis module deconstructs facial texture features after acquiring a facial visual image through a multi-parameter acquisition module, deconstructs facial structural features after acquiring a facial depth image, deconstructs facial infrared line features after acquiring a facial infrared image, and fuses the deconstructed facial texture features, facial structural features, and facial infrared line features to obtain a 3D facial image; The dynamic analysis module obtains the acquisition frequency through the multi-parameter acquisition module and constructs an acquisition pulse curve. The dynamic analysis module obtains 3D facial images through the information fusion analysis module and performs dynamic combination analysis on each acquired 3D facial image according to the pulse curve to generate a three-dimensional facial model. The feedback correction module obtains the three-dimensional facial model through the dynamic analysis module, verifies it based on the three-dimensional facial model, obtains a modeling normality index based on the verification result, determines whether the modeling is complete based on the modeling normality index, and then feeds back the judgment result to the dynamic analysis module; After the dynamic analysis module completes the modeling, the visualization generation module obtains the three-dimensional facial modeling and outputs the three-dimensional facial modeling in a visual and dynamic manner.

[0006] As a preferred embodiment of the present invention, when the multi-parameter acquisition module acquires multi-dimensional facial features, the sensor matrix includes a visible light camera, an infrared sensor, and a 3D structured light device. The multi-parameter acquisition module recognizes the acquired image through the visible light camera, highlights facial information in the image using a preset algorithm, generates position coordinates based on the position of the highlighted face in the image, controls the visible light camera to perform floating zoom, acquires high-definition facial images, and obtains facial visual images. The multi-parameter acquisition module locates the recognition position of the infrared sensor according to the position coordinates generated by the visible light camera, collects the infrared picture of the face, and obtains the facial infrared image; The multi-parameter acquisition module locates the acquisition position of the 3D structured light device according to the position coordinates generated by the visible light camera, acquires the face picture to obtain a 3D point cloud, and performs three-dimensional picture construction processing on the 3D point cloud to obtain a facial depth image.

[0007] As a preferred embodiment of the present invention, the method for the information fusion analysis module to obtain facial texture features is: The information fusion analysis module performs pixel grayscale processing on the facial visual image to obtain a grayscale image, and finds the grayscale value of the central pixel as a threshold, sets the positive direction of the side length of X pixels as a neighborhood, and binarizes the neighborhood pixels to obtain the grayscale of the neighborhood pixels, calculates the absolute value of the grayscale difference between the grayscale of the neighborhood pixels and the grayscale of the center pixel, and sets an upper limit for the grayscale difference. If the absolute value of the difference between the grayscale of the neighborhood pixels and the grayscale of the center pixel is greater than the set upper limit for the grayscale value, it is determined that texture exists and the texture is retained. If the absolute value of the grayscale difference between the grayscale of the neighborhood pixels and the center pixel is less than or equal to the set upper limit for the grayscale value, it is determined that texture does not exist. The information fusion analysis module retains all pixel areas determined to be textures to form facial texture features.

[0008] As a preferred embodiment of the present invention, the method for the information fusion analysis module to obtain facial structural features is: After acquiring the 3D point cloud, the information fusion analysis module selects a set of spatial points as the origin to create a spatial coordinate system, calculates the vector distance between each point in the 3D point cloud and the origin one by one, obtains the coordinates of each point in the 3D point cloud, and records each point in the 3D point cloud in the spatial coordinate system; The information fusion analysis module performs smoothing processing between adjacent points of the 3D point cloud to generate a complete surface and obtain facial structural features.

[0009] As a preferred embodiment of the present invention, the method for the information fusion analysis module to obtain facial infrared texture features is: The information fusion analysis module divides the facial infrared image into pixels to obtain an image composed of multiple pixel points. The information fusion analysis module marks the temperature corresponding to each pixel point, and divides the gears according to the temperature corresponding to the pixel points. The pixel points corresponding to different gear temperatures are successively eliminated in the divided gears in order from low to high. Each time a gear is eliminated, a first-level infrared texture image is obtained. Multiple infrared texture images are drawn to generate facial infrared texture features.

[0010] As a preferred embodiment of the present invention, the information fusion analysis module sets positioning points in the facial infrared pattern features and facial texture features, and creates positioning points at the same position in the facial structure features, and fuses the facial infrared pattern features in sequence by overlapping the positioning points to obtain a 3D facial image.

[0011] As a preferred embodiment of the present invention, the pulse curve constructed by the dynamic analysis module includes a time axis and an acquisition trigger. When the multi-parameter acquisition module is acquiring, a pulse is recorded on the pulse curve. When no acquisition is performed, no pulse is recorded on the pulse curve. The dynamic analysis module divides each acquired 3D facial image into several sub-regions and records the positions of the sub-regions. The dynamic analysis module overlaps the sub-regions at the same position in multiple 3D facial images and reassembles them into a complete 3D facial image after all overlaps are completed to generate a three-dimensional facial model.

[0012] As a preferred embodiment of the present invention, when verifying the three-dimensional facial model, the feedback correction module selects a sub-region of a set size in the three-dimensional facial model and verifies the completeness of the surface in the sub-region. If the surface is complete, the verification value is recorded as 1; if the surface is incomplete, the verification value is recorded as 0; The feedback correction module records the ratio of the sum of the verification values ​​to the number of verification times as the modeling normality index. If the modeling normality index is greater than the set threshold, a modeling completion signal is fed back; if the modeling normality index is not greater than the set threshold, a modeling incomplete signal is fed back.

[0013] Compared with the prior art, the present invention has the following beneficial effects: In the present invention, by fusing visual, depth and infrared image data through a multimodal sensor matrix, accurate three-dimensional reconstruction of facial features can be achieved. Traditional facial recognition technology often relies on single optical imaging, which makes it difficult to fully capture key biological features such as skin texture, subcutaneous microvascular distribution and bone contours. This system uses multi-spectral collaborative acquisition to retain high-resolution surface details and integrate depth information and thermal radiation characteristics, significantly improving the robustness under complex lighting conditions. The multi-dimensional data fusion mechanism effectively solves the feature distortion problem caused by the single information dimension of traditional methods.

[0014] In the present invention, through the real-time interactive verification of dynamic pulse curves and three-dimensional modeling, the continuity analysis of facial dynamic changes can be achieved. The traditional static modeling method has the defect of insufficient sampling rate when capturing instantaneous features such as micro-expressions and muscle tremors. The system constructs time series pulses and associates discrete acquisition frames into a coherent signal stream. It can not only track dynamic features such as pupil changes and facial blood flow changes, but also eliminate interference factors such as breathing and jitter through motion compensation algorithms. Compared with traditional video analysis technology, this solution breaks through the bottleneck of information discontinuity between frames and provides a more accurate time domain analysis tool.

[0015] In the present invention, automatic detection and feedback of modeling errors are achieved through a real-time feedback mechanism for modeling. The normal modeling index is calculated in real time to ensure the completeness of the three-dimensional modeling when the three-dimensional image is output. Automatic optimization can be performed when the three-dimensional modeling is not completed. The self-optimization system avoids the inefficient mode of manual repeated debugging in the traditional process, thereby improving the efficiency of later optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] To facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0017] Figure 1 is a system block diagram of the present invention; Figure 2 It is a system flow chart of the present invention. DETAILED DESCRIPTION

[0018] The following is a clear and complete description of the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0019] Example 1: Please refer to Figure 1 - Figure 2As shown in FIG, a facial feature analysis system based on multimodal fusion and dynamic visualization includes a multi-parameter acquisition module, an information fusion analysis module, a dynamic analysis module, a feedback correction module, and a visualization generation module. The multi-parameter acquisition module acquires multi-dimensional facial features through a sensor matrix. When the multi-parameter acquisition module acquires multi-dimensional facial features, the sensor matrix includes a visible light camera, an infrared sensor, and a 3D structured light device. The multi-parameter acquisition module uses a visible light camera to identify the captured image and highlights the facial information in the image using a preset algorithm. It generates position coordinates based on the position of the highlighted face in the image, controls the visible light camera to perform floating zoom, and acquires high-definition facial images to obtain facial visual images. The multi-parameter acquisition module locates the recognition position of the infrared sensor according to the position coordinates generated by the visible light camera, collects the infrared picture of the face, and obtains the facial infrared image; The multi-parameter acquisition module locates the acquisition position of the 3D structured light device according to the position coordinates generated by the visible light camera, collects the facial image, obtains a 3D point cloud, and performs three-dimensional image construction processing on the 3D point cloud to obtain a facial depth image; After the information fusion analysis module obtains the facial visual image through the multi-parameter acquisition module, it deconstructs the facial texture features. The method by which the information fusion analysis module obtains the facial texture features is as follows: The information fusion analysis module performs pixel grayscale processing on the facial visual image to obtain a grayscale image, and finds the grayscale value of the central pixel as the threshold. It sets the positive direction with a side length of X pixels as the neighborhood, and binarizes the neighborhood pixels to obtain the grayscale of the neighborhood pixels. It calculates the absolute value of the grayscale difference between the grayscale of the neighborhood pixels and the grayscale of the center pixel, and sets an upper limit for the grayscale difference. If the absolute value of the grayscale difference between the grayscale of the neighborhood pixels and the grayscale of the center pixel is greater than the set grayscale value upper limit, it is determined that texture exists and the texture is retained. If the absolute value of the grayscale difference between the grayscale of the neighborhood pixels and the grayscale of the center pixel is less than or equal to the set grayscale value upper limit, it is determined that texture does not exist. The information fusion analysis module retains all pixel areas determined to be textures to form facial texture features; After acquiring the facial depth image, the facial structural features are deconstructed. The information fusion analysis module acquires the facial structural features in the following way: After acquiring the 3D point cloud, the information fusion analysis module selects a set of spatial points as the origin to create a spatial coordinate system. It then calculates the vector distance between each point in the 3D point cloud and the origin one by one to obtain the coordinates of each point in the 3D point cloud, and records each point in the 3D point cloud in the spatial coordinate system. The information fusion analysis module smoothes the adjacent points of the 3D point cloud to generate a complete surface and obtain facial structural features; After acquiring the facial infrared image, the facial infrared texture features are deconstructed. The information fusion analysis module obtains the facial infrared texture features in the following way: The information fusion analysis module divides the facial infrared image into pixels to obtain an image composed of multiple pixels. The information fusion analysis module labels the temperature corresponding to each pixel and divides the temperature into gears according to the temperature corresponding to the pixel. The pixels corresponding to different gear temperatures are removed one by one in the divided gears in ascending order. Each time a gear is removed, a first-level infrared texture image is obtained. Multiple infrared texture images are mapped to generate facial infrared texture features. Specifically, when drawing the infrared texture image, the area aspect ratio of the infrared texture image is calculated. If the aspect ratio is greater than a set value, the infrared texture image is converged to generate a linear pattern. When the aspect ratio is less than or equal to the set value, the infrared texture image is subjected to a dendrite growth process to generate a mesh pattern. The information fusion analysis module sets positioning points in the facial infrared pattern features and facial texture features, and creates positioning points at the same position in the facial structure features. The deconstructed facial texture features, facial structure features and facial infrared pattern features are sequentially fused by overlapping the positioning points to obtain a 3D facial image. Among them, the selection of positioning points can be done by using algorithms to identify the picture and select the midpoint of the distance between the eyes, the midpoint of the cheekbone line and other facial features for positioning.

[0020] The dynamic analysis module obtains the acquisition frequency through the multi-parameter acquisition module and constructs an acquisition pulse curve. The pulse curve constructed by the dynamic analysis module includes a time axis and acquisition trigger. When the multi-parameter acquisition module is acquiring, a pulse is recorded on the pulse curve. When no acquisition is performed, no pulse is recorded on the pulse curve. The dynamic analysis module obtains 3D facial images through the information fusion analysis module. The dynamic analysis module divides each acquired 3D facial image into several sub-regions and records the locations of the sub-regions. The dynamic analysis module overlaps the sub-regions at the same position in multiple 3D facial images and reassembles them into a complete 3D facial image after all overlaps are completed, generating a three-dimensional facial model. After the dynamic analysis module completes modeling, the visualization generation module obtains the three-dimensional facial modeling and outputs the three-dimensional facial modeling in a visual dynamic manner. The output three-dimensional modeling can be rotated through the image viewing tool to view images at any angle. At the same time, the tool can be used to zoom in, which improves the flexibility of viewing.

[0021] Example 2: Please refer to Figure 1 - Figure 2 As shown, the feedback correction module obtains the three-dimensional facial model through the dynamic analysis module and performs verification based on the three-dimensional facial model. When verifying the three-dimensional facial model, the feedback correction module selects a sub-region of a set size in the three-dimensional facial model and verifies the completeness of the surface in the sub-region. If the surface is complete, the verification value is recorded as 1, and if the surface is incomplete, the verification value is recorded as 0; The feedback correction module records the ratio of the sum of the check values ​​and the number of checks as the modeling normality index. If the modeling normal index is greater than the set threshold, a modeling completion signal is fed back; if the modeling normal index is not greater than the set threshold, a modeling incomplete signal is fed back, and the judgment result is fed back to the dynamic analysis module, thereby realizing automatic verification of the three-dimensional modeling. It is also scalable and can automatically record the defect position through the algorithm, so that when the three-dimensional facial modeling is obtained again in the future, the defect position can be automatically optimized or prompted for manual processing.

[0022] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to specific embodiments. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A facial feature analysis system based on multimodal fusion and dynamic visualization, characterized by: It includes a multi-parameter acquisition module, an information fusion analysis module, a dynamic analysis module, a feedback correction module and a visualization generation module. The multi-parameter acquisition module collects multi-faceted facial features through a sensor matrix to obtain facial visual images, facial depth images and facial infrared images; The information fusion analysis module deconstructs facial texture features after acquiring a facial visual image through a multi-parameter acquisition module, deconstructs facial structural features after acquiring a facial depth image, deconstructs facial infrared line features after acquiring a facial infrared image, and fuses the deconstructed facial texture features, facial structural features, and facial infrared line features to obtain a 3D facial image; The dynamic analysis module obtains the acquisition frequency through the multi-parameter acquisition module and constructs an acquisition pulse curve. The dynamic analysis module obtains 3D facial images through the information fusion analysis module and performs dynamic combination analysis on each acquired 3D facial image according to the pulse curve to generate a three-dimensional facial model. The feedback correction module obtains the three-dimensional facial model through the dynamic analysis module, verifies it based on the three-dimensional facial model, obtains a modeling normality index based on the verification result, determines whether the modeling is complete based on the modeling normality index, and then feeds back the judgment result to the dynamic analysis module; After the dynamic analysis module completes the modeling, the visualization generation module obtains the three-dimensional facial modeling and outputs the three-dimensional facial modeling in a visual and dynamic manner.

2. The facial feature analysis system based on multimodal fusion and dynamic visualization according to claim 1, characterized in that: When the multi-parameter acquisition module acquires multi-faceted facial features, the sensor matrix includes a visible light camera, an infrared sensor, and a 3D structured light device. The multi-parameter acquisition module recognizes the acquired image through the visible light camera, highlights the facial information in the image using a preset algorithm, generates position coordinates based on the position of the highlighted face in the image, controls the visible light camera to perform floating zoom, acquires the facial image in high definition, and obtains a facial visual image. The multi-parameter acquisition module locates the recognition position of the infrared sensor according to the position coordinates generated by the visible light camera, collects the infrared picture of the face, and obtains the facial infrared image; The multi-parameter acquisition module locates the acquisition position of the 3D structured light device according to the position coordinates generated by the visible light camera, acquires the face picture to obtain a 3D point cloud, and performs three-dimensional picture construction processing on the 3D point cloud to obtain a facial depth image.

3. The facial feature analysis system based on multimodal fusion and dynamic visualization according to claim 1, characterized in that: The method for the information fusion analysis module to obtain facial texture features is: The information fusion analysis module performs pixel grayscale processing on the facial visual image to obtain a grayscale image, and finds the grayscale value of the central pixel as a threshold, sets the positive direction of the side length of X pixels as a neighborhood, and binarizes the neighborhood pixels to obtain the grayscale of the neighborhood pixels, calculates the absolute value of the grayscale difference between the grayscale of the neighborhood pixels and the grayscale of the center pixel, and sets an upper limit for the grayscale difference. If the absolute value of the difference between the grayscale of the neighborhood pixels and the grayscale of the center pixel is greater than the set upper limit for the grayscale value, it is determined that texture exists and the texture is retained. If the absolute value of the grayscale difference between the grayscale of the neighborhood pixels and the center pixel is less than or equal to the set upper limit for the grayscale value, it is determined that texture does not exist. The information fusion analysis module retains all pixel areas determined to be textures to form facial texture features.

4. The facial feature analysis system based on multimodal fusion and dynamic visualization according to claim 1, characterized in that: The method for the information fusion analysis module to obtain facial structural features is: After acquiring the 3D point cloud, the information fusion analysis module selects a set of spatial points as the origin to create a spatial coordinate system, calculates the vector distance between each point in the 3D point cloud and the origin one by one, obtains the coordinates of each point in the 3D point cloud, and records each point in the 3D point cloud in the spatial coordinate system; The information fusion analysis module performs smoothing processing between adjacent points of the 3D point cloud to generate a complete surface and obtain facial structural features.

5. The facial feature analysis system based on multimodal fusion and dynamic visualization according to claim 1, characterized in that: The method for the information fusion analysis module to obtain facial infrared texture features is: The information fusion analysis module divides the facial infrared image into pixels to obtain an image composed of multiple pixel points. The information fusion analysis module marks the temperature corresponding to each pixel point, and divides the gears according to the temperature corresponding to the pixel points. The pixel points corresponding to different gear temperatures are successively eliminated in the divided gears in order from low to high. Each time a gear is eliminated, a first-level infrared texture image is obtained. Multiple infrared texture images are drawn to generate facial infrared texture features.

6. The facial feature analysis system based on multimodal fusion and dynamic visualization according to claim 1, characterized in that: The information fusion analysis module sets positioning points in the facial infrared pattern features and facial texture features, and creates positioning points at the same position in the facial structure features, and fuses the facial infrared pattern features in sequence by overlapping the positioning points to obtain a 3D facial image.

7. The facial feature analysis system based on multimodal fusion and dynamic visualization according to claim 1 is characterized in that: The pulse curve constructed by the dynamic analysis module includes a time axis and an acquisition trigger. When the multi-parameter acquisition module is acquiring, a pulse is recorded on the pulse curve. When no acquisition is performed, no pulse is recorded on the pulse curve. The dynamic analysis module divides each acquired 3D facial image into several sub-regions and records the positions of the sub-regions. The dynamic analysis module overlaps the sub-regions at the same position in multiple 3D facial images and reassembles them into a complete 3D facial image after all overlaps are completed to generate a three-dimensional facial model.

8. The facial feature analysis system based on multimodal fusion and dynamic visualization according to claim 1, characterized in that: When verifying the three-dimensional facial model, the feedback correction module selects a sub-region of a set size in the three-dimensional facial model and verifies the completeness of the surface in the sub-region. If the surface is complete, the verification value is recorded as 1; if the surface is incomplete, the verification value is recorded as 0; The feedback correction module records the ratio of the sum of the verification values ​​to the number of verification times as the modeling normality index. If the modeling normality index is greater than the set threshold, a modeling completion signal is fed back; if the modeling normality index is not greater than the set threshold, a modeling incomplete signal is fed back.

Images