Facial feature analysis system based on multi-modal fusion and dynamic visualization

Abstract

The application relates to the field of face recognition and aims to solve the problems that a face feature analysis system cannot realize visual output and the face feature items are single, cannot realize automatic parameter fusion, and the feature capturing effect is reduced, specifically a face feature analysis system based on multi-modal fusion and dynamic visualization, which comprises a multi-parameter acquisition module, an information fusion analysis module, a dynamic analysis module, a feedback correction module and a visualization generation module; in the application, multi-modal sensor matrix fusion visual, depth and infrared image data can realize stereoscopic and accurate reconstruction of face features, thereby improving the feature information missing problem caused by single modal acquisition in traditional methods, and through real-time interactive verification of the continuity of face acquisition and three-dimensional modeling, the limitation that micro-expression and physiological changes cannot be captured in static modeling in traditional methods is improved, the modeling error is automatically corrected through a feedback mechanism, and the problem of low artificial intervention efficiency in traditional methods is improved.

Description

Technical Field

[0001] This invention relates to the field of facial recognition, specifically a facial feature analysis system based on multimodal fusion and dynamic visualization. Background Technology

[0002] Facial recognition software can be categorized into a large class of technologies called biometrics. Facial recognition systems, with face recognition technology at their core, are an emerging biometric technology. Faces are popular because they are unique, easy to collect, and do not require the cooperation of the person being photographed. Facial recognition systems have a wide range of applications, including eliminating voting fraud, verifying identity for cash withdrawals, and computer security.

[0003] The basic process of facial recognition software for capturing and comparing images is as follows: When the system is connected to a video surveillance system, the recognition software searches for facial information in the camera's field of view. It uses a multi-scale algorithm to search for facial images at low resolution. After detecting a head-like shape, it switches to a high-resolution search to determine the head's position, size, and posture. Regardless of the head's position or distance from the camera, a normalization process can be performed. Light does not affect the normalization process. The system converts facial data into a unique code. Through encoding, it is easier to compare newly captured facial data with stored facial data. The newly captured facial data is compared with the stored data and linked to at least one stored facial image. The core of the facial recognition system is the local feature analysis algorithm.

[0004] In existing technologies, when analyzing local features, a single feature analysis method is generally used. Therefore, it is impossible to comprehensively compare features from multiple directions, which reduces the accuracy of feature capture and comparison. At the same time, in order to ensure the efficiency of comparison, existing technologies generally record facial images in the form of character encoding, which results in a lack of visual output of the recorded facial images and affects the scope of application of facial recognition in some specific aspects.

[0005] To address the aforementioned technical problems, this application proposes a solution. Summary of the Invention

[0006] In this invention, by fusing visual, depth, and infrared image data through a multimodal sensor matrix, a three-dimensional and accurate reconstruction of facial features can be achieved. This improves the problem of missing feature information caused by single-modal acquisition in traditional methods. Furthermore, by continuously acquiring facial features and interactively verifying 3D modeling in real time, the limitations of static modeling in traditional methods, which struggles to capture micro-expressions and physiological changes, are overcome. Through a feedback mechanism, automatic correction of modeling errors and intuitive presentation of results are achieved, thereby improving the low efficiency of manual intervention in traditional methods. This invention also addresses the problems of facial feature analysis systems being unable to provide visual output and the limitations of single facial feature items, which prevent automatic parameter fusion and reduce feature capture effectiveness. Therefore, a facial feature analysis system based on multimodal fusion and dynamic visualization is proposed.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] The facial feature analysis system based on multimodal fusion and dynamic visualization includes a multi-parameter acquisition module, an information fusion analysis module, a dynamic analysis module, a feedback correction module, and a visualization generation module. The multi-parameter acquisition module acquires multi-dimensional facial features through a sensor matrix to obtain facial visual images, facial depth images, and facial infrared images.

[0009] The information fusion analysis module acquires facial visual images through the multi-parameter acquisition module, deconstructs facial texture features, acquires facial depth images, deconstructs facial structural features, acquires facial infrared images, deconstructs facial infrared texture features, and then fuses the deconstructed facial texture features, facial structural features, and facial infrared texture features to obtain a 3D facial image.

[0010] The dynamic analysis module obtains the acquisition frequency through the multi-parameter acquisition module and constructs the acquisition pulse curve. The dynamic analysis module obtains 3D facial images through the information fusion analysis module and performs dynamic combination analysis on each acquired 3D facial image according to the pulse curve to generate a three-dimensional facial model.

[0011] The feedback correction module obtains a 3D facial model through the dynamic analysis module, verifies the 3D facial model, obtains a modeling normality index based on the verification result, determines whether the modeling is complete based on the modeling normality index, and then feeds the judgment result back to the dynamic analysis module.

[0012] After the dynamic analysis module completes the modeling, the visualization generation module acquires the 3D facial model and outputs the 3D facial model in a dynamic visualization.

[0013] In a preferred embodiment of the present invention, when the multi-parameter acquisition module acquires multiple facial features, the sensor matrix includes a visible light camera, an infrared sensor, and a 3D structured light device. The multi-parameter acquisition module identifies the acquired image through the visible light camera and highlights the facial information in the image through a preset algorithm. Based on the position of the highlighted face in the image, position coordinates are generated, and the visible light camera is controlled to perform floating zoom to acquire the facial image in high definition and obtain a facial visual image.

[0014] The multi-parameter acquisition module locates the infrared sensor's recognition position based on the position coordinates generated by the visible light camera, acquires the infrared image of the face, and obtains a facial infrared image.

[0015] The multi-parameter acquisition module locates the acquisition position of the 3D structured light device based on the position coordinates generated by the visible light camera, acquires the face image, obtains a 3D point cloud, and performs three-dimensional image construction processing on the 3D point cloud to obtain a facial depth image.

[0016] In a preferred embodiment of the present invention, the method by which the information fusion analysis module obtains facial texture features is as follows:

[0017] The information fusion analysis module performs pixel-level grayscale processing on the facial visual image to obtain a grayscale image. It then finds the grayscale value of the center pixel as a threshold, sets the positive direction with a side length of X pixels as the neighborhood, and binarizes the neighborhood pixels to obtain their grayscale values. It calculates the absolute value of the grayscale difference between the grayscale values ​​of the neighborhood pixels and the center pixel, and sets an upper limit for the grayscale difference. If the absolute value of the grayscale difference between the neighborhood pixels and the center pixel is greater than the set upper limit, it is determined that a texture exists and the texture is preserved. If the absolute value of the grayscale difference between the neighborhood pixels and the center pixel is less than or equal to the set upper limit, it is determined that no texture exists.

[0018] The information fusion analysis module retains all pixel regions identified as textures to form facial texture features.

[0019] In a preferred embodiment of the present invention, the method by which the information fusion analysis module acquires facial structural features is as follows:

[0020] After acquiring the 3D point cloud, the information fusion analysis module selects a set of spatial points as the origin to create a spatial coordinate system. It then calculates the vector distance between each point in the 3D point cloud and the origin to obtain the coordinates of each point in the 3D point cloud and records each point in the 3D point cloud into the spatial coordinate system.

[0021] The information fusion analysis module smooths the adjacent points of the 3D point cloud to generate a complete curved surface, thus obtaining facial structure features.

[0022] In a preferred embodiment of the present invention, the method by which the information fusion analysis module obtains facial infrared texture features is as follows:

[0023] The information fusion analysis module divides the facial infrared image into pixels to obtain an image composed of multiple pixels. The information fusion analysis module labels the temperature corresponding to each pixel and divides the temperature into levels according to the temperature of the pixel. Then, in order from low to high, the pixels corresponding to different temperature levels are successively removed. Each removal of a level yields a first-level infrared texture image. Multiple infrared texture images are drawn to generate facial infrared texture features.

[0024] In a preferred embodiment of the present invention, the information fusion analysis module sets positioning points in facial infrared texture features and facial texture features, and creates positioning points at the same positions in facial structural features. The facial infrared texture features are fused sequentially by means of overlapping positioning points to obtain a 3D facial image.

[0025] In a preferred embodiment of the present invention, the pulse curve constructed by the dynamic analysis module includes a time axis and acquisition trigger. When the multi-parameter acquisition module performs acquisition, a pulse is recorded on the pulse curve. When no acquisition is performed, no pulse is recorded on the pulse curve.

[0026] The dynamic analysis module divides each acquired 3D facial image into several sub-regions and records the location of each sub-region. The dynamic analysis module overlaps the sub-regions at the same location in multiple 3D facial images and recombines them into a complete 3D facial image after all overlaps are completed, generating a three-dimensional facial model.

[0027] In a preferred embodiment of the present invention, when the feedback correction module verifies the three-dimensional facial model, it selects a sub-region of a set size in the three-dimensional facial model and verifies the integrity of the surface in the sub-region. If the surface is complete, the verification value is recorded as 1; if the surface is incomplete, the verification value is recorded as 0.

[0028] The feedback correction module records the ratio of the sum of the verification values ​​to the number of verifications as the modeling normal index. If the modeling normal index is greater than the set threshold, a modeling completion signal is fed back. If the modeling normal index is not greater than the set threshold, a modeling incomplete signal is fed back.

[0029] Compared with the prior art, the beneficial effects of the present invention are:

[0030] In this invention, by fusing visual, depth, and infrared image data through a multimodal sensor matrix, it is possible to achieve accurate three-dimensional reconstruction of facial features. Traditional facial recognition technologies often rely on single optical imaging, which makes it difficult to fully capture key biological features such as skin texture, subcutaneous microvascular distribution, and skeletal contours. This system, through multispectral collaborative acquisition, not only preserves high-resolution surface details but also integrates depth information and thermal radiation features, significantly improving robustness under complex lighting conditions. The multidimensional data fusion mechanism effectively solves the feature distortion problem caused by the single information dimension in traditional methods.

[0031] In this invention, the continuous analysis of facial dynamic changes can be achieved through real-time interactive verification of dynamic pulse curves and 3D modeling. Traditional static modeling methods suffer from insufficient sampling rate when capturing instantaneous features such as micro-expressions and muscle tremors. However, this system constructs time-series pulses to associate discrete acquisition frames into a coherent signal stream. It can not only track dynamic features such as pupil changes and facial blood flow changes, but also eliminate interference factors such as breathing and shaking through motion compensation algorithms. Compared with traditional video analysis technology, this solution breaks through the bottleneck of inter-frame information breakage and provides a more accurate temporal domain analysis tool.

[0032] In this invention, a real-time modeling feedback mechanism is used to achieve automated detection and feedback of modeling errors. By calculating the modeling normality index in real time, the integrity of the 3D modeling is guaranteed when outputting the 3D image. When the 3D modeling is not completed, it can be automatically optimized. The self-optimization system avoids the inefficient mode of repeated manual debugging in the traditional process and improves the efficiency of later optimization. Attached Figure Description

[0033] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0034] Figure 1 This is a system block diagram of the present invention;

[0035] Figure 2 This is a system flowchart of the present invention. Detailed Implementation

[0036] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0037] Example 1: Please refer to Figure 1 - Figure 2As shown, the facial feature analysis system based on multimodal fusion and dynamic visualization includes a multi-parameter acquisition module, an information fusion analysis module, a dynamic analysis module, a feedback correction module, and a visualization generation module. The multi-parameter acquisition module acquires multi-faceted facial features through a sensor matrix. When the multi-parameter acquisition module acquires multi-faceted facial features, the sensor matrix includes a visible light camera, an infrared sensor, and a 3D structured light device.

[0038] The multi-parameter acquisition module identifies the captured images through a visible light camera and highlights the facial information in the images using a preset algorithm. Based on the position of the highlighted face in the image, it generates position coordinates and controls the visible light camera to perform floating zoom, thereby capturing high-definition facial images and obtaining facial visual images.

[0039] The multi-parameter acquisition module locates the infrared sensor's recognition position based on the position coordinates generated by the visible light camera, acquires infrared images of the face, and obtains facial infrared images.

[0040] The multi-parameter acquisition module locates the acquisition position of the 3D structured light device based on the position coordinates generated by the visible light camera, acquires the face image, obtains a 3D point cloud, and performs three-dimensional image construction processing on the 3D point cloud to obtain a facial depth image.

[0041] After acquiring facial visual images through the multi-parameter acquisition module, the information fusion analysis module deconstructs the facial texture features. The method used by the information fusion analysis module to acquire facial texture features is as follows:

[0042] The information fusion analysis module performs pixel-level grayscale processing on the facial visual image to obtain a grayscale image. It then finds the grayscale value of the center pixel as a threshold, sets the positive direction with a side length of X pixels as the neighborhood, and binarizes the neighborhood pixels to obtain their grayscale values. It calculates the absolute value of the grayscale difference between the grayscale of the neighborhood pixels and the grayscale of the center pixel, and sets an upper limit for the grayscale difference. If the absolute value of the grayscale difference between the neighborhood pixels and the center pixel is greater than the set upper limit, it is determined that a texture exists and the texture is preserved. If the absolute value of the grayscale difference between the neighborhood pixels and the center pixel is less than or equal to the set upper limit, it is determined that no texture exists.

[0043] The information fusion analysis module retains all pixel regions identified as textures to form facial texture features;

[0044] After acquiring the facial depth image, the facial structural features are deconstructed. The information fusion analysis module acquires the facial structural features using the following method:

[0045] After acquiring the 3D point cloud, the information fusion analysis module selects a set of spatial points as the origin to create a spatial coordinate system. It then calculates the vector distance between each point in the 3D point cloud and the origin to obtain the coordinates of each point in the 3D point cloud and records each point in the 3D point cloud into the spatial coordinate system.

[0046] The information fusion analysis module smooths the adjacent points of the 3D point cloud to generate a complete surface and obtain facial structure features.

[0047] After acquiring the facial infrared image, the facial infrared texture features are deconstructed. The method used by the information fusion analysis module to acquire the facial infrared texture features is as follows:

[0048] The information fusion analysis module divides the facial infrared image into pixels to obtain an image composed of multiple pixels. The information fusion analysis module labels the temperature corresponding to each pixel and divides the temperature into levels according to the temperature of the pixel. Then, in order from low to high, the pixels corresponding to different temperature levels are successively removed. Each level removed yields a first-level infrared texture image. Multiple infrared texture images are drawn to generate facial infrared texture features.

[0049] Specifically, when drawing infrared texture images, the aspect ratio of the region of the infrared texture image is calculated. If the aspect ratio is greater than the set value, the infrared texture image is converged to generate line patterns. When the aspect ratio is less than or equal to the set value, the infrared texture image is subjected to tree diagram growth processing to generate mesh patterns.

[0050] The information fusion analysis module sets positioning points in facial infrared texture features and facial texture features, and creates positioning points at the same positions in facial structural features. By merging the deconstructed facial texture features, facial structural features and facial infrared texture features in sequence through the overlapping of positioning points, a 3D facial image is obtained.

[0051] Among them, the selection of positioning points can be achieved by using algorithms to identify the image and select the midpoint between the eyes, the midpoint of the line connecting the cheekbones, and other facial features for positioning.

[0052] The dynamic analysis module obtains the acquisition frequency through the multi-parameter acquisition module and constructs the acquisition pulse curve. The pulse curve constructed by the dynamic analysis module includes a time axis and acquisition trigger. When the multi-parameter acquisition module is acquiring data, one pulse is recorded on the pulse curve. When no acquisition is being performed, no pulse is recorded on the pulse curve.

[0053] The dynamic analysis module acquires 3D facial images through the information fusion analysis module. The dynamic analysis module divides each acquired 3D facial image into several sub-regions and records the location of each sub-region. The dynamic analysis module overlaps the sub-regions at the same location in multiple 3D facial images and recombines them into a complete 3D facial image after all overlaps are completed, generating a three-dimensional facial model.

[0054] After the dynamic analysis module completes the modeling, the visualization generation module acquires a 3D facial model and outputs it dynamically and visually. The output 3D model can be rotated using an image viewing tool to view the image from any angle, and can also be zoomed in to improve viewing flexibility.

[0055] Example 2: Please refer to Figure 1 - Figure 2 As shown, the feedback correction module obtains the 3D facial model through the dynamic analysis module and performs verification based on the 3D facial model. When verifying the 3D facial model, the feedback correction module selects a sub-region of a set size in the 3D facial model and verifies the integrity of the surface in the sub-region. If the surface is complete, the verification value is recorded as 1; if the surface is incomplete, the verification value is recorded as 0.

[0056] The feedback correction module records the ratio of the sum of the verification values ​​to the number of verifications as the modeling normality index.

[0057] If the modeling normality index is greater than the set threshold, a modeling completion signal is fed back. If the modeling normality index is not greater than the set threshold, a modeling incomplete signal is fed back. The judgment result is then fed back to the dynamic analysis module, thereby realizing automatic verification of 3D modeling. It is also scalable and can automatically record the location of defects through algorithms, so that when 3D facial modeling is acquired again in the future, the location of defects can be automatically optimized or manual processing can be reminded.

[0058] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A facial feature analysis system based on multimodal fusion and dynamic visualization, characterized in that, It includes a multi-parameter acquisition module, an information fusion analysis module, a dynamic analysis module, a feedback correction module, and a visualization generation module. The multi-parameter acquisition module acquires multi-faceted facial features through a sensor matrix to obtain facial visual images, facial depth images, and facial infrared images. The information fusion analysis module acquires facial visual images through the multi-parameter acquisition module, deconstructs facial texture features, acquires facial depth images, deconstructs facial structural features, acquires facial infrared images, deconstructs facial infrared texture features, and then fuses the deconstructed facial texture features, facial structural features, and facial infrared texture features to obtain a 3D facial image. The dynamic analysis module obtains the acquisition frequency through the multi-parameter acquisition module and constructs the acquisition pulse curve. The dynamic analysis module obtains 3D facial images through the information fusion analysis module and performs dynamic combination analysis on each acquired 3D facial image according to the pulse curve to generate a three-dimensional facial model. The feedback correction module obtains a 3D facial model through the dynamic analysis module, verifies the 3D facial model, obtains a modeling normality index based on the verification result, determines whether the modeling is complete based on the modeling normality index, and then feeds the judgment result back to the dynamic analysis module. After the dynamic analysis module completes the modeling, the visualization generation module obtains the 3D facial model and outputs the 3D facial model in a dynamic visualization. The pulse curve constructed by the dynamic analysis module includes a time axis and acquisition trigger. When the multi-parameter acquisition module acquires data, a pulse is recorded on the pulse curve. When no acquisition is performed, no pulse is recorded on the pulse curve. The dynamic analysis module divides each acquired 3D facial image into several sub-regions and records the location of each sub-region. The dynamic analysis module overlaps the sub-regions at the same location in multiple 3D facial images and recombines them into a complete 3D facial image after all overlaps are completed, generating a three-dimensional facial model.

2. The facial feature analysis system based on multimodal fusion and dynamic visualization according to claim 1, characterized in that, When the multi-parameter acquisition module acquires multiple facial features, the sensor matrix includes a visible light camera, an infrared sensor, and a 3D structured light device. The multi-parameter acquisition module identifies the acquired image through the visible light camera and highlights the facial information in the image through a preset algorithm. It generates position coordinates based on the position of the highlighted face in the image, controls the visible light camera to perform floating zoom, and acquires the facial image in high definition to obtain a facial visual image. The multi-parameter acquisition module locates the infrared sensor's recognition position based on the position coordinates generated by the visible light camera, acquires the infrared image of the face, and obtains a facial infrared image. The multi-parameter acquisition module locates the acquisition position of the 3D structured light device based on the position coordinates generated by the visible light camera, acquires the face image, obtains a 3D point cloud, and performs three-dimensional image construction processing on the 3D point cloud to obtain a facial depth image.

3. The facial feature analysis system based on multimodal fusion and dynamic visualization according to claim 1, characterized in that, The method for deconstructing facial texture features by the information fusion analysis module is as follows: The information fusion analysis module performs pixel-level grayscale processing on the facial visual image to obtain a grayscale image. It then finds the grayscale value of the center pixel as a threshold, sets the positive direction with a side length of X pixels as the neighborhood, and binarizes the neighborhood pixels to obtain their grayscale values. It calculates the absolute value of the grayscale difference between the grayscale values ​​of the neighborhood pixels and the center pixel, and sets an upper limit for the grayscale difference. If the absolute value of the grayscale difference between the neighborhood pixels and the center pixel is greater than the set upper limit, it is determined that a texture exists and the texture is preserved. If the absolute value of the grayscale difference between the neighborhood pixels and the center pixel is less than or equal to the set upper limit, it is determined that no texture exists. The information fusion analysis module retains all pixel regions identified as textures to form facial texture features.

4. The facial feature analysis system based on multimodal fusion and dynamic visualization according to claim 2, characterized in that, The information fusion analysis module uses the following method to deconstruct facial structural features: After acquiring the 3D point cloud, the information fusion analysis module selects a set of spatial points as the origin to create a spatial coordinate system. It then calculates the vector distance between each point in the 3D point cloud and the origin to obtain the coordinates of each point in the 3D point cloud and records each point in the 3D point cloud into the spatial coordinate system. The information fusion analysis module smooths the adjacent points of the 3D point cloud to generate a complete curved surface, thus obtaining facial structure features.

5. The facial feature analysis system based on multimodal fusion and dynamic visualization according to claim 1, characterized in that, The method used by the information fusion analysis module to deconstruct facial infrared texture features is as follows: The information fusion analysis module divides the facial infrared image into pixels to obtain an image composed of multiple pixels. The information fusion analysis module labels the temperature corresponding to each pixel and divides the temperature into levels according to the temperature of the pixel. Then, in order from low to high, the pixels corresponding to different temperature levels are successively removed. Each removal of a level yields a first-level infrared texture image. Multiple infrared texture images are drawn to generate facial infrared texture features.

6. The facial feature analysis system based on multimodal fusion and dynamic visualization according to claim 1, characterized in that, The information fusion analysis module sets positioning points in facial infrared texture features and facial texture features, and creates positioning points at the same positions in facial structural features. By merging the facial infrared texture features sequentially through the overlapping of positioning points, a 3D facial image is obtained.

7. The facial feature analysis system based on multimodal fusion and dynamic visualization according to claim 1, characterized in that, When verifying the 3D facial model, the feedback correction module selects a sub-region of a set size in the 3D facial model and verifies the integrity of the surface in the sub-region. If the surface is complete, the verification value is recorded as 1; if the surface is incomplete, the verification value is recorded as 0. The feedback correction module records the ratio of the sum of the verification values ​​to the number of verifications as the modeling normal index. If the modeling normal index is greater than the set threshold, a modeling completion signal is fed back. If the modeling normal index is not greater than the set threshold, a modeling incomplete signal is fed back.

Images