Facial dynamic feature multi-modal recognition and analysis system based on artificial intelligence

Abstract

The invention belongs to the technical field of facial feature recognition, and particularly discloses a facial dynamic feature multi-modal recognition analysis system based on artificial intelligence. The system comprises an environment data acquisition module, a modal data acquisition module, a modal feature extraction module, a modal fusion confirmation module, a feature fusion processing module and an identification feedback display terminal. According to the method, the interference degree of the illumination environment is counted, the modal data combination mode is confirmed, a large amount of redundant information brought by excessive types of data is avoided, the collected data size is relatively controllable, and on the other hand, a system capable of carrying out reliable data collection in a full-illumination scene is further constructed. And meanwhile, by confirming the multi-modal data fusion type, the problem that current modal fusion is not high in pertinence is solved, and by setting the feature correction factors corresponding to the modal data, the recognition quality of the facial special data and the accuracy and effectiveness of feature extraction are guaranteed.

Description

Technical Field

[0001] The present invention belongs to the technical field of facial feature recognition, and in particular, relates to an artificial intelligence-based multimodal recognition and analysis system for facial dynamic features. Background Art

[0002] With the popularity of smart devices, human-computer interaction has become increasingly important. Multimodal recognition and analysis of facial dynamic features can achieve more natural and intelligent human-computer interaction. For example, by analyzing the user's facial expressions and movements, smart devices can understand the user's intentions and emotional state, thereby providing more personalized services and responses, and improving the user experience.

[0003] Data of different modalities, such as video images, audio, depth information, etc., need to be precisely synchronized in time and will also be affected by environmental factors. The current multimodal recognition and analysis of facial dynamic features still has the following deficiencies: 1. The current modal fusion is not very targeted. Due to the different sampling frequencies of various sensors at different modal acquisition ends, it is difficult to ensure complete synchronization of data, which brings difficulties to subsequent multimodal fusion analysis. Currently, the impact of time synchronization has not been further processed and analyzed, resulting in certain errors in the final multimodal recognition and analysis results of facial dynamic features, and accuracy cannot be guaranteed.

[0004] 2. The collected data may be affected by various noises, such as lighting changes, background noise, sensor noise, etc., which will affect the quality of the data and the accuracy of feature extraction. Currently, the results of different modal analysis have not been corrected according to the specific environmental impacts, and the effectiveness and reliability of subsequent identification cannot be guaranteed. Summary of the invention

[0005] In view of this, in order to solve the problems raised in the above background technology, a facial dynamic feature multimodal recognition and analysis system based on artificial intelligence is proposed.

[0006] The purpose of the present invention can be achieved through the following technical solutions: The present invention provides a facial dynamic feature multimodal recognition and analysis system based on artificial intelligence, the system comprising: an environmental data acquisition module, used to collect environmental data in the user's area to obtain collected environmental information.

[0007] The modal data acquisition module is used to collect multimodal data from users, obtain the collected modal data, and record the acquisition characteristics of each modal data.

[0008] The modal feature extraction module is used to preprocess each modal data and extract the features of each modal data.

[0009] The modal fusion confirmation module is used to confirm the multimodal data fusion type based on the characteristics and acquisition characteristics of each modal data.

[0010] The feature fusion processing module is used to set the feature correction factor corresponding to each modal data, import the facial feature recognition task, and then analyze the features of each modal data to output the facial feature recognition result of the target user.

[0011] The recognition feedback display terminal is used to provide the target user's facial feature recognition results to the user in a visual manner.

[0012] Compared with the prior art, the present invention has the following beneficial effects: (1) By statistically analyzing the interference degree of the lighting environment and confirming the combination pattern of modal data, the present invention avoids a large amount of redundant information caused by too many types of data, making the amount of collected data relatively controllable and not causing resource waste or slow processing speed due to too complex data. On the other hand, it constructs a system that can perform reliable data collection in full-light scenes, greatly enhancing the overall resistance to lighting environment interference, and ensuring that sufficient data for accurate identification of facial dynamic features can still be collected in complex interference environments.

[0013] (2) The present invention determines the type of multimodal data fusion by combining the characteristics and acquisition characteristics of each modal data, effectively solving the problem of the current modal fusion being not very targeted, fully considering and alleviating the impact of data asynchrony caused by the sampling frequencies of various sensors, ensuring the convenience of subsequent multimodal data fusion, and reducing the error of the final facial dynamic feature multimodal recognition and analysis results, further improving the accuracy of the subsequent final facial dynamic feature multimodal recognition and analysis results.

[0014] (3) The present invention sets the feature correction factor corresponding to each modal data based on the collection characteristics and environmental information of each modal data, thereby making up for the current deficiency of not correcting the analysis results of different modalities according to the specific environmental influences, reducing the interference of environmental factors such as illumination and background noise, ensuring the recognition quality of special facial data and the accuracy of feature extraction, and improving the reliability and effectiveness of subsequent recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0016] Figure 1 It is a schematic diagram of the system module structure connection of the present invention.

[0017] Figure 2 It is a schematic flow chart of the overall implementation steps of the present invention.

[0018] Figure 3 It is a schematic diagram of the multimodal data fusion type confirmation process of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] See also Figure 1 to Figure 2 As shown, the present invention provides a facial dynamic feature multimodal recognition and analysis system based on artificial intelligence, the system includes an environmental data acquisition module, a modal data acquisition module, a modal feature extraction module, a modal fusion confirmation module, a feature fusion processing module and a recognition feedback display terminal.

[0021] In the above, the modal data acquisition module is connected to the modal feature extraction module and the environmental data acquisition module respectively, the modal fusion confirmation module is connected to the modal feature extraction module and the feature fusion processing module respectively, and the feature fusion processing module is also connected to the recognition feedback display terminal.

[0022] The environmental data collection module is used to collect environmental data in the area where the user is located to obtain collected environmental information.

[0023] Specifically, the environmental information includes the light intensity at each collection point at each collection time point, the temperature at each collection time point, and the signal-to-noise ratio at each collection time point.

[0024] Among them, the light intensity is collected by an interactive device, such as a light intensity sensor at each position carried by an interactive robot, the temperature is collected by a carried temperature sensor, and the signal-to-noise ratio is collected by a carried noise sensor.

[0025] The modal data acquisition module is used to perform multimodal data acquisition on the user, obtain the acquired modal data, and record the acquisition characteristics of each modal data.

[0026] Specifically, before performing multimodal data collection, the modal data combination mode is confirmed, and the specific confirmation process includes: S1, extracting the collected light intensity of each collection point at each collection time point from the collected environmental information, calculating the standard deviation of the collected light intensity at different collection points at the same collection time point, and using the standard deviation calculation result as the light intensity difference.

[0027] S2. Filter out the maximum light intensity difference from the light intensity differences at each acquisition time point, and record it as

[0028] S3, with the collection time point as the horizontal coordinate and the collection light intensity as the vertical coordinate, construct the light intensity change curve of each collection point, calculate its slope as the light intensity change rate, and select the minimum value, recorded as p E .

[0029] S4. Calculate the average of the light intensity collected at different collection points at the same collection time point, and select the minimum value from the average light intensity collected at each collection point, which is recorded as E. min .

[0030] S5. Count the interference of the lighting environment, denoted as λ E , E′ and are the image acquisition interference light intensity and image acquisition interference light intensity difference of the set reference respectively, and p′ is the light intensity change rate of the set reference.

[0031] S6. Combining the visual image data, the depth image data and the audio data as mode I, and combining the visual image data, the depth image data, the infrared image data and the audio data as mode II.

[0032] S7, if λ E ≤0, take mode I as the modal data combination mode, if λ E >0, mode II is used as the modal data combination mode.

[0033] The embodiments of the present invention statistically analyze the interference degree of the lighting environment and confirm the combination pattern of modal data. On the one hand, this avoids a large amount of redundant information caused by too many types of data, making the amount of collected data relatively controllable and avoiding problems such as waste of resources or slow processing speed due to excessively complex data. On the other hand, this system constructs a system that can perform reliable data collection in full-lighting scenes, greatly enhancing the overall resistance to lighting environment interference and ensuring that sufficient data for accurately identifying facial dynamic features can still be collected in a complex interference environment.

[0034] What needs to be added about step S5 is that the embodiment of the present invention is designed with the background of human-computer interaction based on the dynamic facial features of personnel as the research object, so it is mostly used in general indoor environment scenes. It is generally believed that when the light intensity is lower than 50-100Lux for image acquisition in general indoor environments, it may cause obvious interference to image acquisition. Because such lighting is dim, it is easy to cause noise in the image, dark main parts such as the face, loss of details, etc., affecting subsequent facial feature recognition and other related processing. Therefore, in this scenario, the reference image acquisition interference light intensity can be set to a value of about 50-100Lux. For example, E′ can be 100Lux, and Lux ​​represents the unit of illumination.

[0035] It should be added that in some scenarios with high requirements on image quality and accurate recognition of facial features, such as face recognition access control system application scenarios, the difference in light intensity at different collection points, such as light sources at different angles and positions, should be as small as possible. It is generally hoped that the standard deviation is controlled within 10-20l ux. If it exceeds this range, it may cause uneven lighting in different areas of the face, resulting in shadows, unnatural brightness transitions and other problems, affecting the accuracy of facial feature extraction and recognition. Therefore, the reference image acquisition interference light intensity difference can be set to a value of about 10-20l ux. For some ordinary monitoring or image acquisition scenarios, when the requirements for light uniformity are not so strict, the reference image acquisition interference light intensity difference can be set to a value of 30-50l ux. Since the application scenario of the present invention is human-computer interaction, that is, the accuracy is also required to be high, for example, The value can be 20l ux.

[0036] It should also be added that in an indoor environment with relatively fixed and stable lighting, the light intensity will theoretically not change rapidly over time, and the rate of change of light intensity is expected to be close to 0. The reference light intensity change rate can usually be set to a very small range of 0-0.1l ux / second. Once the rate of change exceeds this value, such as when the light suddenly flickers or other temporary light sources interfere, resulting in a rapid change in light intensity, the brightness may flicker in the captured image sequence, interfering with the analysis of dynamic facial features. Therefore, in this scenario, it is better to take a value close to the lower limit. At the same time, when the light intensity is weakening, it has a greater impact on image acquisition. Therefore, for example, p′ can be taken as -0.1.

[0037] In another specific embodiment, performing multimodal data collection includes: if the current modal data combination mode is mode I.

[0038] A high-definition camera is used to continuously obtain a sequence of visual images of the user's face at different angles with a set sampling interval.

[0039] The depth camera is used to continuously obtain a sequence of depth images of the user's face at different angles with a set sampling interval.

[0040] The microphone is used to collect the user's audio data in real time to form a user audio data set.

[0041] If the current modal data combination mode is mode II, the multimodal data acquisition steps of mode I are repeated, and an infrared camera is used to continuously acquire infrared image sequences of the user's face at different angles at a set frame rate.

[0042] It should be added that, as the user moves, when continuously acquiring a sequence of visual images of the user's face at different angles at a set acquisition frequency, images under multiple lighting conditions and multiple distances can be covered to ensure that various dynamic changes of the face can be captured.

[0043] The modal feature extraction module is used to preprocess each modal data and extract the features of each modal data.

[0044] It should be added that the purpose of preprocessing the data of each modality is to improve data quality, remove noise, and make the data format meet the requirements of subsequent feature extraction. For example, for image data, operations such as denoising, normalization, and cropping may be performed. For audio data, noise reduction, filtering, and framing will be performed.

[0045] In a specific embodiment, the image data can be denoised by a mean filter or a median filter. Specifically, when the mean filter is selected, the average value of each pixel in the image and its neighboring pixels can be calculated, and the average value is used to replace the value of the current pixel, so as to achieve the purpose of smoothing the image and removing noise. For example, for a 3×3 neighborhood window, the grayscale values ​​of the 9 pixels in the window are added and divided by 9, and the average value is assigned to the pixel in the center of the window. In this way, the isolated noise points in the image can be averaged and integrated into the grayscale values ​​of the surrounding pixels, making the image smoother. The median filter is selected to replace the grayscale value of each pixel in the image with the median of the grayscale values ​​of its neighboring pixels. For example, in a 5×5 neighborhood, the grayscale values ​​of all pixels are sorted, and the middle value is taken as the new grayscale value of the current pixel. In this way, the abnormal grayscale value of the noise point is not easy to affect the processed image, and the edge and other detail information of the image can be better preserved.

[0046] In a specific embodiment, the normalization of image data can be performed by linear normalization, which can be performed by linearly mapping the grayscale values ​​of image pixels from the original value range to a new specified range, usually [0, 1] or [0, 255], and the calculation formula is: new value = (original value - minimum value) / (maximum value - minimum value) × new range maximum value. For example, the original range of the grayscale value of pixels in the image is [10, 200], and it is normalized to the range of [0, 1]. For a pixel with a grayscale value of 50, its normalized value is obtained by calculating (50-10) / (200-10), so that the pixel values ​​of different images or different regions of the same image have a unified scale, which is convenient for subsequent processing and comparison, wherein [0, 255] corresponds to an 8-bit image.

[0047] In another specific embodiment, the purpose of cropping is to determine the image area to be retained and remove the parts outside the area. For example, in face recognition, only the central area containing the face may be of interest. By setting the coordinate range of cropping, the redundant background around the face in the image can be cropped to reduce the amount of unnecessary data, while highlighting the facial area of ​​focus, which is beneficial to subsequent feature extraction and other operations.

[0048] It should be added that the spectral subtraction method can be used to reduce the noise of audio data. First, the noisy audio signal is fast Fourier transformed to convert it from the time domain to the frequency domain to obtain a spectrum representation. Then the noise spectrum is estimated, and the noise spectrum is subtracted from the noise spectrum of the noisy frequency signal. The result is then converted back to the time domain through the inverse fast Fourier transform to achieve the purpose of noise reduction. It can be understood that the estimated noise spectrum can usually be obtained in the silent part at the beginning of the audio or through some statistical methods to obtain the noise spectrum estimate.

[0049] Specifically, the audio data may be filtered through high-pass filtering or low-pass filtering, wherein high-pass filtering and low-pass filtering are common existing filtering methods and will not be described in detail here. The audio data is mainly processed through corresponding filters.

[0050] Specifically, the audio data is divided into frames to facilitate subsequent operations such as feature extraction. It is usually divided into shorter frames. Adjacent frames may overlap partially, and the overlapping portion generally accounts for a certain proportion of the frame length, such as 50% overlap. For example, a 5-second audio segment is divided into frames with a length of 0.02 seconds per frame. If there is a 50% overlap, there will be 10 milliseconds of repetition between two adjacent frames. After framing, the audio features can be analyzed independently on each frame, while taking into account the continuity and time correlation of the audio signal, so that the extracted features can better reflect the changes of the audio over time.

[0051] In a specific embodiment, the features of visual image data, depth image data and infrared image data can be extracted by image recognition technology, and the features of audio data can be extracted by speech recognition technology. Among them, image recognition technology and speech recognition technology are relatively mature technologies, and their specific extraction process will not be repeated.

[0052] Specifically, each modal data includes visual image data, depth image data, infrared image data and audio data. The features of the visual image data include the pixel coordinates of each facial feature point and the facial expression intensity features at each acquisition time point, etc. The features of the depth image data include the depth value of each facial feature point and the average depth value of each facial feature area. The features of the infrared image data are the average temperature values ​​of each facial area, and the features of the audio data are the speaking speed features corresponding to each acquisition time point.

[0053] In a specific embodiment, the facial feature points include feature points such as eyes, nose, and mouth, and specific facial regions include forehead, cheeks, chin, and other regions.

[0054] The modality fusion confirmation module is used to confirm the multimodal data fusion type based on the characteristics and acquisition characteristics of each modality data.

[0055] Specifically, see Figure 3 As shown, confirm the multimodal data fusion type, including: K1, extract and set the sampling interval duration from the acquisition features of each modal data, count the acquisition synchronization of the modal data, and at the same time, count the correlation of the modal data based on the extracted features of each modal data.

[0056] K2. Define that the modal data acquisition synchronization is greater than the set reference first modal data acquisition synchronization and the modal data correlation is greater than the set first reference modal data correlation as fusion evaluation condition 1.

[0057] K3. Define that the modal data acquisition synchronization degree is less than the set reference second modal data acquisition synchronization degree and the modal data correlation degree is less than the set second reference modal data correlation degree as fusion evaluation condition 2.

[0058] K4. Traverse fusion assessment condition 1 and fusion assessment condition 2.

[0059] K5. If fusion assessment condition 1 is met, early fusion is used as the multimodal data fusion type. If fusion assessment condition 2 is met, late fusion is used as the multimodal data fusion type. If both fusion assessment condition 1 and fusion assessment condition 2 are not met, mixed fusion is used as the multimodal data fusion type.

[0060] The embodiment of the present invention determines the type of multimodal data fusion by combining the characteristics and acquisition characteristics of each modal data, effectively solving the problem that the current modal fusion is not targeted, fully considering and alleviating the impact of data asynchrony caused by the sampling frequencies of various sensors, ensuring the convenience of subsequent multimodal data fusion, and reducing the error of the final facial dynamic feature multimodal recognition and analysis results, further improving the accuracy of the subsequent final facial dynamic feature multimodal recognition and analysis results.

[0061] It should be added that if the data of different modalities are highly synchronized in acquisition time, and the correlation between the modalities is very close, there are obvious complementary features between each other, such as visual image data and depth image data of the face, the depth information can well supplement the details of the three-dimensional structure of the face in the visual image, and the combination of the two can more accurately analyze the dynamic features of the face such as movements and expressions. In this case, early fusion may be a more appropriate choice. Early fusion is to merge the data of different modalities according to certain rules before feature extraction, such as splicing the data of different modalities into a longer data vector in sequence or assigning different weights according to factors such as the importance of each modality before splicing, and then uniformly extracting features from the fused data.

[0062] Understandably, if the synchronization of data of different modalities is not ideal during collection, or the features reflected by each modal data are relatively independent, such as facial visual image data and collected audio data, but each modality has its own unique and reliable information value. At this time, late fusion is more applicable. Late fusion is to first perform complete feature extraction, recognition analysis and other independent processing processes on the data of each modality. Each modality first forms its own recognition result, and then in the final decision-making stage, the results of these different modalities are fused according to a specific method to obtain the final comprehensive judgment.

[0063] It is also understandable that the above-mentioned specific method is specifically a voting method, a weighted average method, etc. in a specific embodiment, wherein the specific execution process of the voting method and the weighted average method is not the focus of the present invention and is a relatively mature fusion method, which will not be elaborated here.

[0064] Regarding the acquisition synchronization of statistical modal data in step K1, it includes: K11, calculating the standard deviation of the set sampling interval of each modal data, and taking the calculation result as the modal data acquisition duration difference, recorded as

[0065] K12. Perform difference calculation on the set sampling interval duration of each modal data to obtain the set sampling interval duration difference between each modal data, and compare it with the set reference sampling interval duration difference interval.

[0066] K13. If the set sampling interval time difference between a certain modal data and another modal data is within the set sampling interval time difference range, the other modal data is used as the same-frequency data of the modal data.

[0067] K14. Count the number of same-frequency data of each modal data, and select the maximum number of same-frequency data, and divide it by the total number of modal data to obtain the modal data same-frequency ratio, denoted as k a .

[0068] K15, TB, the acquisition synchronization degree of statistical modal data, To set the reference modal data acquisition time difference, a0 and a1 represent the weights corresponding to the modal data acquisition time difference and the modal data frequency ratio, respectively, e is a natural constant, and a0+a1=1.

[0069] In a specific embodiment, The specific value can be 3 milliseconds. The value interval is [0,1]. By setting the value interval to between 0 and 1, it is convenient to ensure that the overall value of the acquisition synchronization of the modal data is between 0 and 1, so as to facilitate subsequent further analysis. The reference sampling interval time difference interval can be set to [-2 milliseconds, 2 milliseconds].

[0070] It should be added that the difference in acquisition time is calculated by calculating the standard deviation of the sampling interval time of each modal data, which reflects the degree of discreteness of the acquisition time interval of each modal data as a whole. It is given a certain weight because in the process of multimodal data acquisition, if the time interval of each modal acquisition fluctuates greatly, that is, the standard deviation is large, then it means that the stability of the data on the time axis is poor. When performing subsequent feature fusion and other processing, it is difficult to accurately match different modal data at the same time or similar time, which will seriously affect the accuracy of multimodal recognition. For example, the acquisition interval of visual image data is sometimes long and sometimes short, and it is difficult to accurately align with depth images, audio data, etc. in time, which makes it easy for feature mismatches to occur.

[0071] The modal data co-frequency ratio is obtained by counting the number of co-frequency data of each modal data, screening out the maximum number of co-frequency data, and then dividing it by the total number of modal data. It intuitively reflects the degree of matching and co-frequency between the modal data in terms of the sampling interval. It is given a weight because when multimodal data is collected for subsequent recognition and analysis, if a large amount of modal data can become co-frequency data with each other, it means that these data are easier to align and coordinate in time, which is of great significance for the precise fusion of features of different modalities, such as the fusion of visual image features and audio features at the same time. For example, in the recognition of dynamic facial features, if the visual image and audio data can be in the same frequency state more often, it will be more accurate when analyzing the coordination of expressions and voices, reducing the recognition errors caused by time dislocation.

[0072] Understandably, the performance and working principles of data acquisition devices of different modalities are different, which may cause fluctuations in the sampling interval, and external environmental factors such as power supply stability and temperature will also affect the sampling interval of the device. By assigning corresponding weights to the difference in acquisition duration, the degree of influence of the time inconsistency caused by equipment and environmental factors on the overall acquisition synchronization can be quantified, so as to reasonably evaluate the synchronization quality of multimodal data acquisition. At the same time, by considering the frequency co-ratio of modal data, the importance of this data synergy advantage can be highlighted in the comprehensive evaluation of acquisition synchronization, so that the measurement of synchronization is more in line with the needs of effective use of multimodal data in practical applications. And in summary, a0 and a1 can be taken as 0.4 and 0.6 respectively.

[0073] Regarding the correlation of statistical modal data in step K1, it includes: step 1, if the current modal data combination is mode I, extract the features of the visual image data from the features of each modal data, then extract the pixel coordinates of each facial feature point, and construct the pixel vector of each facial feature point.

[0074] Step 2: Extract the features of the depth image data from the features of each modal data, and then extract the depth value of each facial feature point, construct the depth vector of each facial feature point, and obtain the correlation between the visual image modality and the depth image modality through canonical correlation analysis based on the pixel vector and depth vector of each facial feature point.

[0075] Step 3: Select the facial expression intensity features at each acquisition time point from the features of the visual image data to form a facial expression intensity feature sequence M.

[0076] Step 4: Extract the features of the audio data from the features of each modal data, and then select the speech rate features corresponding to each acquisition time point to form a speech rate feature sequence V. Import M and V into the Pearson correlation coefficient, and use the output result as the correlation between the visual image modality and the audio modality.

[0077] Step 5: Set the weight coefficients of the correlation between the visual image modality and the depth image modality and the correlation between the visual image modality and the audio modality, and obtain the correlation of the modal data by weighted average calculation.

[0078] Step 6: If the current modal data combination is mode II, calculate the correlation of the image modal data, set the weight coefficients of the correlation of the image modal data and the correlation of the visual image modality and the audio modality, and obtain the correlation of the modal data by weighted average calculation.

[0079] It should be added that the weight coefficient of the correlation degree of the image modality data is the same as the weight coefficient of the correlation degree of the visual image modality and the depth image modality.

[0080] In a specific embodiment, the example process of step 2 is: suppose we extract the coordinates of key facial feature points from the visual image to form pixel vectors, such as the two-dimensional coordinates of feature points such as eyes, nose, mouth, etc., and form the pixel vectors of each facial feature point into a matrix X. The depth value of the corresponding position of each facial feature point is extracted from the depth image to form a depth vector, and the depth vectors of each facial feature point are further formed into a matrix Y. Use canonical correlation analysis to analyze the correlation between them, first standardize X and Y, and then calculate the covariance matrix and other related steps to obtain a high correlation coefficient, and use the obtained correlation coefficient as the correlation between the visual image modality and the depth image modality.

[0081] In a specific embodiment, the specific extraction process of facial expression intensity features is exemplified as follows: 1) using computer vision technology to accurately locate each key feature point of the face as each facial feature point.

[0082] Understandably, common methods include active shape models, active appearance models, and deep learning-based methods such as convolutional neural networks, CNNs, etc. For example, a CNN model can be used to detect multiple feature points around key parts such as eyes, eyebrows, noses, and mouths in a face image.

[0083] 2) Calculate the coordinate displacement of each facial feature point under different expression states relative to the neutral expression, that is, the reference state when there is no obvious expression, including the horizontal displacement and the vertical displacement.

[0084] Understandably, the horizontal displacement and vertical displacement can be used to preliminarily measure the intensity of expression. Taking the corners of the mouth as an example, assuming that the coordinates of the two feature points of the corners of the mouth are (x1, y1) and (x2, y2) in a neutral expression, and become (x1′, y1′) and (x′2, y′2) in a smiling expression, then the horizontal displacement can be calculated by The vertical displacement is calculated in the same way.

[0085] 3) Multiply the horizontal displacement by the vertical displacement and use the result as the facial expression intensity feature.

[0086] Understandably, for different expressions, specific facial feature points will undergo corresponding displacement changes. For example, when smiling, the feature points at the corners of the mouth will move to the sides and upwards, and the feature points at the cheeks will shift to a certain extent as the muscles stretch. When frowning, the feature points of the eyebrows will move closer to each other and move downward, etc. The comprehensive displacement size can reflect the intensity of the smiling expression to a certain extent. The larger the displacement, the stronger the expression.

[0087] In a specific embodiment, it is necessary to supplement step 6 that in human-computer interaction applications, such as facial expression recognition of smart devices for personalized interactive feedback, if the visual image is highly correlated with the depth image, and can better capture subtle facial expressions and spatial changes, the device can more accurately understand the user's intentions and emotions, for example, according to the accuracy of the user's smile, frown and other expression changes, to provide a service response that is more in line with the user's needs, thereby improving the naturalness and accuracy of the interactive experience. At the same time, when the interaction process needs to consider the user's voice instructions and the emotional state conveyed by the voice and the facial expression, for example, the intelligent voice assistant analyzes the user's speech intonation, tone and corresponding facial expressions, and more comprehensively judges whether the user is asking questions happily or urging impatiently, thereby adjusting the tone and content of the reply, and achieving a more intelligent and humanized interactive effect. In summary, specifically, the weight coefficients of the correlation between the visual image modality and the depth image modality and the correlation between the visual image modality and the audio modality can be 0.55 and 0.45 respectively.

[0088] Furthermore, regarding the calculation of the correlation degree of the image modality data in step 6, it includes: extracting the depth mean of each facial region from the features of the depth image data to form a regional depth feature vector.

[0089] The features of the infrared image data are extracted from the features of each modality data, and then the average temperature value of each facial area is extracted to form a regional temperature feature vector. The correlation between the depth image modality and the infrared image modality is calculated based on the Pearson correlation coefficient based on the regional depth feature vector and the regional temperature feature vector.

[0090] The correlation between the visual image modality and the depth image modality and the correlation between the depth image modality and the infrared image modality are averaged and the calculation result is used as the correlation of the image modality data.

[0091] It should be added that when the face moves, not only can the depth image record the spatial displacement and morphological changes of various parts of the face, but the infrared image will also reflect the heat changes generated by muscle movement. For example, when frowning, the depth image shows the contraction of the brow muscles in three dimensions, and the infrared image can capture the temperature increase in the brow area due to muscle activity. The two together provide a basis for analyzing the physiological effects and energy changes of facial movements, and can reflect the synergistic relationship of facial dynamic features from different physical dimensions.

[0092] The feature fusion processing module is used to set the feature correction factor corresponding to each modal data, and import the facial feature recognition task, and then analyze the features of each modal data, and output the facial feature recognition result of the target user.

[0093] Specifically, the feature correction factor corresponding to each modal data is set, including: W1, if the current modal data combination is mode I, the visual image data is extracted from the collected modal data, and then the clarity of each collected visual image is extracted, and the feature correction factor corresponding to the visual image data is set, which is recorded as ε F .

[0094] The setting of the feature correction factor corresponding to the visual image data includes: calculating the average clarity of each collected visual image to obtain an average clarity, which is recorded as ζ.

[0095] Set the feature correction factor ε corresponding to the visual image data F , ζ′ is the image definition used as the reference.

[0096] W2, set the feature correction factor corresponding to the depth image data, denoted as ε h ,

[0097]

[0098] W3. Extract the signal-to-noise ratio of each acquisition time point from the collected environmental data, calculate the average signal-to-noise ratio by the mean, denoted as Q, and set the feature correction factor corresponding to the audio data, denoted as Q' is the interference signal-to-noise ratio of the setting reference.

[0099] W4. If the current modal data combination is mode II, repeat steps W1 to W3, and extract the temperature of each acquisition time point from the collected environmental information, and obtain the average temperature by mean calculation, recorded as T, and set the characteristic correction factor ε corresponding to the infrared image data r , T′ and T″ are respectively the high temperature value and low temperature value of infrared acquisition affecting the reference.

[0100] What needs to be added about step W1 is that the clarity of the visual image can be directly extracted through image processing tools, which is a relatively mature technical means available. The specific extraction process and extraction principle will not be repeated here.

[0101] What needs to be added about step W1 is that It is set by referring to the activation function, the purpose is to limit the value interval of the clarity analysis results to (0,1), so as to ensure the consistency of the value interval of the subsequent analysis data. Taking the minimum value also ensures the consistency of the corresponding value range of the characteristic correction factor, thereby ensuring the validity of the subsequent analysis results. In a specific embodiment, the clarity is measured by the echelon mean. When the gradient mean is approximately lower than 50, the imaging quality has been disturbed. Specifically, ζ′ takes a value of 50.

[0102] What needs to be added about step W3 is that the specific value of Q′ can be 20dB.

[0103] It should be added about step W4 that T′ and T″ can be 35° C. and 15° C. respectively.

[0104] In another specific embodiment, the features of each modal data are analyzed, including: based on the feature correction factors corresponding to each modal data, the modal data are sorted from large to small according to their feature correction factors as the feature fusion order of each modal data.

[0105] Based on the multimodal data fusion type and the feature fusion order of each modal data, fusion is performed to obtain fusion features.

[0106] According to the fusion features and the facial feature recognition task, the facial feature recognition result is obtained by recognizing a recognition model corresponding to the facial feature recognition task which is preset.

[0107] It should be added that fusion means to sequentially stitch together the features of each modality after being filtered according to the correction factor into a longer feature vector as a fusion feature. For example, the key feature vectors of the visual image, the key feature vectors of the depth image, the key feature vectors of the infrared image, and the key feature vectors of the audio modality are sequentially stitched together to form a comprehensive fusion feature vector, and then the classification, regression and other models are used to perform facial feature recognition analysis based on the fusion feature vector.

[0108] In a specific embodiment, each pre-set facial feature recognition task can use a traditional machine learning model, such as a support vector machine, a decision tree, a naive Bayes, etc. to perform classification tasks. It is also possible to use a deep learning model, such as a convolutional neural network, a recurrent neural network and its variants such as a long short-term memory network, LSTM, etc., by constructing a corresponding network structure and using a large amount of labeled facial feature data for training, so that it can learn the mapping relationship from the fused feature vector to the facial feature recognition result, and then obtain the recognition model of each facial feature recognition task.

[0109] Exemplarily, in the recognition model corresponding to the facial feature recognition task, the model will output the corresponding facial feature recognition results. For example, in the face recognition task, the identity information of the target user, such as the specific name, number, etc., is output; in the expression recognition task, the expression category, such as happy, sad, angry, etc., or the expression intensity value is output. In the emotion judgment task, the emotional state, such as positive, negative, neutral, etc., and the corresponding confidence score, etc., are output, thereby completing the facial feature recognition of the target user.

[0110] It should be noted that the entire process requires continuous experimentation, verification, and adjustment of correction factors and model parameters to optimize the accuracy and reliability of facial feature recognition so that it can adapt to different application scenarios and data characteristics. For example, the degree of smile is represented by a value between 0 and 1.

[0111] The embodiments of the present invention set the feature correction factor corresponding to each modal data based on the collection characteristics and environmental information of each modal data, thereby making up for the current deficiency that different modal analysis results are not corrected according to the specific environmental influences, reducing the interference of environmental factors such as lighting and background noise, ensuring the recognition quality of special facial data and the accuracy of feature extraction, and improving the reliability and effectiveness of subsequent recognition results.

[0112] The recognition feedback display terminal is used to feed back the facial feature recognition result of the target user to the user in a visual manner.

[0113] The above contents are merely examples and explanations of the concept of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, they shall all fall within the protection scope of the present invention.

Claims

1. A facial dynamic feature multimodal recognition and analysis system based on artificial intelligence, characterized in that: include: The environmental data collection module is used to collect environmental data in the area where the user is located to obtain collected environmental information; The modal data collection module is used to collect multimodal data from users, obtain the collected modal data, and record the collection characteristics of each modal data; A modal feature extraction module is used to preprocess each modal data and extract the features of each modal data; A modal fusion confirmation module is used to confirm the type of multimodal data fusion based on the characteristics and acquisition characteristics of each modal data; The feature fusion processing module is used to set the feature correction factor corresponding to each modal data, import the facial feature recognition task, and then analyze the features of each modal data to output the facial feature recognition result of the target user; The recognition feedback display terminal is used to provide the target user's facial feature recognition results to the user in a visual manner.

2. The facial dynamic feature multimodal recognition and analysis system based on artificial intelligence as claimed in claim 1, characterized in that: Before the multimodal data collection is performed, the modal data combination mode is confirmed, and the specific confirmation process includes: Extract the collected light intensity of each collection point at each collection time point from the collected environmental information, calculate the standard deviation of the collected light intensity at different collection points at the same collection time point, and use the standard deviation calculation result as the light intensity difference; The maximum light intensity difference is selected from the light intensity differences at each acquisition time point and recorded as With the acquisition time point as the horizontal coordinate and the acquisition light intensity as the vertical coordinate, the light intensity change curve of each acquisition point is constructed, its slope is calculated as the light intensity change rate, and the minimum value is selected and recorded as p E ; The average light intensity of different collection points at the same collection time point is calculated, and the minimum value is selected from the average light intensity of each collection point, which is recorded as E min ; Statistical light environment interference, denoted as λ E , E′ and are the image acquisition interference light intensity and image acquisition interference light intensity difference of the set reference, respectively, and p′ is the light intensity change rate of the set reference; The combination of visual image data, depth image data and audio data is called mode I, and the combination of visual image data, depth image data, infrared image data and audio data is called mode II; If E ≤0, take mode I as the modal data combination mode, if λ E >0, mode II is used as the modal data combination mode.

3. The facial dynamic feature multimodal recognition and analysis system based on artificial intelligence as claimed in claim 2, characterized in that: The multimodal data collection comprises: If the current modal data combination mode is mode I; Use a high-definition camera to continuously obtain visual image sequences of the user's face at different angles with a set sampling interval; Using the depth camera to continuously obtain a sequence of depth images of the user's face at different angles with a set sampling interval; Use microphones to collect user audio data in real time to form a user audio data set; If the current modal data combination mode is mode II, the multimodal data acquisition steps of mode I are repeated, and an infrared camera is used to continuously acquire infrared image sequences of the user's face at different angles at a set frame rate.

4. The facial dynamic feature multimodal recognition and analysis system based on artificial intelligence as claimed in claim 2, characterized in that: The confirming multimodal data fusion type includes: Extract the set sampling interval duration from the acquisition features of each modal data, count the acquisition synchronization of the modal data, and count the correlation of the modal data based on the extracted features of each modal data; Define that the acquisition synchronization degree of modal data is greater than the acquisition synchronization degree of the set reference first modal data and the correlation degree of modal data is greater than the correlation degree of the set first reference modal data as fusion evaluation condition 1; Define that the modal data acquisition synchronization degree is less than the set reference second modal data acquisition synchronization degree and the modal data correlation degree is less than the set second reference modal data correlation degree as fusion evaluation condition 2; Traverse fusion assessment condition 1 and fusion assessment condition 2; If fusion assessment condition 1 is met, early fusion is used as the multimodal data fusion type. If fusion assessment condition 2 is met, late fusion is used as the multimodal data fusion type. If both fusion assessment condition 1 and fusion assessment condition 2 are not met, mixed fusion is used as the multimodal data fusion type.

5. The facial dynamic feature multimodal recognition and analysis system based on artificial intelligence as claimed in claim 4, characterized in that: The acquisition synchronization of the statistical modal data includes: The standard deviation of the set sampling interval of each modal data is calculated, and the calculation result is used as the difference of the modal data acquisition time length, recorded as The difference between the set sampling intervals of each modal data is calculated to obtain the set sampling interval difference between each modal data, and compared with the set reference sampling interval difference interval; If the set sampling interval time difference between a certain modal data and another modal data is within the set sampling interval time difference range, the other modal data is used as the same-frequency data of the modal data; Count the number of same-frequency data of each modal data, and select the maximum number of same-frequency data from them, and divide it by the total number of modal data to obtain the modal data same-frequency ratio, denoted as k a ; The acquisition synchronization degree of statistical modal data is TB, To set the reference modal data acquisition time difference, a0 and a1 represent the weights corresponding to the modal data acquisition time difference and the modal data frequency ratio, respectively, e is a natural constant, and a0+a1=1.

6. The facial dynamic feature multimodal recognition and analysis system based on artificial intelligence as claimed in claim 4, characterized in that: The correlation degree of the statistical modal data includes: If the current modality data combination is mode I, the features of the visual image data are extracted from the features of each modality data, and then the pixel coordinates of each facial feature point are extracted, and the pixel vector of each facial feature point is constructed; Extracting the features of the depth image data from the features of each modality data, and then extracting the depth value of each facial feature point, constructing the depth vector of each facial feature point, and obtaining the correlation between the visual image modality and the depth image modality through canonical correlation analysis based on the pixel vector and depth vector of each facial feature point; Select facial expression intensity features at each acquisition time point from the features of the visual image data to form a facial expression intensity feature sequence M; Extract the features of audio data from the features of each modal data, and then select the speech rate features corresponding to each acquisition time point to form a speech rate feature sequence V. Import M and V into the Pearson correlation coefficient, and use the output result as the correlation between the visual image modality and the audio modality. The weight coefficients of the correlation between the visual image modality and the depth image modality and the correlation between the visual image modality and the audio modality are set, and the correlation of the modality data is obtained by weighted average calculation; If the current modal data combination is mode II, the correlation of the image modal data is calculated, the weight coefficients of the correlation of the image modal data and the correlation of the visual image modality and the audio modality are set, and the correlation of the modal data is obtained by weighted average calculation.

7. The facial dynamic feature multimodal recognition and analysis system based on artificial intelligence as claimed in claim 6, characterized in that: The calculating the correlation degree of the image modality data comprises: Extract the depth mean of each facial region from the features of the depth image data to form a regional depth feature vector; Extract the features of infrared image data from the features of each modality data, and then extract the average temperature value of each facial region to form a regional temperature feature vector. Based on the regional depth feature vector and the regional temperature feature vector, the correlation between the depth image modality and the infrared image modality is calculated by the Pearson correlation coefficient. The correlation between the visual image modality and the depth image modality and the correlation between the depth image modality and the infrared image modality are averaged and the calculation result is used as the correlation of the image modality data.

8. The facial dynamic feature multimodal recognition and analysis system based on artificial intelligence as claimed in claim 2, characterized in that: The step of setting the characteristic correction factor corresponding to each modal data includes: W1. If the current modal data combination is mode I, extract the visual image data from each collected modal data, and then extract the clarity of each collected visual image, and set the feature correction factor corresponding to the visual image data, denoted as ε F ; W2, set the feature correction factor corresponding to the depth image data, denoted as ε h , W3. Extract the signal-to-noise ratio of each acquisition time point from the collected environmental data, calculate the average signal-to-noise ratio by the mean, denoted as Q, and set the feature correction factor corresponding to the audio data, denoted as ε u , Q′ is the interference signal-to-noise ratio of the setting reference; W4. If the current modal data combination is mode II, repeat steps W1 to W3, and extract the temperature of each acquisition time point from the collected environmental information, and obtain the average temperature by mean calculation, recorded as T, and set the characteristic correction factor ε corresponding to the infrared image data r , T′ and T″ are respectively the high temperature value and low temperature value of infrared acquisition affecting the reference.

9. The facial dynamic feature multimodal recognition and analysis system based on artificial intelligence as claimed in claim 8, characterized in that: The step of setting the feature correction factor corresponding to the visual image data includes: The clarity of each collected visual image is averaged to obtain the average clarity, which is recorded as ζ; Set the feature correction factor ε corresponding to the visual image data F , ζ′ is the image definition used as the reference.

10. The facial dynamic feature multimodal recognition and analysis system based on artificial intelligence as claimed in claim 1, characterized in that: The analysis of the characteristics of each modal data includes: Based on the feature correction factors corresponding to each modal data, the modal data are sorted from large to small according to their feature correction factors as the feature fusion order of each modal data; Based on the multimodal data fusion type and the feature fusion order of each modal data, fusion is performed to obtain fusion features; According to the fusion features and the facial feature recognition task, the facial feature recognition result is obtained by recognizing a recognition model corresponding to the facial feature recognition task which is preset.

Images