A method and device for detecting subtle changes in human face

Abstract

The present invention relates to the field of image detection technology, and discloses a method and device for detecting subtle changes in human faces. The method uses a neural network to perform face recognition and crop videos on original human body videos, retaining only the face area to form a first face video; uses an image processing algorithm and a signal processing algorithm to extract feature signals in the first face video, amplifies subtle changes in the video by applying an amplification factor, and reconstructs the video to form a second face video; extracts red channel data, green channel data, and blue channel data from the second face video; calculates the subtle change frequency of the original human body video by analyzing the data of each color channel. The present invention reduces the interference of irrelevant background pixels and the amount of calculation by cropping the face video, improves the convenience of observing the face video by amplifying the feature information, and realizes the observation of subtle changes in the face.

Description

Technical Field

[0001] The present invention relates to the technical field of image detection, and in particular to a method and device for detecting subtle changes in a human face. Background Art

[0002] Blood spectrum imaging technology is a contactless physiological signal monitoring technology. Due to the translucent nature of human skin, when visible light shines on the skin, the hemoglobin content of the subcutaneous blood vessels will change, causing the intensity of light reflected by the skin to vary. Therefore, the optical image of the skin contains information about the subcutaneous blood vessels. By using a camera to shoot a video of a person's face, real-time recording and analysis of changes in the skin of the face, and then accurately obtaining data on changes in the subcutaneous blood vessels of the face, subtle changes in the face can be observed. This technology can be applied to scientific research and practical application fields such as medicine, psychology, and marketing.

[0003] The current existing technology uses a camera to capture facial features of a human body, which can only see facial features under visible light reaction, and cannot observe the change data of subcutaneous blood vessels, and cannot observe subtle changes in the human face. Summary of the invention

[0004] The present invention provides a method and device for detecting subtle changes in a human face, which reduces the interference of irrelevant background pixels and the amount of calculation, improves the convenience of observing a human face video by amplifying feature information, and realizes the observation of subtle changes in the human face.

[0005] In order to solve the above technical problems, the present invention provides a method for detecting subtle changes in a face, comprising:

[0006] Using a preset face recognition neural network algorithm to perform face recognition and cropping on the original human body video to obtain a first face video;

[0007] Extracting video feature information from the first face video using a feature extraction algorithm consisting of a number of downsampling operations;

[0008] amplifying the video feature information, and combining the amplified video feature information with the first face video to form a second face video;

[0009] Extracting image data of each frame from the second face video, and respectively extracting red channel data, green channel data, and blue channel data from the image data;

[0010] Analyze the red channel data, the green channel data and the blue channel data to obtain red channel pixel value data, green channel pixel value data and blue channel pixel data under each color channel respectively;

[0011] The subtle change frequency of the original human body video is calculated using the red channel pixel value data, the green channel pixel value data and the blue channel pixel data.

[0012] Further, the method of using a preset face recognition neural network algorithm to perform face recognition and cropping on the original human body video to obtain the first face video is specifically as follows:

[0013] Using a preset constructor to read the original human body video, obtain the original frame rate information of the original human body video and the image data of each frame;

[0014] Using a preset face recognition neural network algorithm to perform face recognition on the first frame image of the original human body video, and generate the coordinate point information of the box where the face of the first frame image is located;

[0015] Cropping each frame of the original human body video according to the coordinate point information of the box within the range where the face of the first frame of the image is located;

[0016] According to the original frame rate information, each frame image after cropping the original human body video is saved using a preset saving function to generate a first face video.

[0017] The present invention performs face recognition and cropping on the original human body video through a preset face recognition neural network algorithm, retaining only pixels in the face area to reduce interference from other irrelevant background pixels and reduce the amount of calculation.

[0018] Furthermore, the extracting of video feature information from the first face video using a feature extraction algorithm composed of several downsampling operations is specifically as follows:

[0019] Read each frame of the first face video, and convert the format of each frame of the first face video into a YIQ format;

[0020] Using a feature extraction algorithm composed of several downsampling operations, video feature information of different scales is extracted from each frame image of the first face video converted into the YIQ format; wherein the first face video reduces the image size while undergoing the downsampling operation.

[0021] Further, the signal amplification of the video feature information and merging the signal-amplified video feature information with the first face video to form the second face video is specifically as follows:

[0022] Using a preset interpolation function to perform a zero-order interpolation operation on the first face video processed by the feature extraction algorithm to restore the size of the first face video;

[0023] The video feature information is converted into a frequency domain signal by Fourier transform, and the signal is filtered by a preset bandpass filter;

[0024] Convert the video feature information after signal filtering back to spatial domain signal;

[0025] Multiplying the spatial domain signal by a preset amplification factor to amplify the video feature information;

[0026] The amplified video feature information is merged with the first face video in the YIQ format, and the merged data is saved using a preset saving function according to the original frame rate information to generate a second face video.

[0027] The present invention performs Fourier transform on the pixel information of face video to convert its spatial domain signal into frequency domain signal, uses a bandpass filter to select the color signal of a specific frequency, and then performs inverse Fourier operation to convert the filtered frequency domain signal back into a spatial domain signal, and applies a specific amplification factor to the feature signal with weak changes, so as to facilitate more intuitive observation of the changes.

[0028] Further, the red channel data, the green channel data and the blue channel data are analyzed to obtain red channel pixel value data, green channel pixel value data and blue channel pixel data under each color channel, respectively, as follows:

[0029] Traversing all pixel points of the second face video; wherein each pixel point has the same position in each frame of the second face video;

[0030] Determine the amplitude values ​​of all pixels of the second face video in the red channel, the green channel and the blue channel respectively according to the red channel data, the green channel data and the blue channel data;

[0031] Determine the maximum amplitude point corresponding to the maximum amplitude value under the red channel, green channel and blue channel, and obtain the pixel value data of the maximum amplitude point under each color channel in the corresponding red channel data, green channel data and blue channel data to form red channel pixel value data, green channel pixel value data and blue channel pixel data.

[0032] Furthermore, the calculating the subtle change frequency of the original human body video by using the red channel pixel value data, the green channel pixel value data and the blue channel pixel data further includes:

[0033] The extreme value method is used to calculate the subtle change frequency of the original human body video. The extreme value method is specifically as follows:

[0034] Performing polynomial fitting on the red channel pixel value data, the green channel pixel value data and the blue channel pixel data respectively;

[0035] Derivatives are respectively performed on the fitted red channel pixel value data, green channel pixel value data and blue channel pixel data;

[0036] Obtaining maximum and minimum points in the fitting curves corresponding to the derived red channel pixel value data, green channel pixel value data, and blue channel pixel value data, respectively;

[0037] The average distance between the maximum value points under each color channel and the average distance between the minimum value points under each color channel are determined as the average change frequency of each color channel;

[0038] The average change frequency of each color channel is averaged to obtain the subtle change frequency of the original human body video.

[0039] Furthermore, the calculating the subtle change frequency of the original human body video by using the red channel pixel value data, the green channel pixel value data and the blue channel pixel data further includes:

[0040] The array method is used to calculate the frequency of subtle changes in the original human body video, and the array method is specifically as follows:

[0041] Performing Fourier transform on the red channel pixel value data, the green channel pixel value data and the blue channel pixel data respectively;

[0042] Obtaining frequency arrays of each color channel from the red channel pixel value data, the green channel pixel value data, and the blue channel pixel data after Fourier transformation respectively;

[0043] Get the absolute value of the frequency array of each color channel and sort them according to the frequency;

[0044] Calculate the average value of several maximum frequencies of the frequency array of each color channel to determine the average change frequency of each color channel;

[0045] The average change frequency of each color channel is averaged to obtain the subtle change frequency of the original human body video.

[0046] Furthermore, after extracting the image data of each frame from the second face video, the method further includes:

[0047] For a first pixel point of the second facial video, respectively extract a red pixel value, a green pixel value, and a blue pixel value at a position where the first pixel point is located in image data of each frame of the second facial video;

[0048] The red pixel value, green pixel value and blue pixel value of the first pixel are used to determine a red channel pixel value change curve graph, a green channel pixel value change curve graph and a blue channel pixel value change curve graph of the first pixel respectively.

[0049] Furthermore, the formation process of the original human body video is specifically as follows:

[0050] Under dark room conditions, low-frequency strobe lighting is used for fill lighting, and a high-sampling rate camera is used to record the human body; wherein the human body remains still during video recording.

[0051] The invention provides a method for detecting subtle changes in human faces. A neural network is used to perform human face recognition and video cropping on an original human body video, and only a human face area is retained to form a first human face video, so as to reduce the interference of other irrelevant background pixels and reduce the amount of calculation; an image processing algorithm and a signal processing algorithm are used to extract a characteristic signal in the first human face video, and subtle changes in the video are amplified by applying an amplification coefficient so that the subtle signal can be more easily observed, and the video is reconstructed to form a second human face video; red channel data, green channel data and blue channel data are respectively extracted from the second human face video; and the subtle change frequency of the original human body video is calculated by analyzing the data of each color channel, so as to realize the observation of subtle changes in the human face.

[0052] Accordingly, the present invention provides a device for detecting subtle changes in a face, comprising: a connection module, an upgrade package generation module, a sending module and a verification module;

[0053] The connection module is used for the identification module, the feature extraction module, the merging module, the channel data acquisition module, the pixel value data acquisition module and the frequency calculation module;

[0054] The recognition module is used to perform face recognition and cropping on the original human body video using a preset face recognition neural network algorithm to obtain a first face video;

[0055] The feature extraction module is used to extract video feature information from the first face video using a feature extraction algorithm composed of several downsampling operations;

[0056] The merging module is used to amplify the signal of the video feature information, and merge the amplified video feature information with the first face video to form a second face video;

[0057] The channel data acquisition module is used to extract image data of each frame in the second face video, and respectively extract red channel data, green channel data and blue channel data from the image data;

[0058] The pixel value data acquisition module is used to analyze the red channel data, the green channel data and the blue channel data to obtain the red channel pixel value data, the green channel pixel value data and the blue channel pixel data under each color channel respectively;

[0059] The frequency calculation module is used to calculate the subtle change frequency of the original human body video by using the red channel pixel value data, the green channel pixel value data and the blue channel pixel data.

[0060] The invention provides a device for detecting subtle changes in human face. The device is based on the organic combination of modules and uses a neural network to perform human face recognition and crop videos on original human body videos, retaining only the human face area to form a first human face video, so as to reduce the interference of other irrelevant background pixels and reduce the amount of calculation; an image processing algorithm and a signal processing algorithm are used to extract characteristic signals in the first human face video, subtle changes in the video are amplified by applying an amplification coefficient so as to more easily observe the subtle signals, and the video is reconstructed to form a second human face video; red channel data, green channel data and blue channel data are respectively extracted from the second human face video; the subtle change frequency of the original human body video is calculated by analyzing the data of each color channel, so as to realize the observation of subtle changes in the human face. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for detecting subtle changes in a face provided by the present invention;

[0062] Figure 2 A schematic diagram of a flow chart of an embodiment of a face recognition method provided by the present invention;

[0063] Figure 3 A pixel value variation curve diagram provided by the present invention;

[0064] Figure 4 Another channel pixel value variation curve diagram provided by the present invention;

[0065] Figure 5 A schematic structural diagram of an embodiment of a device for detecting subtle changes in a face provided by the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the multimedia resources involved in this disclosure are all obtained with full authorization.

[0068] Example 1

[0069] See also Figure 1 , is a flow chart of an embodiment of a method for detecting subtle changes in a face provided by the present invention, the method comprising steps 101 to 106, each of which is specifically as follows:

[0070] Step 101: Use a preset face recognition neural network algorithm to perform face recognition and cropping on an original human body video to obtain a first face video.

[0071] Furthermore, in the first embodiment of the present invention, the process of forming the original human body video is specifically as follows:

[0072] Under dark room conditions, low-frequency strobe lighting is used for fill lighting, and a high-sampling rate camera is used to record the human body; wherein the human body remains still during video recording.

[0073] Further, in the first embodiment of the present invention, a preset face recognition neural network algorithm is used to perform face recognition and cropping on the original human body video to obtain a first face video, specifically:

[0074] Using a preset constructor to read the original human body video, obtain the original frame rate information of the original human body video and the image data of each frame;

[0075] Using a preset face recognition neural network algorithm to perform face recognition on the first frame image of the original human body video, and generate the coordinate point information of the box where the face of the first frame image is located;

[0076] Cropping each frame of the original human body video according to the coordinate point information of the box within the range where the face of the first frame of the image is located;

[0077] According to the original frame rate information, each frame image after cropping the original human body video is saved using a preset saving function to generate a first face video.

[0078] In the first embodiment of the present invention, the original human body video of the present invention is recorded in a dark room using a low-frequency strobe light for supplementary lighting and a high-sampling rate camera. During the recording and shooting, the human body is determined to remain still, thereby obtaining a static human body video. Figure 2 , is a flow chart of an embodiment of the face recognition method provided by the present invention. After obtaining the original human body video, the VideoCapture function of the OpenCV library can be selected to read the video file, so as to obtain the frame rate information of the original human body video and the image data of each frame in the video; the pre-trained ResNet-18 convolutional neural network model is loaded to perform face recognition on the first frame image of the original human body video, and generate the box coordinate point information of the range where the face is located; according to the box coordinate point information, each frame image of the original human body video is cropped, and only the face area pixels within the box are retained to reduce the interference of other irrelevant background pixels and reduce the amount of calculation; the cropped image is regenerated into a cropped video according to the original frame rate information using the VideoWr iter function of the OpenCV library to generate the first face video.

[0079] Step 102: extracting video feature information from the first face video using a feature extraction algorithm consisting of several downsampling operations.

[0080] Furthermore, in the first embodiment of the present invention, a feature extraction algorithm composed of several downsampling operations is used to extract video feature information from the first face video, specifically:

[0081] Read each frame of the first face video, and convert the format of each frame of the first face video into a YIQ format;

[0082] Using a feature extraction algorithm composed of several downsampling operations, video feature information of different scales is extracted from each frame image of the first face video converted into the YIQ format; wherein the first face video reduces the image size while undergoing the downsampling operation.

[0083] In the first embodiment of the present invention, the first face video is read frame by frame and converted into image data in YIQ format; the image data in YIQ format of each frame is respectively input into a feature extraction algorithm composed of multiple downsampling operations. In each downsampling operation, the image data is sequentially subjected to Gaussian feature extraction operation and image size compression operation to extract feature information of different scales; wherein, each time the downsampling operation is performed, the size of the image data is halved accordingly. In addition, the number of downsampling layers can be selected as 4 layers, 6 layers or 8 layers.

[0084] Step 103: amplify the video feature information, and merge the amplified video feature information with the first face video to form a second face video.

[0085] Furthermore, in the first embodiment of the present invention, the video feature information is amplified, and the amplified video feature information is combined with the first face video to form a second face video, specifically:

[0086] Using a preset interpolation function to perform a zero-order interpolation operation on the first face video processed by the feature extraction algorithm to restore the size of the first face video;

[0087] The video feature information is converted into a frequency domain signal by Fourier transform, and the signal is filtered by a preset bandpass filter;

[0088] Convert the video feature information after signal filtering back to spatial domain signal;

[0089] Multiplying the spatial domain signal by a preset amplification factor to amplify the video feature information;

[0090] The amplified video feature information is merged with the first face video in the YIQ format, and the merged data is saved using a preset saving function according to the original frame rate information to generate a second face video.

[0091] In the first embodiment of the present invention, the insert function of the numpy library can be used to perform zero-order interpolation operation on the first face video after downsampling, and upsampling operation can be performed to restore the size of the image data, thereby completing the preliminary feature information extraction. The numpy.fft function is used to perform Fourier transform on the first face video after the size is restored to obtain its frequency domain signal; a preset bandpass filter is used to select a signal within a specific frequency range, and filters signals of other frequencies in the frequency domain signal. Preferably, the selection range of the bandpass filter can be set to [0.85,1], that is, only the high-frequency band information between 85% and 100% of the total spectrum range of the frequency component after Fourier transform is retained. The numpy.ifft function can be used to perform inverse Fourier transform on the signal data after signal filtering, and the frequency domain signal is converted back to the spatial domain signal. The spatial domain signal is multiplied by a specific amplification factor to amplify the video feature information. Among them, the amplification factor can be set to 100, 200, 300, etc., and different amplification factors will directly affect the change in the result. The video feature information of the amplified signal is merged with the YIQ format image data of the original unprocessed first face video to highlight the changes in the weak facial signal; the merged signal is converted into an RGB data signal, and the numpy.clip function is used to compress the data signal to within the range of [0,255]; based on the original frame rate information of the original human body video, the VideoWriter function of the OpenCV library can be used to merge each frame of compressed RGB data signal into the second face video.

[0092] Step 104: extracting image data of each frame from the second face video, and respectively extracting red channel data, green channel data and blue channel data from the image data.

[0093] In the first embodiment of the present invention, the second video file can be read separately using the VideoCapture function of the OpenCV library, thereby obtaining image data of each frame in the second video file; red channel data, green channel data and blue channel data can be extracted separately from the image data.

[0094] Step 105: Analyze the red channel data, the green channel data, and the blue channel data to obtain red channel pixel value data, green channel pixel value data, and blue channel pixel data under each color channel, respectively.

[0095] Further, in the first embodiment of the present invention, the red channel data, the green channel data and the blue channel data are analyzed to obtain red channel pixel value data, green channel pixel value data and blue channel pixel data under each color channel, respectively, specifically:

[0096] Traversing all pixel points of the second face video; wherein each pixel point has the same position in each frame of the second face video;

[0097] Determine the amplitude values ​​of all pixels of the second face video in the red channel, the green channel and the blue channel respectively according to the red channel data, the green channel data and the blue channel data;

[0098] Determine the maximum amplitude point corresponding to the maximum amplitude value under the red channel, green channel and blue channel, and obtain the pixel value data of the maximum amplitude point under each color channel in the corresponding red channel data, green channel data and blue channel data to form red channel pixel value data, green channel pixel value data and blue channel pixel data.

[0099] In the first embodiment of the present invention, the pixel value variation curve corresponding to the maximum amplitude point of each color channel can be obtained by analyzing the red channel data, the green channel data and the blue channel data, specifically:

[0100] For the red channel data, the pixels at the same position of each frame of the second face video are extracted as the red channel data list, and Fourier transform calculation is performed. The numpy.fft.fftfreq function and the numpy.max function are used to obtain the first amplitude value of the pixel at the position; all the pixels in the second face video are traversed to obtain the amplitude values ​​of the pixels at all positions; the amplitude values ​​corresponding to the pixels at all positions are integrated into a two-dimensional matrix (the two-dimensional matrix has the same size as the image of the second face video), and the numpy.argmax function and the numpy.unrave l_index function are used to find the pixel position corresponding to the maximum amplitude value in the red channel as the feature pixel position under the red channel data; according to the red channel data and the feature pixel position under the red channel data, the pixel value data of the red channel of the feature pixel position is obtained, and it is determined as the red channel pixel value data.

[0101] For the green channel data, the pixels at the same position of each frame of the second face video are extracted as the green channel data list, and Fourier transform calculation is performed. The numpy.fft.fftfreq function and the numpy.max function are used to obtain the first amplitude value of the pixel at the position; all the pixels in the second face video are traversed to obtain the amplitude values ​​of the pixels at all positions; the amplitude values ​​corresponding to the pixels at all positions are integrated into a two-dimensional matrix (the two-dimensional matrix has the same size as the image of the second face video), and the numpy.argmax function and the numpy.unrave l_index function are used to find the pixel position corresponding to the maximum amplitude value in the green channel as the feature pixel position under the green channel data; according to the green channel data and the feature pixel position under the green channel data, the pixel value data of the green channel at the feature pixel position is obtained and determined as the green channel pixel value data.

[0102] For the blue channel data, extract the pixels at the same position in each frame of the second face video as the blue channel data list, perform Fourier transform calculation, and use the numpy.fft.fftfreq function and the numpy.max function to obtain the first amplitude value of the pixel at the position; traverse all the pixels in the second face video to obtain the amplitude values ​​of all the pixels at the position; integrate the amplitude values ​​corresponding to the pixels at all positions into a two-dimensional matrix (the two-dimensional matrix has the same size as the image of the second face video), and use the numpy.argmax function and the numpy.unrave l_index function to find the pixel position corresponding to the maximum amplitude value in the blue channel as the feature pixel position under the blue channel data; according to the blue channel data and the feature pixel position under the blue channel data, obtain the pixel value data of the blue channel at the feature pixel position and determine it as the blue channel pixel value data. See Figure 3 , is a channel pixel value change curve diagram provided by the present invention. Based on the pixel value data of each color channel, the pixel value change curve diagram can be drawn using the matplotlib library.

[0103] Step 106: Calculate the subtle change frequency of the original human body video using the red channel pixel value data, the green channel pixel value data and the blue channel pixel data.

[0104] Furthermore, in the first embodiment of the present invention, the red channel pixel value data, the green channel pixel value data and the blue channel pixel data are used to calculate the subtle change frequency of the original human body video, further comprising:

[0105] The extreme value method is used to calculate the subtle change frequency of the original human body video. The extreme value method is specifically as follows:

[0106] Performing polynomial fitting on the red channel pixel value data, the green channel pixel value data and the blue channel pixel data respectively;

[0107] Derivatives are respectively performed on the fitted red channel pixel value data, green channel pixel value data and blue channel pixel data;

[0108] Obtaining maximum and minimum points in the fitting curves corresponding to the derived red channel pixel value data, green channel pixel value data, and blue channel pixel value data, respectively;

[0109] The average distance between the maximum value points under each color channel and the average distance between the minimum value points under each color channel are determined as the average change frequency of each color channel;

[0110] The average change frequency of each color channel is averaged to obtain the subtle change frequency of the original human body video.

[0111] As an example of the first embodiment of the present invention, the method for calculating the subtle change frequency of the original human body video can adopt the following method: use the numpy.polyfit function to perform polynomial fitting on the red channel pixel value data, the green channel pixel value data and the blue channel pixel data respectively; use the numpy.polyder function to derive the fitted data of different color channels respectively; use numpy.roots to calculate the zero point of the fitting data after the differentiation of different color channels respectively, that is, obtain the extreme point of its fitting curve; use the numpy.polyval function to obtain the numerical value corresponding to each extreme point, and use the numpy.diff function and the numpy.sign function to divide the extreme points into maximum points and minimum points; use the numpy.where function to obtain the coordinates corresponding to the maximum points and the coordinates corresponding to the minimum points of each color channel; use the numpy.diff function and the numpy.mean function to calculate the average distance between the maximum points and the average distance between the minimum points, and the average distance is used as the average change frequency of each color channel; after calculating the average change frequency of each color channel, use the numpy.mean function to calculate the average value of the average change frequency of each color channel to obtain the subtle change frequency in the original human body video.

[0112] Furthermore, in the first embodiment of the present invention, the red channel pixel value data, the green channel pixel value data and the blue channel pixel data are used to calculate the subtle change frequency of the original human body video, further comprising:

[0113] The array method is used to calculate the frequency of subtle changes in the original human body video, and the array method is specifically as follows:

[0114] Performing Fourier transform on the red channel pixel value data, the green channel pixel value data and the blue channel pixel data respectively;

[0115] Obtaining frequency arrays of each color channel from the red channel pixel value data, the green channel pixel value data, and the blue channel pixel data after Fourier transformation respectively;

[0116] Get the absolute value of the frequency array of each color channel and sort them according to the frequency;

[0117] Calculate the average value of several maximum frequencies of the frequency array of each color channel to determine the average change frequency of each color channel;

[0118] The average change frequency of each color channel is averaged to obtain the subtle change frequency of the original human body video.

[0119] As an example of the first embodiment of the present invention, the method for calculating the subtle change frequency of the original human body video can adopt the following method: use the numpy.fft.fft function to perform Fourier transform operation on the red channel pixel value data, the green channel pixel value data and the blue channel pixel data respectively; use the numpy.fft.fftfreq function on the data after Fourier transform to obtain the frequency array of each color channel; use the numpy.abs function and the numpy.argsort function to obtain the absolute value of the frequency array of each color channel and sort it, and use the numpy.unique function to remove duplicate elements; use the numpy.mean function to average the top 5 frequencies of the frequency array of each color channel to obtain the average change frequency of each color channel; use the numpy.mean function to average the average change frequency of the red channel, the average change frequency of the green channel, and the average change frequency of the blue channel to obtain the subtle change frequency of the original human body video.

[0120] Furthermore, in the first embodiment of the present invention, after extracting the image data of each frame from the second face video, the method further includes:

[0121] For a first pixel point of the second facial video, respectively extract a red pixel value, a green pixel value, and a blue pixel value at a position where the first pixel point is located in image data of each frame of the second facial video;

[0122] The red pixel value, green pixel value and blue pixel value of the first pixel are used to determine a red channel pixel value change curve graph, a green channel pixel value change curve graph and a blue channel pixel value change curve graph of the first pixel respectively.

[0123] In the first embodiment of the present invention, see Figure 4 , is another channel pixel value change curve graph provided by the present invention. The second face video is read respectively by using the VideoCapture function of the OpenCV library, and the image data of each frame of the video can be obtained. When the user clicks on any point in the video, the red pixel value, green pixel value and blue pixel value at the same position of each frame of the video can be extracted respectively. The red channel pixel value change curve graph, the green channel pixel value change curve graph and the blue channel pixel value change curve graph can be drawn according to the red pixel value, the green pixel value and the blue pixel value using the matplotlib library to intuitively display the pixel value change of the face video.

[0124] In summary, the first embodiment of the present invention provides a method for detecting subtle changes in a face, which uses a neural network to perform face recognition and crop the original human body video, retaining only the face area to form a first face video, so as to reduce the interference of other irrelevant background pixels and reduce the amount of calculation; uses an image processing algorithm and a signal processing algorithm to extract feature signals in the first face video, amplifies the subtle changes in the video by applying an amplification factor so that the subtle signals can be more easily observed, and reconstructs the video to form a second face video; extracts red channel data, green channel data, and blue channel data from the second face video; calculates the subtle change frequency of the original human body video by analyzing the data of each color channel, and realizes the observation of subtle changes in the face.

[0125] Example 2

[0126] See also Figure 5 , is a schematic diagram of the structure of an embodiment of a device for detecting subtle changes in a face provided by the present invention, the device comprising a recognition module 201, a feature extraction module 202, a merging module 203, a channel data acquisition module 204, a pixel value data acquisition module 205 and a frequency calculation module 206;

[0127] The recognition module 201 is used to perform face recognition and cropping on the original human body video using a preset face recognition neural network algorithm to obtain a first face video;

[0128] The feature extraction module 202 is used to extract video feature information from the first face video using a feature extraction algorithm composed of several downsampling operations;

[0129] The merging module 203 is used to amplify the signal of the video feature information, and merge the amplified video feature information with the first face video to form a second face video;

[0130] The channel data acquisition module 204 is used to extract image data of each frame in the second face video, and respectively extract red channel data, green channel data and blue channel data from the image data;

[0131] The pixel value data acquisition module 205 is used to analyze the red channel data, the green channel data and the blue channel data, and obtain the red channel pixel value data, the green channel pixel value data and the blue channel pixel data under each color channel respectively;

[0132] The frequency calculation module 206 is used to calculate the subtle change frequency of the original human body video by using the red channel pixel value data, the green channel pixel value data and the blue channel pixel data.

[0133] Further, in the second embodiment of the present invention, a preset face recognition neural network algorithm is used to perform face recognition and cropping on the original human body video to obtain a first face video, specifically:

[0134] Using a preset constructor to read the original human body video, obtain the original frame rate information of the original human body video and the image data of each frame;

[0135] Using a preset face recognition neural network algorithm to perform face recognition on the first frame image of the original human body video, and generate the coordinate point information of the box where the face of the first frame image is located;

[0136] Cropping each frame of the original human body video according to the coordinate point information of the box within the range where the face of the first frame of the image is located;

[0137] According to the original frame rate information, each frame image after cropping the original human body video is saved using a preset saving function to generate a first face video.

[0138] Furthermore, in the second embodiment of the present invention, a feature extraction algorithm composed of several downsampling operations is used to extract video feature information from the first face video, specifically:

[0139] Read each frame of the first face video, and convert the format of each frame of the first face video into a YIQ format;

[0140] Using a feature extraction algorithm composed of several downsampling operations, video feature information of different scales is extracted from each frame image of the first face video converted into the YIQ format; wherein the first face video reduces the image size while undergoing the downsampling operation.

[0141] Furthermore, in the second embodiment of the present invention, the video feature information is amplified, and the amplified video feature information is merged with the first face video to form a second face video, specifically:

[0142] Using a preset interpolation function to perform a zero-order interpolation operation on the first face video processed by the feature extraction algorithm to restore the size of the first face video;

[0143] The video feature information is converted into a frequency domain signal by Fourier transform, and the signal is filtered by a preset bandpass filter;

[0144] Convert the video feature information after signal filtering back to spatial domain signal;

[0145] Multiplying the spatial domain signal by a preset amplification factor to amplify the video feature information;

[0146] The amplified video feature information is merged with the first face video in the YIQ format, and the merged data is saved using a preset saving function according to the original frame rate information to generate a second face video.

[0147] Further, in the second embodiment of the present invention, the red channel data, the green channel data and the blue channel data are analyzed to obtain red channel pixel value data, green channel pixel value data and blue channel pixel data under each color channel, respectively, specifically:

[0148] Traversing all pixel points of the second face video; wherein each pixel point has the same position in each frame of the second face video;

[0149] Determine the amplitude values ​​of all pixels of the second face video in the red channel, the green channel and the blue channel respectively according to the red channel data, the green channel data and the blue channel data;

[0150] Determine the maximum amplitude point corresponding to the maximum amplitude value under the red channel, green channel and blue channel, and obtain the pixel value data of the maximum amplitude point under each color channel in the corresponding red channel data, green channel data and blue channel data to form red channel pixel value data, green channel pixel value data and blue channel pixel data.

[0151] Furthermore, in the second embodiment of the present invention, the red channel pixel value data, the green channel pixel value data and the blue channel pixel data are used to calculate the subtle change frequency of the original human body video, further comprising:

[0152] The extreme value method is used to calculate the subtle change frequency of the original human body video. The extreme value method is specifically as follows:

[0153] Performing polynomial fitting on the red channel pixel value data, the green channel pixel value data and the blue channel pixel data respectively;

[0154] Derivatives are respectively performed on the fitted red channel pixel value data, green channel pixel value data and blue channel pixel data;

[0155] Obtaining maximum and minimum points in the fitting curves corresponding to the derived red channel pixel value data, green channel pixel value data, and blue channel pixel value data, respectively;

[0156] The average distance between the maximum value points under each color channel and the average distance between the minimum value points under each color channel are determined as the average change frequency of each color channel;

[0157] The average change frequency of each color channel is averaged to obtain the subtle change frequency of the original human body video.

[0158] Furthermore, in the second embodiment of the present invention, the red channel pixel value data, the green channel pixel value data and the blue channel pixel data are used to calculate the subtle change frequency of the original human body video, further comprising:

[0159] The array method is used to calculate the frequency of subtle changes in the original human body video, and the array method is specifically as follows:

[0160] Performing Fourier transform on the red channel pixel value data, the green channel pixel value data and the blue channel pixel data respectively;

[0161] Obtaining frequency arrays of each color channel from the red channel pixel value data, the green channel pixel value data, and the blue channel pixel data after Fourier transformation respectively;

[0162] Get the absolute value of the frequency array of each color channel and sort them according to the frequency;

[0163] Calculate the average value of several maximum frequencies of the frequency array of each color channel to determine the average change frequency of each color channel;

[0164] The average change frequency of each color channel is averaged to obtain the subtle change frequency of the original human body video.

[0165] Furthermore, in the second embodiment of the present invention, after extracting the image data of each frame from the second face video, the method further includes:

[0166] For a first pixel point of the second facial video, respectively extract a red pixel value, a green pixel value, and a blue pixel value at a position where the first pixel point is located in image data of each frame of the second facial video;

[0167] The red pixel value, green pixel value and blue pixel value of the first pixel are used to determine a red channel pixel value change curve graph, a green channel pixel value change curve graph and a blue channel pixel value change curve graph of the first pixel respectively.

[0168] Furthermore, in the second embodiment of the present invention, the process of forming the original human body video is specifically as follows:

[0169] Under dark room conditions, low-frequency strobe lighting is used for fill lighting, and a high-sampling rate camera is used to record the human body; wherein the human body remains still during video recording.

[0170] In summary, the second embodiment of the present invention provides a device for detecting subtle changes in a face. Based on the organic combination of modules, a neural network is used to perform face recognition and crop the original human body video, and only the face area is retained to form a first face video, so as to reduce the interference of other irrelevant background pixels and reduce the amount of calculation; an image processing algorithm and a signal processing algorithm are used to extract characteristic signals in the first face video, and the subtle changes in the video are amplified by applying an amplification factor so that the subtle signals can be more easily observed, and the video is reconstructed to form a second face video; red channel data, green channel data and blue channel data are respectively extracted from the second face video; by analyzing the data of each color channel, the subtle change frequency of the original human body video is calculated, so as to realize the observation of subtle changes in the face.

[0171] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting subtle changes in a face, characterized in that: include: Using a preset face recognition neural network algorithm to perform face recognition and cropping on the original human body video to obtain a first face video; Extracting video feature information from the first face video using a feature extraction algorithm consisting of a number of downsampling operations; amplifying the video feature information, and combining the amplified video feature information with the first face video to form a second face video; Extracting image data of each frame from the second face video, and respectively extracting red channel data, green channel data, and blue channel data from the image data; Analyze the red channel data, the green channel data and the blue channel data to obtain red channel pixel value data, green channel pixel value data and blue channel pixel data under each color channel respectively; The red channel pixel value data, the green channel pixel value data and the blue channel pixel data are used to calculate the subtle change frequency of the original human body video by using an extreme value method or an array method; wherein the extreme value method is specifically: Performing polynomial fitting on the red channel pixel value data, the green channel pixel value data and the blue channel pixel data respectively; Derivatives are respectively performed on the fitted red channel pixel value data, green channel pixel value data and blue channel pixel data; Obtaining maximum and minimum points in the fitting curves corresponding to the derived red channel pixel value data, green channel pixel value data, and blue channel pixel value data, respectively; The average distance between the maximum value points under each color channel and the average distance between the minimum value points under each color channel are determined as the average change frequency of each color channel; The average change frequency of each color channel is averaged to obtain the subtle change frequency of the original human body video; The array method is specifically: Performing Fourier transform on the red channel pixel value data, the green channel pixel value data and the blue channel pixel data respectively; Obtaining frequency arrays of each color channel from the red channel pixel value data, the green channel pixel value data, and the blue channel pixel data after Fourier transformation respectively; Get the absolute value of the frequency array of each color channel and sort them according to the frequency; Calculate the average value of several maximum frequencies of the frequency array of each color channel to determine the average change frequency of each color channel; The average change frequency of each color channel is averaged to obtain the subtle change frequency of the original human body video.

2. The method for detecting subtle changes in a face according to claim 1, characterized in that: The method uses a preset face recognition neural network algorithm to perform face recognition and cropping on the original human body video to obtain the first face video, specifically: Using a preset constructor to read the original human body video, obtaining original frame rate information of the original human body video and image data of each frame; Using a preset face recognition neural network algorithm to perform face recognition on the first frame image of the original human body video, and generate the coordinate point information of the frame where the face of the first frame image is located; Cropping each frame of the original human body video according to the coordinate point information of the box within the range where the human face of the first frame of image is located; According to the original frame rate information, each frame image after cropping the original human body video is saved using a preset saving function to generate a first face video.

3. The method for detecting subtle changes in a face according to claim 2, characterized in that: The extracting of video feature information from the first face video using a feature extraction algorithm composed of several downsampling operations is specifically: Read each frame of the first face video, and convert the format of each frame of the first face video into a YIQ format; Using a feature extraction algorithm composed of several downsampling operations, video feature information of different scales is extracted from each frame image of the first face video converted into the YIQ format; wherein the first face video reduces the image size while undergoing the downsampling operation.

4. The method for detecting subtle changes in a face according to claim 3, characterized in that: The signal amplification of the video feature information and merging the signal-amplified video feature information with the first face video to form the second face video is specifically: Using a preset interpolation function to perform a zero-order interpolation operation on the first face video processed by the feature extraction algorithm to restore the size of the first face video; The video feature information is converted into a frequency domain signal by Fourier transform, and the signal is filtered by a preset bandpass filter; Convert the video feature information after signal filtering back to spatial domain signal; Multiplying the spatial domain signal by a preset amplification factor to amplify the video feature information; The amplified video feature information is merged with the first face video in the YIQ format, and the merged data is saved using a preset saving function according to the original frame rate information to generate a second face video.

5. The method for detecting subtle changes in a face according to claim 4, characterized in that: The analysis of the red channel data, the green channel data and the blue channel data to obtain red channel pixel value data, green channel pixel value data and blue channel pixel data under each color channel is specifically as follows: Traversing all pixel points of the second face video; wherein each pixel point has the same position in each frame of the second face video; Determine the amplitude values ​​of all pixels of the second face video in the red channel, the green channel and the blue channel respectively according to the red channel data, the green channel data and the blue channel data; Determine the maximum amplitude point corresponding to the maximum amplitude value under the red channel, green channel and blue channel, and obtain the pixel value data of the maximum amplitude point under each color channel in the corresponding red channel data, green channel data and blue channel data to form red channel pixel value data, green channel pixel value data and blue channel pixel data.

6. The method for detecting subtle changes in a face according to claim 1, characterized in that: After extracting the image data of each frame from the second face video, the method further includes: For a first pixel point of the second facial video, respectively extract a red pixel value, a green pixel value, and a blue pixel value at a position where the first pixel point is located in image data of each frame of the second facial video; The red pixel value, green pixel value and blue pixel value of the first pixel are used to determine a red channel pixel value change curve graph, a green channel pixel value change curve graph and a blue channel pixel value change curve graph of the first pixel respectively.

7. The method for detecting subtle changes in a face according to claim 1, characterized in that: The formation process of the original human body video is specifically as follows: Under dark room conditions, low-frequency strobe lighting is used for fill lighting, and a high-sampling rate camera is used to record the human body; wherein the human body remains still during video recording.

8. A device for detecting subtle changes in a face, characterized in that: include: Recognition module, feature extraction module, merging module, channel data acquisition module, pixel value data acquisition module and frequency calculation module; The recognition module is used to perform face recognition and cropping on the original human body video using a preset face recognition neural network algorithm to obtain a first face video; The feature extraction module is used to extract video feature information from the first face video using a feature extraction algorithm composed of several downsampling operations; The merging module is used to amplify the signal of the video feature information, and merge the amplified video feature information with the first face video to form a second face video; The channel data acquisition module is used to extract image data of each frame in the second face video, and respectively extract red channel data, green channel data and blue channel data from the image data; The pixel value data acquisition module is used to analyze the red channel data, the green channel data and the blue channel data to obtain the red channel pixel value data, the green channel pixel value data and the blue channel pixel data under each color channel respectively; The frequency calculation module is used to calculate the subtle change frequency of the original human body video by using the red channel pixel value data, the green channel pixel value data and the blue channel pixel data, using an extreme value method or an array method; The extreme value method is specifically as follows: Performing polynomial fitting on the red channel pixel value data, the green channel pixel value data and the blue channel pixel data respectively; Derivatives are respectively performed on the fitted red channel pixel value data, green channel pixel value data and blue channel pixel data; Obtaining maximum and minimum points in the fitting curves corresponding to the derived red channel pixel value data, green channel pixel value data, and blue channel pixel value data, respectively; The average distance between the maximum value points under each color channel and the average distance between the minimum value points under each color channel are determined as the average change frequency of each color channel; The average change frequency of each color channel is averaged to obtain the subtle change frequency of the original human body video; The array method is specifically: Performing Fourier transform on the red channel pixel value data, the green channel pixel value data and the blue channel pixel data respectively; Obtaining frequency arrays of each color channel from the red channel pixel value data, the green channel pixel value data, and the blue channel pixel data after Fourier transformation respectively; Get the absolute value of the frequency array of each color channel and sort them according to the frequency; Calculate the average value of several maximum frequencies of the frequency array of each color channel to determine the average change frequency of each color channel; The average change frequency of each color channel is averaged to obtain the subtle change frequency of the original human body video.

Images