System and method for judging traditional Chinese medicine pulse condition through face video based on multi-wavelength light source

Abstract

The invention relates to the technical field of medical treatment and information, in particular to a system and method for judging traditional Chinese medicine pulse conditions through face videos based on a multi-wavelength light source. The video acquisition module acquires face videos under light sources with different wavelengths, the video preprocessing module cuts the videos into images with fixed frame number and size, and the waveform extraction module is combined with the deep neural network to extract and detect rough signals of a face interested area. After multiple filters are used for filtering, fusion representation information is obtained through a waveform fusion unit, eight elements of pulse conditions are judged through a pulse condition recognition module in combination with a deep neural network, meanwhile, prediction of the eight elements is conducted, judgment error data are sent to a model upgrading module for retraining, and therefore model closed-loop is achieved. According to the method, the filtered multi-wavelength pulse waves can be obtained, the pertinence of facial pulse information extraction is improved, the deep pulse condition information of the face is further extracted, and the accuracy of pulse condition diagnosis is improved in combination with the deep neural network.

Description

Technical Field

[0001] The present invention relates to the fields of medical treatment and information technology, and in particular to a system and method for distinguishing traditional Chinese medicine pulse based on facial video fusion of multi-wavelength light sources. Background Art

[0002] Pulse diagnosis is one of the four diagnostic methods in traditional Chinese medicine and has important clinical guidance significance. In recent years, with the continuous breakthrough of modern science and technology, the objectivity of pulse diagnosis has made rapid progress.

[0003] Imaging photoplethysmography, or IPPG, is a transcutaneous optical signal acquisition method that can be used to detect various indicators related to arterial pulsation. The IPPG method records the periodic changes in the intensity of reflected light through the detector, capturing the subtle changes in the irradiated part caused by the pulse pulsation, thereby achieving pulse diagnosis; the process mostly uses imaging devices as detectors and records signals in the form of video, effectively avoiding physical connection with the patient, and has the characteristics of non-contact, easy platform implementation, convenient operation, and low cost.

[0004] Among the many IPPG detection sites, the face is highly similar to the TCM pulse diagnosis "Cunkou pulse" in terms of TCM theory and holographic restoration, providing the optimal collection site for non-contact pulse diagnosis.

[0005] The key technology of the IPPG-based facial video detection method for TCM pulse is to extract the pulse wave in the facial video, so as to identify the TCM pulse. The specific method is: based on the interaction between light and facial skin and the photoelectric effect, such as skin penetration and reflection, the periodic changes of reflected light with blood vessel pulsation are captured, and the pulse wave is obtained through noise filtering and other means. The main steps of this method currently include: (1) Facial video capture based on natural light / conventional lighting illuminating the face; (2) After image processing, the red, green, and blue three-channel signals are extracted, and only the green channel is extracted and simple signal denoising is performed to process the signal to obtain the pulse wave; The technical solutions currently used have the following problems and defects: (1) The video acquisition light source is limited to the visible light band, ignoring the ability of near-infrared light to obtain skin signals, which reduces the accuracy of judging the pulse condition of traditional Chinese medicine; (2) The signal collection light source is messy and has complex photoelectric interactions with the skin, which can easily cause significant interference, increasing the difficulty of subsequent signal analysis and reducing the accuracy of TCM pulse identification. Summary of the invention

[0006] The purpose of the present invention is to improve the problem in the prior art that the signal acquisition light source is limited to natural light, resulting in insufficient extraction of deep facial pulse information and high signal noise, and low accuracy in distinguishing the Chinese medicine pulse, and to provide a system and method for distinguishing the Chinese medicine pulse based on facial video fusion of multi-wavelength light sources.

[0007] To achieve the above object of the invention, the present invention provides a method for distinguishing Chinese medicine pulse condition based on facial video fused with multi-wavelength light sources, comprising the following steps: S1: The video acquisition module collects face videos under different wavelengths of light for identification of TCM pulses; S2: The video preprocessing module preprocesses the collected face video to generate an image with a fixed frame number and a fixed size; S3: The coarse signal extraction unit of the waveform extraction module detects the facial region of interest in the image using a deep neural network model, and extracts coarse signals of different wavelengths by spatial pixel averaging; S4: The filtering unit of the waveform extraction module filters the rough signals of different wavelengths to obtain pure signals of different wavelengths; S5: The waveform fusion unit processes the pure signals at different wavelengths to construct a signal fusion network to obtain fusion representation information; S6: The pulse recognition module recognizes the pulse according to the fusion representation information, obtains the recognition result of the eight elements of the pulse in combination with the deep neural network model, and predicts the eight elements of the pulse in combination with a multi-task learning method; S7: The model upgrade module sends the incorrectly judged pulse data into the deep neural network model for retraining.

[0008] Preferably, the video acquisition module in step S1 acquires facial videos under illumination of light sources of different wavelengths for distinguishing TCM pulses, and includes a multi-wavelength light acquisition unit and a video shooting unit. The multi-wavelength light acquisition unit includes a light source acquisition platform for visible light and near-infrared light, which respectively emit light source signals of corresponding wavelengths. The video shooting unit is operated in a darkroom and uses a high-definition black-and-white camera to acquire facial videos under illumination of light sources of different wavelengths. During acquisition, the subject sits or stands in front of the camera with his eyes closed to avoid wearing glasses, masks or other facial obstructions.

[0009] In step S2, the video preprocessing module uses audio and video codecs to process the face video respectively, and crops the face video into image frames with a fixed number of frames and a fixed size as input of the waveform extraction module.

[0010] Preferably, the waveform extraction module includes a coarse signal extraction unit and a filtering unit. The coarse signal extraction unit uses a deep neural network to detect the facial region of interest, and obtains coarse signals of different wavelengths by spatial pixel averaging. The filtering unit uses a variety of filters to filter the signal. In step S3, the coarse signal extraction unit uses a deep neural network to detect the facial region of interest, and the coarse signals C(n) of different wavelengths are obtained by spatial pixel averaging. The method is as follows: Among them, M(x,y,n) is the pixel intensity of the pixel point with coordinates (x,y) in the nth frame image, X skin is the facial region of interest in the nth frame, K is X skin The number of pixels detected in .

[0011] Preferably, in step S4, the filtering unit of the waveform extraction module filters the rough signal to obtain pure signals at different wavelengths. The filtering unit uses three methods, namely, Bland-Altman analysis method, ensemble empirical mode decomposition method, and Savitzky-Golay filter, to perform multi-wavelength rough signal filtering to obtain pure signals at different wavelengths. The Bland-Altman analysis method filters mutation noise, the Savitzky-Golay filter filters high-frequency noise, and the ensemble empirical mode decomposition method filters low-frequency noise to obtain pure signals at different wavelengths.

[0012] Preferably, in step S5, the waveform fusion unit processes the pure signals at different wavelengths to construct a signal fusion network, uses transformer and its variants to collaboratively represent and encode the pure signals at different wavelengths, adopts a cross-attention mechanism to perform pairwise interactive training, and fuses the obtained interactive fusion features again to automatically obtain the fused representation information of the pulse wave corresponding to the multi-wavelength video.

[0013] Preferably, in step S6, the pulse recognition module obtains the recognition results of the eight elements of the pulse based on the above-mentioned extracted features through a deep neural network, applies multi-task learning to predict the eight elements of the pulse, and predicts the eight elements of the pulse by sharing underlying features, and the underlying features are the common features of the eight elements of the pulse.

[0014] Step S7 performs model training, and the pulse results of each patient are authenticated and marked by experts, and the expert authentication result is used as the correct label of the case. When the model prediction result is different from it, the result is identified as model judgment error data, input into the model upgrade module, and enter the model for retraining to improve the model's attention to difficult-to-classify samples, thereby achieving model closed-loop optimization.

[0015] The present invention provides a system for distinguishing TCM pulse condition based on facial video fusion of multi-wavelength light sources. Figure 1 As shown, the system includes a video acquisition module, a video preprocessing module, a waveform extraction module, a waveform fusion unit, a pulse recognition module, and a model upgrade module. Information is entered into the video acquisition module. The multi-wavelength light acquisition unit builds a light source acquisition platform including visible light and near-infrared light. The light source is composed of LED circuits of different wavelengths, which respectively emit light source signals of corresponding wavelengths. The video shooting unit is operated in a darkroom and uses a high-definition black and white camera to collect facial videos under the irradiation of light sources of different wavelengths. During the collection, the subject sits or stands in front of the camera, closes his eyes and avoids wearing glasses, masks and other facial obstructions. The collected multi-wavelength facial video signals are input into the video preprocessing module. The video preprocessing module The processed picture frame is input into the waveform extraction module, the coarse signal extraction unit in the waveform extraction module extracts the coarse signals of different wavelengths in the picture frame and sends them to the filtering unit, the filtering unit inputs the extracted pure signals under different wavelengths into the waveform fusion unit, the waveform fusion unit inputs the fusion representation information of the pulse wave corresponding to the processed multi-wavelength video into the pulse recognition module, and the pulse recognition module inputs the judged result into the model upgrading module. Through the improvement of the present invention, compared with the existing method of detecting Chinese medicine pulse by facial video based on IPPG, the diagnostic accuracy of the eight elements of pulse is improved by about 6.9%.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention provides a system and method for judging TCM pulse based on facial video of multi-wavelength light source, which collects facial video under visible light and near-infrared light for pulse judgment, processes and fuses signals of different wavelengths, obtains fusion representation information of pulse waves corresponding to multi-wavelength videos, and further extracts deep facial pulse wave information. Compared with the existing method of detecting TCM pulse based on facial video of IPPG, the diagnostic accuracy of eight pulse elements is improved by 6.9%; 2. The present invention provides a system and method for distinguishing TCM pulse based on facial video of multi-wavelength light source, which extracts and filters multi-wavelength signals to obtain pure signals at different wavelengths, thereby improving the problem of high signal noise and reduced pulse diagnosis accuracy. Compared with the existing method of detecting TCM pulse based on facial video of IPPG, the diagnostic accuracy of the eight elements of pulse is improved by 6.9%. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a structural block diagram of a system and method for identifying TCM pulse based on facial video with multi-light source fusion.

[0018] Figure 2This is the schematic diagram of the waveform fusion unit.

[0019] Figure 3 Schematic diagram of the video shooting module. DETAILED DESCRIPTION

[0020] The present invention is further described in detail below in conjunction with test examples and specific implementation methods. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0021] Example 1 The present invention provides a system and method for distinguishing TCM pulse condition based on facial video of multi-wavelength light source, such as Figure 1 As shown, after the information is entered, it is sequentially sent to the video acquisition module, the video preprocessing module, the waveform extraction module, the waveform fusion unit, the pulse recognition module and the model upgrading module.

[0022] First is the video acquisition module. The doctor logs into the system, enters the subject's information (name, gender, age, ID number, etc.) and starts TCM pulse identification. The shooting environment in the multi-wavelength light acquisition unit of the video acquisition module is a dark room. The subject sits or stands, keeps his face still, closes his eyes, and places a black and white high-definition camera in front of the human face. The multi-wavelength lamp is placed above the camera in turn, and the video shooting unit shoots facial videos for about 40 seconds under each wavelength.

[0023] After the video acquisition is completed, the video preprocessing module is entered. The video codec tool FFmpeg is used to read the video streams of different wavelengths and divide them into frames, and the video frames are adjusted to image frames with a fixed number of frames and a fixed size. In this embodiment, the number of frames is 256 frames and the size is 256×256 pixels.

[0024] The acquired image frame is sent to the waveform extraction module. The coarse signal extraction unit uses a deep neural network to detect image frames, detects the facial region of interest, obtains a coarse signal by spatial pixel averaging, and sends it to the filtering unit for signal filtering using multiple filters. The deep neural network determines that the largest skin area of ​​the face is the facial region of interest, and uses a semantic segmentation model based on the Diffusion deep neural network for extraction. The filtering unit uses the Bland-Altman analysis method, the ensemble empirical mode decomposition algorithm, and the Savitzky-Golay filter to denoise high and low frequencies and mutation noise.

[0025] The pure signals at different wavelengths extracted by the waveform extraction module are sent to the waveform fusion unit, and are collaboratively represented and encoded using transformer and its variants, such as Figure 2As shown, the cross-attention mechanism is used to interactively train the pure signals at different wavelengths in pairs to construct a signal fusion network, extract the fused pulse wave features, and obtain the fused representation information of the pulse wave corresponding to the multi-wavelength video.

[0026] The pulse recognition module obtains the recognition results of the eight elements of the pulse condition based on the fusion representation information through a deep neural network; multi-task learning is applied to predict the eight elements of the pulse condition, namely pulse position, pulse length, pulse width, pulse rate, tension, fluency, strength, and uniformity, and the prediction of the eight elements of the pulse condition is achieved by sharing the underlying features. In this embodiment, the eight elements of the pulse condition are divided into eight subtasks, and each subtask is refined into a two-classification or three-classification task depending on the outcome indicator.

[0027] The model upgrade module classifies and identifies the pulse of the subject. If the system identification result is inconsistent with the expert diagnosis result, the data is sent to the model for retraining using the OHME training method, thereby achieving closed-loop optimization of the model.

[0028] Example 2 The video acquisition module of the present invention comprises the multi-wavelength light acquisition unit and the video shooting unit.

[0029] The multi-wavelength light acquisition unit mainly uses LED light sources and cameras of different wavelengths to emit light source signals of corresponding wavelengths. This embodiment uses three LED light sources with wavelengths of 530nm, 650nm and 940nm, with light output power > 20mw and electrical power ≥ 20w. This embodiment selects the Maidewei MV-GE134 industrial camera as the video acquisition device, and the lens is selected as MV-LD-4-4M-G, with a focal length of 6mm, f1.4.

[0030] like Figure 3 As shown, the video shooting unit is used for shooting in a dark room. During the acquisition process, the subject sits quietly facing the camera and about 80 cm away from it horizontally, closes his eyes, avoids wearing masks, eyes and other facial coverings, and avoids hair covering the face. A multi-wavelength LED light is placed above the camera, and the LED lights are turned on separately to shoot a facial video of about 40 seconds at each wavelength.

[0031] After the video is shot, it undergoes video preprocessing, waveform extraction, and waveform fusion and is sent to the pulse recognition module for eight-element pulse diagnosis. The study found that compared with the existing IPPG-based facial video detection of TCM pulse, the diagnostic accuracy of the eight-element pulse was improved by 6.9%.

[0032] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for distinguishing Chinese medicine pulse based on facial video fusion of multi-wavelength light sources, characterized in that: The following steps are involved: S1: The video acquisition module collects face videos under different wavelengths of light for identification of TCM pulses; S2: The video preprocessing module preprocesses the collected face video to generate an image with a fixed frame number and a fixed size; S3: The coarse signal extraction unit of the waveform extraction module detects the facial region of interest in the image using a deep neural network model, and extracts coarse signals of different wavelengths by spatial pixel averaging; S4: The filtering unit of the waveform extraction module filters the rough signals of different wavelengths to obtain pure signals of different wavelengths; S5: The waveform fusion unit processes the pure signals at different wavelengths to construct a signal fusion network to obtain fusion representation information; S6: The pulse recognition module recognizes the pulse according to the fusion representation information, obtains the recognition result of the eight elements of the pulse in combination with the deep neural network model, and predicts the eight elements of the pulse in combination with a multi-task learning method; S7: The model upgrade module sends the incorrectly judged pulse data into the deep neural network model for retraining.

2. A method for distinguishing Chinese medicine pulse based on facial video of multi-wavelength light source fusion according to claim 1, characterized in that: The video acquisition module in step S1 includes a multi-wavelength light acquisition unit and a video shooting unit. The multi-wavelength light acquisition unit includes a light source acquisition platform for visible light and near-infrared light, which respectively emit light source signals of corresponding wavelengths. The video shooting unit is operated in a dark room and uses a high-definition black and white camera to capture facial videos under light sources of different wavelengths.

3. A method for distinguishing Chinese medicine pulse based on facial video of multi-wavelength light source fusion according to claim 1, characterized in that: In step S3, rough signals C(n) of different wavelengths are obtained by spatial pixel averaging as follows: Among them, M(x,y,n) is the pixel intensity of the pixel point with coordinates (x,y) in the nth frame image, X skin is the facial region of interest in the nth frame, K is X skin The number of pixels detected in .

4. A method for distinguishing Chinese medicine pulse based on facial video of multi-wavelength light source fusion according to claim 1, characterized in that: The filtering unit of the waveform extraction module in step S4 uses three methods, namely, Bland-Altman analysis, ensemble empirical mode decomposition, and Savitzky-Golay filter, to perform multi-wavelength coarse signal filtering to obtain pure signals at different wavelengths. The Bland-Altman analysis method filters mutation noise, the Savitzky-Golay filter filters high-frequency noise, and the ensemble empirical mode decomposition method filters low-frequency noise.

5. A method for distinguishing Chinese medicine pulse based on facial video of multi-wavelength light source fusion according to claim 1, characterized in that: The waveform fusion unit in step S5 uses transformer and its variants to collaboratively represent and encode signals of different wavelengths, adopts cross-attention mechanism to perform pairwise interactive training, and fuses the obtained interactive fusion features again to automatically obtain the fused representation information of the pulse wave corresponding to the multi-wavelength video.

6. A method for distinguishing Chinese medicine pulse based on facial video of multi-wavelength light source fusion according to claim 1, characterized in that: The pulse recognition module in step S6 obtains the recognition results of the eight elements of the pulse based on the above-mentioned extracted features through a deep neural network, applies multi-task learning to predict the eight elements of the pulse, and predicts the eight elements of the pulse by sharing the underlying features.

7. A system for distinguishing TCM pulse based on facial video using multi-wavelength light source fusion, characterized in that: It includes video acquisition module, video preprocessing module, waveform extraction module, waveform fusion unit, pulse recognition module and model upgrade module. The video acquisition module includes a multi-wavelength light acquisition unit and a video shooting unit. The multi-wavelength light acquisition unit uses a multi-wavelength light source to illuminate the human face. The video shooting unit collects facial videos illuminated by light sources of different wavelengths as input to the video preprocessing module. The video preprocessing module uses an audio and video codec to process the facial video collected by the video acquisition module, and cuts the video into image frames with a fixed number of frames and a fixed size as inputs of the waveform extraction module. The waveform extraction module includes a coarse signal extraction unit and a filtering unit. The coarse signal extraction unit uses a deep neural network to detect the facial region of interest, processes the image frame output by the video preprocessing module by spatial pixel averaging, and obtains coarse signals of different wavelengths. The filtering unit uses a variety of filters to filter the coarse signals to obtain pure signals at different wavelengths as inputs of the waveform fusion unit. The waveform fusion unit uses transformer and its variants for collaborative representation and encoding, adopts cross-attention mechanism to perform two-by-two interactive training on the pure signals under different wavelengths to construct a signal fusion network, and obtains the fusion representation information of the pulse wave corresponding to the multi-wavelength video as the input of the pulse recognition module. The pulse recognition module obtains the recognition results of the eight elements of the pulse based on the fusion representation information through a deep neural network, applies multi-task learning to share the underlying features, and simultaneously predicts the eight elements, and uses the recognition results and the prediction results as the input of the model upgrade module. The model upgrade module sends the model's erroneous data to the model for retraining, thereby increasing the model's attention to difficult-to-classify samples, thereby achieving closed-loop optimization of the model.

8. A system for distinguishing TCM pulse based on facial video of multi-wavelength light source fusion according to claim 7, characterized in that: The waveform extraction module includes a coarse signal extraction unit and a filtering unit. The coarse signal extraction unit includes a deep neural network and spatial pixel averaging. The filtering unit includes a Bland-Altman analysis method, an ensemble empirical mode decomposition method, and a Savitzky-Golay filter. The image frames of the videos with different wavelengths output by the video preprocessing module are input, and the pure signals at different wavelengths are output.

9. A system for distinguishing Chinese medicine pulse based on facial video of multi-wavelength light source fusion according to claim 7, characterized in that: The waveform fusion unit includes transformer and its variants and cross-attention mechanism, which inputs the pure signals under different wavelengths and outputs the fusion representation information of the pulse waves corresponding to the multi-wavelength videos.

10. A system for distinguishing TCM pulse based on facial video of multi-wavelength light source fusion according to claim 7, characterized in that: The pulse recognition module includes a deep neural network and multi-task learning, inputs the fusion representation information, and outputs the pulse identification result and the prediction of the eight elements of the pulse.

Images