Foot video evaluation method, device, terminal device and storage medium based on the arrival time difference of two-foot pulse waves

Abstract

The present application relates to the field of biomedical engineering technology. The present application discloses a plantar video evaluation method, apparatus, terminal device and storage medium based on the arrival time difference of the pulse waves of both feet, which can improve the accuracy of the evaluation results. The method includes obtaining a plantar video of both feet of the target user; extracting the first pixel time domain change signal of each pixel point in the plantar region of both feet of each frame image according to the plantar video to obtain the pulse wave signal corresponding to each plantar region; determining multiple regions of interest in the plantar video based on the pulse wave signal corresponding to each plantar region to obtain multiple target regions of interest; extracting the target rPPG signal corresponding to each foot based on the multiple target regions of interest, and constructing a two-foot time difference wave for reflecting the arrival time difference change of the target rPPG signal corresponding to the two feet; performing evaluation processing based on the two-foot time difference wave to obtain an evaluation result, which is used to assist in the diagnosis of peripheral arterial disease and its severity.

Description

Technical Field

[0001] The present application relates to the field of biomedical engineering technology. More specifically, the present application relates to a method, apparatus, terminal device, and storage medium for foot sole video assessment based on the arrival time difference of bilateral pulse waves. Background Art

[0002] Traditional methods for assessing peripheral arterial disease (PAD) involve collecting pulse wave data; extracting the main wave amplitude and the descending isthmus amplitude from the collected pulse wave data; calculating the vascular resistance coefficient based on the main wave amplitude, the descending isthmus amplitude, and preset rules; and evaluating the vascular resistance coefficient according to the preset rules to obtain an assessment result that medical personnel can use to assist in the diagnosis of PAD. The steps of evaluating the vascular resistance coefficient according to the preset rules include analyzing the linear relationship between the vascular resistance coefficient and vascular resistance conditions based on historical data, setting N thresholds for the vascular resistance coefficient based on the vascular resistance conditions in the historical data, and grading the vascular resistance coefficient into N+1 progressive levels. The vascular resistance coefficient is then compared with the N thresholds to determine the level of the vascular resistance coefficient and outputting an assessment result for the level. In other words, the assessment results of traditional methods are limited by historical data. If the historical data contains outliers, the accuracy of the assessment results will be affected, making it difficult for medical personnel to accurately diagnose PAD based on the assessment results. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a method, apparatus, terminal device, and storage medium for foot video assessment based on the arrival time difference of two-foot pulse waves, which can improve the accuracy of the assessment results. The embodiments of the present application are mainly achieved through the following technical solutions:

[0004] A first aspect of the embodiments of the present application provides a foot video assessment method based on the arrival time difference of two-foot pulse waves, comprising:

[0005] Obtain sole videos of the target user's feet;

[0006] Extracting the first pixel time domain change signal of each pixel point in the plantar region of both feet in each frame image according to the plantar video to obtain the pulse wave signal corresponding to each plantar region;

[0007] determining a plurality of regions of interest in the sole video based on pulse wave signals corresponding to each sole region, and obtaining a plurality of target regions of interest;

[0008] Extracting target rPPG signals corresponding to each foot based on the multiple target regions of interest, and constructing a two-foot time difference wave, wherein the two-foot time difference wave is used to reflect the arrival time difference change of the target rPPG signals corresponding to the two feet;

[0009] Evaluation processing is performed based on the time difference wave of the two feet to obtain an evaluation result of the sole video.

[0010] According to one embodiment of the present application, the step of extracting the first pixel time domain change signal of each pixel point in the plantar region of both feet in each frame image according to the plantar video to obtain the pulse wave signal corresponding to each plantar region includes:

[0011] generating a first pixel time-domain change signal for each sole region based on a time sequence of each frame image in the sole video and a pixel value corresponding to each pixel point in each sole region of each frame image;

[0012] The first pixel time domain variation signal of each plantar region is downsampled according to the preset first sliding time window and the first step length to obtain the pulse wave signal corresponding to each plantar region.

[0013] According to one embodiment of the present application, the step of performing downsampling processing on the first pixel time domain change signal of each plantar region according to a preset first sliding time window and a first step length to obtain the pulse wave signal corresponding to each plantar region includes:

[0014] Performing division processing on each first pixel time-domain variation signal based on the first sliding time window and the first step length to obtain a plurality of first sub-signals corresponding to each first pixel time-domain variation signal;

[0015] A filter is used to filter all the first sub-signals to obtain the pulse wave signal corresponding to each sole area.

[0016] According to one embodiment of the present application, the step of determining multiple regions of interest in the sole video based on the pulse wave signals corresponding to each sole area, and obtaining multiple target regions of interest includes:

[0017] The pulse wave signal corresponding to each foot area is converted into the frequency domain, and the signal-to-noise ratio of each foot area is calculated;

[0018] Sort the foot regions according to their signal-to-noise ratios, and select the plantar regions that meet the preset conditions as candidate regions of interest;

[0019] Each candidate region of interest is optimized to obtain a target region of interest corresponding to each candidate region of interest.

[0020] According to one embodiment of the present application, the steps of extracting target rPPG signals corresponding to each foot based on the multiple target regions of interest and constructing a two-foot time difference wave, wherein the two-foot time difference wave is used to reflect the change in the arrival time difference of the target rPPG signals corresponding to the two feet, include:

[0021] Extracting a second pixel time-domain change signal of each pixel point in multiple target regions of interest in the sole video to obtain an initial rPPG signal corresponding to each foot;

[0022] Calculate the average value of the initial rPPG signal corresponding to each foot to obtain the target rPPG signal corresponding to each foot;

[0023] The target algorithm is used to calculate the time difference between the first and second feature points of all target rPPG signals to determine the arrival time difference of multiple bipedal pulse waves;

[0024] The bipedal time difference wave is generated according to the plurality of bipedal pulse wave arrival time differences.

[0025] According to one embodiment of the present application, when the target algorithm is a peak detection algorithm, the step of using the target algorithm to calculate the time difference between the first characteristic point and the second characteristic point between all target rPPG signals and determining the arrival time differences of multiple bipedal pulse waves includes:

[0026] dividing the target rPPG signal corresponding to the left foot based on a preset third sliding time window and a third step size to obtain a plurality of third left sub-signals;

[0027] dividing the target rPPG signal corresponding to the right foot based on the third sliding time window and the third step length to obtain a plurality of third right sub-signals;

[0028] Detecting a first peak of a target third left sub-signal and a second peak of a target third right sub-signal using a peak detection algorithm, wherein the target third left sub-signal is any one of the plurality of third left sub-signals, and the target third right sub-signal is a signal of the plurality of third right sub-signals that occurs at the same time point as the target third left sub-signal;

[0029] A time difference corresponding to the target third left sub-signal is calculated based on the first peak and the second peak.

[0030] According to one embodiment of the present application, the step of performing evaluation processing based on the two-foot time difference wave to obtain the evaluation result of the sole video includes:

[0031] performing feature extraction processing on the bipedal time difference wave to obtain at least one target feature;

[0032] The at least one target feature is input into a classifier for evaluation processing to obtain an evaluation result of the sole video.

[0033] A second aspect of the embodiments of the present application provides a plantar video assessment device based on the arrival time difference of two-foot pulse waves, comprising: a terminal device, an optical imaging device, a fixed bracket, and a transparent plate, wherein the optical imaging device includes a camera and an LED fill light;

[0034] The terminal device includes a processor, which is used to execute the steps of the foot video assessment method based on the arrival time difference of the two-foot pulse waves provided in the first aspect of the embodiment of the present application;

[0035] The transparent plate is placed on the upper part of the fixed bracket. When the camera records the sole video of the target user's feet, the target user's feet are placed on the transparent plate, and the LED fill light is irradiated onto the transparent plate.

[0036] The optical imaging device is placed inside the fixed bracket;

[0037] After recording the sole video, the camera sends the sole video to the processor.

[0038] In a third aspect of an embodiment of the present application, a terminal device is provided, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, and executing the steps of the plantar video assessment method based on the arrival time difference of the pulse waves of both feet provided in the first aspect of the above-mentioned embodiment of the present application.

[0039] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which is used to store a computer program, and the computer program enables a computer to execute the steps of the plantar video evaluation method based on the arrival time difference of the pulse waves of both feet provided in the first aspect of the embodiment of the present application.

[0040] The beneficial effects of the embodiments of the present application include:

[0041] The core of the embodiment of the present application is to obtain an evaluation result by using the two-foot time difference wave of the target rPPG signal corresponding to the two feet of the target user, so that medical staff can assist in the diagnosis of peripheral arterial disease and the severity of peripheral arterial disease based on the evaluation result. Specifically, the embodiment of the present application obtains a plantar video of the two feet of the target user; extracts the first pixel time domain change signal of each pixel point in the plantar area of the two feet of each frame image according to the plantar video, and obtains the pulse wave signal corresponding to each plantar area; determines multiple regions of interest in the plantar video based on the pulse wave signal corresponding to each plantar area, and obtains multiple target regions of interest; extracts the target rPPG signal corresponding to each foot based on the multiple target regions of interest, and constructs a two-foot time difference wave, which is used to reflect the arrival time difference change of the target rPPG signal corresponding to the two feet; performs evaluation processing based on the two-foot time difference wave to obtain the evaluation result of the plantar video. Compared with the prior art, the embodiment of the present application does not use historical data as an evaluation factor, but instead uses the arrival time difference of the target rPPG signals corresponding to the two feet for evaluation, thereby avoiding the situation where the evaluation results are affected by outliers in the historical data. This can improve the accuracy of the evaluation results and enable medical staff to accurately evaluate the condition of lower limb atherosclerosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 This is a flowchart of a method for foot video assessment based on the arrival time difference of two-foot pulse waves of the present application in some embodiments;

[0044] Figure 2 This is a flowchart of the foot video assessment method based on the arrival time difference of two-foot pulse waves in some embodiments of the present application;

[0045] Figure 3 Reference graphs for the target left plantar rPPG signal and the target right plantar rPPG signal in this application;

[0046] Figure 4 This is a principle block diagram of the device for foot video assessment based on the arrival time difference of two-foot pulse waves of the present application in some embodiments;

[0047] Figure 5 This is a principle block diagram of the optical imaging device of the present application in some embodiments;

[0048] Figure 6 This is a principle block diagram of the terminal device of the present application in some embodiments. DETAILED DESCRIPTION

[0049] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0050] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0051] The terms "exemplary" or "for example" are used to indicate an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0052] The terms "comprises," "comprising," or any other variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0053] Peripheral artery disease (PAD) is a common cardiovascular disease characterized by atherosclerosis of the peripheral arteries. This disease primarily develops in the intima of medium-sized and large arteries and is chronic and progressive. The main pathological features of atherosclerosis include lipid deposition and fibrous proliferation in the arterial intima, ultimately leading to stenosis or complete obstruction of the vessel lumen.

[0054] Unless otherwise defined, all technical and scientific terms used in the specification of this application have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in the specification of this application includes any and all combinations of one or more of the relevant listed items.

[0055] The specific implementation of this application is further described below with reference to the accompanying drawings.

[0056] refer to Figure 1 As shown in FIG, it is a flow chart of a foot sole video evaluation method based on the arrival time difference of the double foot pulse waves provided in the first aspect of the embodiment of the present application. Figure 1 In the method, the foot sole video assessment method based on the arrival time difference of the pulse waves of both feet includes:

[0057] S1. Obtain sole videos of the target user's feet.

[0058] The foot sole video is obtained by recording with a camera of a foot sole video evaluation device based on the arrival time difference of the pulse waves of both feet.

[0059] The foot sole video is an RGB video, which refers to video data encoded and represented using the RGB (red, green, blue) color space.

[0060] In the embodiment of the present application, the size of each frame image in the foot sole video is 480px×300px, where px is a pixel unit. In other embodiments, the size of each frame image in the foot sole video can be set by those skilled in the art according to actual needs.

[0061] S1 step can also refer to Figure 2 Follow the "Acquire Foot Plantar Video" step in the .

[0062] S2. Extract the first pixel time-domain change signal of each pixel point in the sole area of both feet in each frame image according to the sole video, and obtain the pulse wave signal corresponding to each sole area.

[0063] Furthermore, step S2 includes:

[0064] S21 . Generate a first pixel temporal change signal for each sole area based on the time sequence of each frame image in the sole video and the pixel value corresponding to each pixel point in each sole area of each frame image.

[0065] Exemplarily, a first pixel time domain change signal of the left sole area is generated based on the time sequence of each frame image in the sole video and the pixel value corresponding to each pixel point in the left sole area; a first pixel time domain change signal of the right sole area is generated based on the time sequence of each frame image in the sole video and the pixel value corresponding to each pixel point in the right sole area.

[0066] It should be understood that each pixel in a left sole region corresponds to one of the first pixel time-domain variation signals of the left sole region, and the pixel values of multiple pixels at the same position in the left sole region are concatenated to form the first pixel time-domain variation signal of the left sole region. Similarly, each pixel in a right sole region corresponds to one of the first pixel time-domain variation signals of the right sole region, and the pixel values of multiple pixels at the same position in the right sole region are concatenated to form the first pixel time-domain variation signal of the right sole region.

[0067] Furthermore, step S21 includes:

[0068] S211 , after downsampling each frame image in the sole video, extract the sole area of both feet in each frame image, and determine the pixel value corresponding to each pixel point in each sole area.

[0069] Specifically, the step of downsampling each frame image in the plantar video is to perform average pooling processing on each frame image. In the embodiment of the present application, average pooling processing is performed with a pooling window of 50px×50px to achieve downsampling. In other embodiments, the specific size of the pooling window can be set by those skilled in the art according to actual needs.

[0070] It should be understood that the purpose of performing the downsampling operation on the sole video is to reduce the amount of calculation, lower the quantization error, weaken the impact of noise on a single pixel, and enhance the robustness of pulse wave signal extraction.

[0071] In other embodiments, maximum pooling, minimum pooling, super-pixel image segmentation, pixel fusion or linear interpolation can be used instead of the average pooling to perform downsampling to achieve regional pixel fusion. Furthermore, the step of extracting the sole area of the two feet in each frame image can be implemented using a target detection algorithm. The target detection algorithm can be R-CNN (Region-based Convolutional Network, region-based convolutional neural network), Fast R-CNN (Fast Region-based Convolutional Network, fast regional convolutional neural network) or Faster R-CNN (Faster Region-based Convolutional Neural Networks, faster regional convolutional neural network).

[0072] The extraction of the sole area of the two feet in each frame image can refer to Figure 2 The "Select the region of interest on the sole of the foot" step in

[0073] Furthermore, the pixel values corresponding to each pixel point in each plantar region are RGB three-channel pixel values or single-channel pixel values. Because the signal-to-noise ratio, contrast, and sensitivity of the G (Green) channel are superior to those of the R (Red) and B (Blue) channels, the G channel pixel values are selected in each plantar region in this embodiment of the application.

[0074] When the pixel values corresponding to the pixel points in the plantar regions are RGB three-channel pixel values, the multi-channel information can be fused to improve the quality of extracting the first pixel time-domain change signal and, at the same time, enhance the stability of the first pixel time-domain change signal.

[0075] When the pixel value corresponding to each pixel point in each sole area is a single-channel pixel value, the speed of extracting the first pixel time-domain change signal can be increased.

[0076] It should be understood that RGB described herein refers to the three colors red, green, and blue.

[0077] S212 : Generate a first pixel time-domain change signal for each sole area based on the time sequence of each frame image and the pixel value corresponding to each pixel point in each sole area.

[0078] S22 , downsampling the first pixel time-domain variation signal of each plantar region according to a preset first sliding time window and a first step length to obtain a pulse wave signal corresponding to each plantar region.

[0079] In the embodiment of the present application, the length of the first sliding time window is 30 seconds, and the first step length is 5 seconds. In other embodiments, the specific values of the first sliding time window and the first step length can be set by those skilled in the art according to actual needs.

[0080] Furthermore, step S22 includes:

[0081] S221 . Perform division processing on each first pixel time-domain variation signal based on the first sliding time window and the first step length to obtain a plurality of first sub-signals corresponding to each first pixel time-domain variation signal.

[0082] The plurality of first sub-signals are selected from a plurality of first left sub-signals or a plurality of first right sub-signals.

[0083] Specifically, the first target pixel time domain change signal of the left sole area is divided and processed based on the first sliding time window and the first step length to obtain a plurality of first left sub-signals corresponding to the first target pixel time domain change signal of the left sole area; the first target pixel time domain change signal of the right sole area is divided and processed based on the first sliding time window and the first step length to obtain a plurality of first right sub-signals corresponding to the first target pixel time domain change signal of the right sole area; the first target pixel time domain change signal of the left sole area is one of the first pixel time domain change signals of all the first pixel time domain change signals of the left sole area; the first target pixel time domain change signal of the right sole area is one of the first pixel time domain change signals of all the first pixel time domain change signals of the right sole area.

[0084] The first sub-signal is an RGB signal.

[0085] S222: Filter all first sub-signals using a filter to obtain pulse wave signals corresponding to each sole area.

[0086] In the embodiment of the present application, the filter is a Butterworth filter, and the filtering frequency of the Butterworth filter is set to [0.6, 4] Hz to remove noise interference in each first sub-signal. In other embodiments, other filters may be used to replace the Butterworth filter, and the specific setting can be made by those skilled in the art based on actual needs.

[0087] It should be understood that the left sole region includes multiple pulse wave signals, each of which is obtained by filtering the first left sub-signals after dividing a first pixel time-domain variation signal from the left sole region. The right sole region also includes multiple pulse wave signals, each of which is obtained by filtering the first right sub-signals after dividing a first pixel time-domain variation signal from the right sole region.

[0088] When the pixel value corresponding to each pixel point in each plantar area is an RGB three-channel pixel value, the pulse wave signal is a three-channel pulse wave signal; when the pixel value corresponding to each pixel point in each plantar area is a single-channel pixel value, the pulse wave signal is a single-channel pulse wave signal.

[0089] In the embodiment of the present application, the G (Green) channel signal is selected as the used signal.

[0090] S3, based on the pulse wave signals corresponding to each sole area, multiple regions of interest in the sole video are determined to obtain multiple target regions of interest. Step S3 can be understood as Figure 2 The "Extract Multiple Target Regions of Interest" step in .

[0091] Furthermore, step S3 includes:

[0092] S31. Convert the pulse wave signal corresponding to each foot sole area into the frequency domain, and calculate the signal-to-noise ratio of each foot area.

[0093] Specifically, step S31 includes:

[0094] S311 , performing frequency domain conversion processing on each pulse wave signal corresponding to the left sole area using Fourier transform to obtain a left frequency domain signal corresponding to each pulse wave signal in the left sole area.

[0095] S312: Calculate a first signal-to-noise ratio of each left frequency domain signal.

[0096] Further, the The calculation formula for the first signal-to-noise ratio of the left frequency domain signal is: ;in, It is The first signal-to-noise ratio of the left frequency domain signal; It is The signal power of the left frequency domain signal; It is The noise power of the left frequency domain signal; is power; It’s a signal; It's noise.

[0097] S313 , performing frequency domain conversion processing on each pulse wave signal corresponding to the right sole area using Fourier transform to obtain a right frequency domain signal corresponding to each pulse wave signal in the right sole area.

[0098] S314: Calculate a second signal-to-noise ratio of each right frequency domain signal.

[0099] Further, the The calculation formula for the second signal-to-noise ratio of the right frequency domain signal is: ;in, It is The second signal-to-noise ratio of the right frequency domain signal; It is The signal power of the right frequency domain signal; It is The noise power of the right frequency domain signal.

[0100] S315: All the first signal-to-noise ratios and all the second signal-to-noise ratios constitute the signal-to-noise ratio of each foot region.

[0101] S32. Sort the foot regions according to their signal-to-noise ratios, and select the foot sole regions that meet preset conditions as candidate regions of interest.

[0102] In the embodiment of the present application, a preset condition is used to screen candidate regions of interest from all plantar regions in the plantar video. This condition can be used as a preset ratio threshold in the signal-to-noise ratio sorting. Taking the sorting from largest to smallest as an example, the preset ratio threshold can be set to the top 20% or other numerical values; taking the sorting from smallest to largest as an example, the preset ratio threshold can be set to the bottom 20% or other numerical values. This embodiment of the present application does not impose specific limitations. For example, in the sorting from largest to smallest, the plantar regions in the top 20% of the signal-to-noise ratio sorting are selected as candidate regions of interest.

[0103] Specifically, all first signal-to-noise ratios are sorted, and the plantar regions corresponding to the top 20% of all first signal-to-noise ratios are selected as candidate regions of interest; all second signal-to-noise ratios are sorted, and the plantar regions corresponding to the top 20% of all second signal-to-noise ratios are also selected as candidate regions of interest.

[0104] The sorting process is a sorting process from large to small, or a sorting process from small to large. The specific sorting method of the sorting process can be set by those skilled in the art according to actual needs.

[0105] S33: Optimize each candidate ROI to obtain a target ROI corresponding to each candidate ROI.

[0106] Specifically, step S33 includes:

[0107] S331 : Perform upsampling processing on each candidate region of interest to obtain a first region to be processed corresponding to each candidate region of interest.

[0108] The upsampling process is an inverse average pooling process. The inverse average pooling process is used to upsample to the size of the original video stream, that is, the size of the original foot sole video. In other embodiments, the upsampling process can also be other processes, which can be specifically configured by those skilled in the art based on actual needs.

[0109] Furthermore, step S331 includes:

[0110] S3311 , performing upsampling processing on each candidate region of interest to obtain a third region to be processed corresponding to each candidate region of interest.

[0111] S3312: Perform boundary processing on each third region to be processed to obtain a first region to be processed corresponding to each candidate region of interest.

[0112] The boundary processing may be filling with zero values or mirror extension processing. In other implementations, the boundary processing may also be other processing methods, which may be specifically configured by those skilled in the art according to actual needs.

[0113] S332: Perform a closing operation on each first region to be processed to obtain a second region to be processed corresponding to each first region to be processed.

[0114] The closing operation processing refers to performing an expansion and then an erosion operation on each first region to be processed, so as to obtain a second region to be processed corresponding to each first region to be processed.

[0115] The closing operation is used to fill small holes or slits in each first area to be processed, connect adjacent objects, and smooth the boundaries of objects.

[0116] Furthermore, step S332 includes:

[0117] S3321. Determine a first structuring element for the morphological operation and the size of the first structuring element.

[0118] The first structural element may be rectangular, circular, or elliptical. In other embodiments, the first structural element may also be triangular or other shapes, which may be specifically configured by those skilled in the art based on actual needs. The size of the first structural element may also be configured by those skilled in the art based on actual needs.

[0119] S3322: Perform an expansion operation on each first region to be processed using the first structuring element and the size of the first structuring element to obtain a fourth region to be processed corresponding to each first region to be processed.

[0120] S3323. Perform an etching operation on each fourth region to be processed using the first structuring element and the size of the first structuring element to obtain a second region to be processed corresponding to each first region to be processed.

[0121] S333 , performing an opening operation on each second region to be processed to obtain a target region of interest corresponding to each second region to be processed.

[0122] The opening operation refers to performing an erosion followed by an expansion operation on each second region to be processed, thereby obtaining a target region of interest corresponding to each second region to be processed.

[0123] The opening operation is used to remove small objects in each second area to be processed, disconnect small connections between objects, and smooth the boundaries of larger objects.

[0124] Furthermore, step S333 includes:

[0125] S3331. Determine the second structuring element of the morphological operation and the size of the second structuring element.

[0126] The second structural element has the same shape as the first structural element. In other embodiments, the shape of the second structural element may be different from that of the first structural element, which may be specifically set by those skilled in the art based on actual needs. The size of the second structural element may be the same as that of the first structural element. In other embodiments, the size of the second structural element may be different from that of the first structural element, which may be specifically set by those skilled in the art based on actual needs.

[0127] S3332. Perform an etching operation on each second region to be processed using the second structuring element and the size of the second structuring element to obtain a fifth region to be processed corresponding to each second region to be processed.

[0128] S3333: Perform a dilation operation on each fifth region to be processed using the second structuring element and the size of the second structuring element to obtain a target region of interest corresponding to each second region to be processed.

[0129] S4. Extract target rPPG signals corresponding to each foot based on the multiple target regions of interest, and construct a two-foot time difference wave, where the two-foot time difference wave is used to reflect the arrival time difference change of the target rPPG signals corresponding to the two feet.

[0130] Furthermore, step S4 includes:

[0131] S41 , extracting a second pixel time-domain variation signal of each pixel point from multiple target regions of interest in the sole video to obtain an initial rPPG signal corresponding to each foot.

[0132] Furthermore, step S41 includes:

[0133] S411 , generating a second pixel temporal change signal for each sole region based on the time sequence of the multiple target regions of interest and the pixel value corresponding to each pixel point in the multiple target regions of interest in the sole video.

[0134] It should be understood that, among the multiple target regions of interest in the sole video, each pixel in a left sole region corresponds to one of the second pixel time-domain variation signals of the left sole region, and the pixel values of multiple pixels at the same position in the left sole region are concatenated to form the second pixel time-domain variation signal of the left sole region. Similarly, each pixel in a right sole region corresponds to one of the second pixel time-domain variation signals of the right sole region, and the pixel values of multiple pixels at the same position in the right sole region are concatenated to form the second pixel time-domain variation signal of the right sole region.

[0135] The pixel value corresponding to each pixel point in the multiple target regions of interest in the sole video is an RGB three-channel pixel value or a single-channel pixel value.

[0136] S412 , downsampling the second pixel time-domain change signal of each plantar region according to the preset second sliding time window and the second step size to obtain an initial rPPG signal corresponding to each foot.

[0137] It should be understood that the rPPG (Remote Photo Plethysmo Graphy) described herein refers to remote photoplethysmography.

[0138] In an embodiment of the present application, the length of the second sliding time window can be the same as the length of the first sliding time window, and the specific value of the second step size can be the same as the specific value of the first step size. In other embodiments, the specific values of the second sliding time window and the second step size can also be set by those skilled in the art according to actual needs.

[0139] Furthermore, step S412 includes:

[0140] S4121. Perform division processing on each second pixel time-domain variation signal based on the second sliding time window and the second step size to obtain a plurality of second sub-signals corresponding to each second pixel time-domain variation signal.

[0141] The plurality of second sub-signals are selected from a plurality of second left sub-signals or a plurality of second right sub-signals.

[0142] Specifically, the second target pixel time domain change signal of the left sole area is divided and processed based on the second sliding time window and the second step size to obtain a plurality of second left sub-signals corresponding to the second target pixel time domain change signal of the left sole area; the second target pixel time domain change signal of the right sole area is divided and processed based on the second sliding time window and the second step size to obtain a plurality of second right sub-signals corresponding to the second target pixel time domain change signal of the right sole area; the second target pixel time domain change signal of the left sole area is one of the second pixel time domain change signals of all the second pixel time domain change signals of the left sole area; the second target pixel time domain change signal of the right sole area is one of the second pixel time domain change signals of all the second pixel time domain change signals of the right sole area.

[0143] The second sub-signal is an RGB signal.

[0144] S4122: Filter all second sub-signals using a filter to obtain an initial rPPG signal corresponding to each foot.

[0145] The filter is a Butterworth filter, and the filtering frequency of the Butterworth filter is set to [0.6, 4] Hz to remove noise interference in each second sub-signal. In other embodiments, other filters can be used to replace the Butterworth filter. The specific filter can be set by those skilled in the art according to actual needs.

[0146] The initial rPPG signal and the pulse wave signal are of the same type.

[0147] It should be understood that the left plantar region includes multiple initial rPPG signals, each of which is obtained by filtering the multiple second left sub-signals after dividing and processing a second pixel time-domain variation signal from the left plantar region. The right plantar region also includes multiple initial rPPG signals, each of which is obtained by filtering the multiple second right sub-signals after dividing and processing a second pixel time-domain variation signal from the right plantar region.

[0148] S42. Calculate the average value of the initial rPPG signals corresponding to each foot to obtain the target rPPG signals corresponding to each foot.

[0149] The average value includes a first average value and a second average value, wherein the first average value is used to indicate the average value of all initial rPPG signals corresponding to the left plantar area at each time point; the second average value is used to indicate the average value of all initial rPPG signals corresponding to the right plantar area at each time point.

[0150] Furthermore, step S42 includes:

[0151] S421. Calculate a first average value of all initial rPPG signals corresponding to the left plantar area at each time point.

[0152] S422. Utilize all first average values to form a target rPPG signal corresponding to the left foot.

[0153] The target rPPG signal corresponding to the left foot can be referred to Figure 3 The signal pointed to by the label L.

[0154] In the embodiment of the present application, the left foot corresponds to only one target rPPG signal.

[0155] S422 steps can refer to Figure 2 The "Extract the target rPPG signal corresponding to the left foot" step in

[0156] S423. Calculate the second average value of all initial rPPG signals corresponding to the right plantar area at each time point.

[0157] S424. All second average values are used to form a target rPPG signal corresponding to the right foot.

[0158] The target rPPG signal corresponding to the right foot can be referred to Figure 3 The signal pointed to by the label R.

[0159] In the embodiment of the present application, the right foot corresponds to only one target rPPG signal.

[0160] S424 steps can refer to Figure 2 The "Extract the target rPPG signal corresponding to the right foot" step in

[0161] S43. Calculate the time difference between the first characteristic point and the second characteristic point of all target rPPG signals using a target algorithm to determine the arrival time differences of multiple bipedal pulse waves.

[0162] The target algorithm may be a peak detection algorithm, a signal cross-correlation algorithm, or a valley detection algorithm, etc., and this embodiment does not impose any specific limitation thereto.

[0163] When the target algorithm is a peak detection algorithm, the first feature point is a first wave peak, and the second feature point is a second wave peak.

[0164] This embodiment of the application takes the peak detection algorithm as an example to illustrate the specific implementation steps of S43, including:

[0165] S431: Divide and process the target rPPG signal corresponding to the left foot based on a preset third sliding time window and a third step size to obtain a plurality of third left sub-signals.

[0166] The specific values of the third sliding time window and the third step size can be set by those skilled in the art according to actual needs.

[0167] Each of the third left sub-signals has only one first peak.

[0168] S432: Divide and process the target rPPG signal corresponding to the right foot based on the third sliding time window and the third step length to obtain a plurality of third right sub-signals.

[0169] Each of the third right sub-signals has only one second peak.

[0170] S433. Use a peak detection algorithm to detect the first peak of the target third left sub-signal and the second peak of the target third right sub-signal, wherein the target third left sub-signal is any one of the multiple third left sub-signals, and the target third right sub-signal is a signal in the multiple third right sub-signals that is at the same time point as the target third left sub-signal.

[0171] S434: Calculate a time difference corresponding to the target third left sub-signal based on the first peak and the second peak.

[0172] The time difference corresponding to the target third left sub-signal is the arrival time difference of the pulse waves of both feet.

[0173] S434 steps can refer to Figure 2 Calculate time difference in the step.

[0174] S435: All the time differences are combined to form the plurality of bipedal pulse wave arrival time differences.

[0175] S44: Generate the bipedal time difference wave according to the multiple bipedal pulse wave arrival time differences.

[0176] Specifically, all arrival time differences of the two foot pulse waves can be connected to form the two-foot time difference wave. The two-foot time difference wave is a curve. In other embodiments, the two-foot time difference wave can also be a straight line, a stepped line segment, a jagged line segment, or other irregular combinations of line segments. The specific configuration can be determined by those skilled in the art based on actual needs.

[0177] S5. Perform evaluation processing based on the two-foot time difference wave to obtain an evaluation result of the sole video.

[0178] Step S5 can refer to Figure 2 The "Evaluate PAD" step in

[0179] Furthermore, step S5 includes:

[0180] S51. Perform feature extraction processing on the bipedal time difference wave to obtain at least one target feature.

[0181] Furthermore, the feature extracted in step S51 is the average value, standard deviation, maximum value and / or minimum value of the bipedal time difference wave. That is, the at least one target feature is the average value, standard deviation, maximum value and / or minimum value.

[0182] Illustratively, in this embodiment, the average value, standard deviation, maximum value, and minimum value may all be input into the classifier for evaluation processing to improve the accuracy of the evaluation result.

[0183] In other implementations, those skilled in the art may set the extracted feature type according to actual needs.

[0184] S52: Input the at least one target feature into a classifier for evaluation processing to obtain an evaluation result of the sole video.

[0185] Medical personnel can assist in diagnosing peripheral arterial disease and the severity of the peripheral arterial disease based on the assessment results.

[0186] The classifier is a supervised learning classifier. Exemplarily, the classifier may be a support vector machine, a random forest, or a multilayer perceptron (MLP).

[0187] In other implementations, the classifier may be replaced with other machine learning models.

[0188] Furthermore, the classifier includes an input layer, a first hidden layer, a second hidden layer and an output layer.

[0189] The specific steps of the classifier evaluating the at least one target feature include:

[0190] S521. After receiving the at least one target feature, the input layer processes the at least one target feature into a column vector to obtain a first eigenvector. The first eigenvector is an m×1 column vector, where m is the number of the at least one target feature. Exemplarily, when the at least one target feature is the mean, standard deviation, maximum, and minimum value of the bipedal time difference wave, the mean value is used as the first row of data of the first eigenvector, the standard deviation is used as the second row of data of the first eigenvector, the maximum value is used as the third row of data of the first eigenvector, and the minimum value is used as the fourth row of data of the first eigenvector.

[0191] S522: Input the first eigenvector into the first hidden layer for feature extraction and nonlinear transformation to obtain a second eigenvector.

[0192] Furthermore, step S522 can be implemented by the following formula:

[0193] ;

[0194] ;

[0195] in, is the result of feature extraction performed by the first hidden layer; is the first weight matrix, representing the connection weight; is the first eigenvector; is the first bias vector, used to adjust the activation threshold of the neuron; is a linear rectification function; is the second eigenvector.

[0196] S523: Input the second eigenvector into the second hidden layer for feature extraction and nonlinear transformation to obtain a third eigenvector.

[0197] Furthermore, step S523 can be implemented by the following formula:

[0198] ;

[0199] ;

[0200] in, is the feature extraction result of the second hidden layer; is the second weight matrix; is the second bias vector; is the third eigenvector; is the hyperbolic tangent function.

[0201] S524: Input the third eigenvector into the output layer for transformation processing to obtain the evaluation result.

[0202] Furthermore, step S524 can be implemented by the following formula:

[0203] ;

[0204] ;

[0205] in, It is the feature extraction result of the output layer; is the third weight matrix; is the third bias vector; is the result of said assessment; It is an S-shaped growth curve and an activation function.

[0206] The core of this embodiment is to use the time difference waves of the target rPPG signals corresponding to the target user's feet to obtain an assessment result, allowing medical personnel to assist in the diagnosis of peripheral arterial disease and the severity of peripheral arterial disease based on the assessment results. Compared with the existing technology, this embodiment does not use historical data as an evaluation factor. Instead, it uses the arrival time difference of the target rPPG signals corresponding to the feet for evaluation. This avoids the situation where the assessment results are affected by outliers in the historical data. This can improve the accuracy of the assessment results and enable medical personnel to accurately assess the condition of lower extremity atherosclerosis.

[0207] In some embodiments, the foot video assessment method based on the arrival time difference of two-foot pulse waves further includes a classifier training step, and the classifier training step includes:

[0208] S6. Obtain a training data set and a true label set, where each training data in the training data set has a one-to-one correspondence with one of the true labels in the true label set.

[0209] Each training data in the training data set is a sole training video of the target user's feet recorded by the camera.

[0210] Each true label in the true label set is a true ultrasonic waveform image.

[0211] The ultrasonic waveform diagram is obtained by a medical ultrasonic instrument.

[0212] S7. Perform feature extraction processing on target training data to obtain training features, where the target training data is any training data in the training data set.

[0213] Specifically, the step S7 includes:

[0214] S71. Extract the first pixel time-domain variation training signal of each pixel point in the plantar region of both feet in each frame image according to the target training data, and obtain the pulse wave training signal corresponding to each plantar region.

[0215] The implementation of step S71 is the same as that of step S2 and will not be repeated here.

[0216] S72. Determine multiple regions of interest in the target training data based on the pulse wave training signals corresponding to each sole area, and obtain multiple target training areas.

[0217] The implementation of step S72 is the same as that of step S3 and will not be repeated here.

[0218] S73. Extract target rPPG training signals corresponding to each foot based on the multiple target training areas, and construct a bi-foot time difference training wave, where the bi-foot time difference training wave is used to reflect changes in arrival time differences of the target rPPG training signals corresponding to the bi-foot.

[0219] The implementation of step S73 is the same as that of step S4 and will not be repeated here.

[0220] S74: Perform feature extraction processing on the bipedal time difference training wave to obtain at least one target training feature.

[0221] The at least one target training feature is an average value, a standard deviation, a maximum value and / or a minimum value of the bipedal time difference training wave.

[0222] S75: Input the at least one target training feature into the original classifier for evaluation processing to obtain a prediction result of the target training data.

[0223] The original classifier is a supervised learning original classifier. Exemplarily, the original classifier can be a support vector machine, a random forest or a multilayer perceptron (MLP).

[0224] S76. Calculate a loss function based on the prediction result and the true label corresponding to the target training data.

[0225] Furthermore, the calculation formula of step S76 is: ;in, is the loss function; is the length of the training data set, that is, the total number of all training data; is the true label corresponding to the target training data, that is, the The true labels corresponding to the training data; is the prediction result, that is, the prediction result corresponding to the target training data.

[0226] In other embodiments, the specific calculation formula of the loss function can be set by those skilled in the art according to actual needs.

[0227] S77. Adjust the parameters of the original classifier based on the loss function to obtain the classifier.

[0228] In some embodiments, the present application uses the time difference between the arrival of the pulse waves of both feet to assess PAD. Similar methods can be further explored to improve diagnostic accuracy by incorporating multiple indices, such as physiological parameters like blood oxygen and blood pressure. Optimizing the ROI (Region of Interest) can also include parameters such as peak deviation to further optimize the ROI and obtain a more appropriate rPPG signal for calculating PATD.

[0229] refer to Figure 4 As shown in FIG, it is a principle block diagram of a foot sole video evaluation device based on the arrival time difference of the double foot pulse waves provided by the second aspect of the embodiment of the present application. Figure 4 The foot video evaluation device 100 based on the arrival time difference of the two-foot pulse wave includes: a terminal device 101, an optical imaging device 102, a fixed bracket 103 and a transparent plate 104, wherein the optical imaging device 102 includes a camera 1021 and an LED fill light 1022. Figure 5 As shown;

[0230] The terminal device 101 includes a processor configured to execute the steps of the foot sole video assessment method based on the arrival time difference of the two-foot pulse waves provided in the first aspect of the embodiment of the present application;

[0231] The transparent plate 104 is placed on the upper part of the fixing bracket 103. When the camera 1021 records the sole video of the target user's feet, the target user's feet are placed on the transparent plate 104, and the LED fill light 1022 is illuminated on the transparent plate 104.

[0232] The optical imaging device 102 is placed inside the fixing bracket 103;

[0233] After recording the sole video, the camera 1021 sends the sole video to the processor.

[0234] The camera 1021 is an RGB camera sensor. Specifically, the camera 1021 can be an IDS UI-3860-CP RGB camera sensor. In other embodiments, the camera 1021 can be any RGB camera sensor that includes an RGB band of 400-700 nm. The camera 1021 is used to capture a video containing continuous image frames of human skin.

[0235] The LED fill light 1022 is an LED light with a continuous broadband spectrum of 400-700nm.

[0236] The optical imaging device 102 , the fixing bracket 103 and the transparent plate 104 form a plantar imager.

[0237] The fixing bracket 103 is made of a steel beam. In other embodiments, other stable and torsion-resistant materials can be used as the manufacturing material of the fixing bracket 103.

[0238] The transparent plate 104 can be a glass plate or a fully transparent acrylic plate. In other embodiments, other transparent material plates can be used to place the feet on top of the fixing bracket 103.

[0239] The plantar video assessment device based on the arrival time difference of the pulse waves of both feet provided in the embodiments of the present application can obtain assessment results for medical personnel to assist in the diagnosis of peripheral arterial disease without the use of additional auxiliary tools such as straps or clips to restrain the target user. Therefore, the embodiments of the present application can improve the comfort of the target user during the assessment process. At the same time, because auxiliary tools such as straps or clips are not used, the embodiments of the present application will not cause the auxiliary tools to fall off the target user and affect the assessment results. Therefore, the embodiments of the present application can also improve the accuracy of the assessment results, allowing medical personnel to accurately assess the condition of lower limb atherosclerosis.

[0240] The embodiment of the present application uses a sole pressurization method to enhance the rPPG signal, so the target user needs to place the sole of the foot on the transparent plate 104. This method greatly reduces the noise interference of the mechanical movement of the sole on the signal, makes the measurement more accurate, and ensures the comfort of the target user during the diagnosis process.

[0241] The embodiments of the present application facilitate and make PAD testing more commonplace, extending the use of PAD testing from hospitals to everyday life. The system of the embodiments of the present application is simple to operate and requires less expertise, making the system's application scenarios more extensive. During the measurement process, the target user only needs to place both feet on the transparent plate 104 for 2 minutes to obtain an algorithm-based assessment of their PAD, making PAD testing universal for home use.

[0242] The third aspect of the embodiment of the present application provides a terminal device, the principle block diagram of the terminal device can be as follows: Figure 6 As shown. The terminal device includes a processor, a memory, a network interface, a display screen and a temperature sensor connected via a system bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a plantar video evaluation method based on the arrival time difference of the pulse waves of both feet is implemented. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor is pre-set inside the terminal device to detect the operating temperature of the internal device.

[0243] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the terminal device to which the solution of the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0244] In some embodiments, an embodiment of the present application provides a terminal device comprising a processor and a memory, the memory being configured to store a computer program, the processor being configured to call and execute the computer program stored in the memory to perform the steps of the method for plantar video assessment based on the arrival time difference of two-foot pulse waves provided in the first aspect of the embodiment of the present application. A fourth aspect of the embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium being configured to store a computer program that causes a computer to perform the steps of the method for plantar video assessment based on the arrival time difference of two-foot pulse waves provided in the first aspect of the embodiment of the present application.

[0245] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0246] The technical features of the above embodiments can be combined without changing the basic principles of this application. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0247] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of patent protection for the present application shall be determined by the appended claims.

Claims

1. A foot video assessment device based on the arrival time difference of two-foot pulse waves, characterized in that: include: Terminal equipment, optical imaging equipment, a fixing bracket and a transparent plate, wherein the optical imaging equipment includes a camera and an LED fill light; The terminal device includes a processor configured to execute a plantar video evaluation method based on a time difference in arrival of two-foot pulse waves. The plantar video evaluation method based on a time difference in arrival of two-foot pulse waves includes: obtaining a plantar video of two feet of a target user; extracting a first pixel time domain change signal of each pixel point in a plantar region of two feet in each frame image according to the plantar video to obtain a pulse wave signal corresponding to each plantar region; determining multiple regions of interest in the plantar video based on the pulse wave signals corresponding to each plantar region to obtain multiple target regions of interest; extracting a target rPPG signal corresponding to each foot based on the multiple target regions of interest, and constructing a two-foot time difference wave, wherein the two-foot time difference wave is used to reflect a change in the arrival time difference of the target rPPG signal corresponding to the two feet; and performing evaluation processing based on the two-foot time difference wave to obtain an evaluation result of the plantar video. Extracting target rPPG signals corresponding to each foot based on the multiple target regions of interest, and constructing a bipedal time difference wave, wherein the bipedal time difference wave is used to reflect changes in arrival time differences of the target rPPG signals corresponding to the two feet, comprises the following steps: extracting a second pixel time domain change signal of each pixel point from the multiple target regions of interest in the sole video to obtain an initial rPPG signal corresponding to each foot; calculating an average value of the initial rPPG signals corresponding to each foot to obtain a target rPPG signal corresponding to each foot; calculating the time difference between a first feature point and a second feature point between all target rPPG signals using a target algorithm to determine multiple bipedal pulse wave arrival time differences; and generating the bipedal time difference wave based on the multiple bipedal pulse wave arrival time differences. In the case where the target algorithm is a peak detection algorithm, the target algorithm is used to calculate the time difference between the first characteristic point and the second characteristic point between all target rPPG signals, and the step of determining the arrival time differences of multiple bipedal pulse waves includes: dividing the target rPPG signal corresponding to the left foot based on a preset third sliding time window and a third step size to obtain multiple third left sub-signals; dividing the target rPPG signal corresponding to the right foot based on the third sliding time window and the third step size to obtain multiple third right sub-signals; detecting a first peak of the target third left sub-signal and a second peak of the target third right sub-signal using the peak detection algorithm, wherein the target third left sub-signal is any one of the multiple third left sub-signals, and the target third right sub-signal is a signal at the same time point as the target third left sub-signal in the multiple third right sub-signals; and calculating the time difference corresponding to the target third left sub-signal based on the first peak and the second peak. The transparent plate is placed on the upper part of the fixed bracket. When the camera records the sole video of the target user's feet, the target user's feet are placed on the transparent plate, and the LED fill light is irradiated onto the transparent plate. The optical imaging device is placed inside the fixed bracket; After recording the sole video, the camera sends the sole video to the processor.

2. The plantar video evaluation device based on the arrival time difference of the two-foot pulse waves according to claim 1 is characterized in that: The steps of extracting the first pixel time domain change signal of each pixel point in the sole area of both feet in each frame image according to the sole video to obtain the pulse wave signal corresponding to each sole area include: generating a first pixel time-domain change signal for each sole region based on a time sequence of each frame image in the sole video and a pixel value corresponding to each pixel point in each sole region of each frame image; The first pixel time domain variation signal of each plantar region is downsampled according to the preset first sliding time window and the first step length to obtain the pulse wave signal corresponding to each plantar region.

3. The plantar video evaluation device based on the arrival time difference of the two-foot pulse waves according to claim 2, characterized in that: The step of performing downsampling processing on the first pixel time domain change signal of each plantar region according to the preset first sliding time window and the first step length to obtain the pulse wave signal corresponding to each plantar region includes: Performing division processing on each first pixel time-domain variation signal based on the first sliding time window and the first step length to obtain a plurality of first sub-signals corresponding to each first pixel time-domain variation signal; A filter is used to filter all the first sub-signals to obtain the pulse wave signal corresponding to each sole area.

4. The plantar video evaluation device based on the arrival time difference of the two-foot pulse waves according to claim 1, characterized in that: The step of determining a plurality of regions of interest in the sole video based on the pulse wave signals corresponding to the respective sole regions, and obtaining the plurality of target regions of interest comprises: The pulse wave signal corresponding to each foot area is converted into the frequency domain, and the signal-to-noise ratio of each foot area is calculated; Sort the foot regions according to their signal-to-noise ratios, and select the plantar regions that meet the preset conditions as candidate regions of interest; Each candidate region of interest is optimized to obtain a target region of interest corresponding to each candidate region of interest.

5. The plantar video evaluation device based on the arrival time difference of the two-foot pulse waves according to claim 1, characterized in that: The step of performing evaluation processing based on the two-foot time difference wave to obtain the evaluation result of the sole video includes: performing feature extraction processing on the bipedal time difference wave to obtain at least one target feature; The at least one target feature is input into a classifier for evaluation processing to obtain an evaluation result of the sole video.

6. A terminal device, characterized in that: include: A processor and a memory, the memory being configured to store a computer program, the processor being configured to call and run the computer program stored in the memory to execute a plantar video evaluation method based on the arrival time difference of two-foot pulse waves, the plantar video evaluation method based on the arrival time difference of two-foot pulse waves comprising: obtaining a plantar video of the target user's two feet; extracting, based on the plantar video, a first pixel time-domain change signal of each pixel point in the plantar region of the two feet of each frame image to obtain a pulse wave signal corresponding to each plantar region; determining, based on the pulse wave signals corresponding to each plantar region, a plurality of regions of interest in the plantar video to obtain a plurality of target regions of interest; extracting, based on the plurality of target regions of interest, a target rPPG signal corresponding to each foot, and constructing a two-foot time difference wave, the two-foot time difference wave being configured to reflect the arrival time difference change of the target rPPG signal corresponding to the two feet; performing evaluation processing based on the two-foot time difference wave to obtain an evaluation result of the plantar video; Extracting target rPPG signals corresponding to each foot based on the multiple target regions of interest, and constructing a bipedal time difference wave, wherein the bipedal time difference wave is used to reflect changes in arrival time differences of the target rPPG signals corresponding to the two feet, comprises the following steps: extracting a second pixel time domain change signal of each pixel point from the multiple target regions of interest in the sole video to obtain an initial rPPG signal corresponding to each foot; calculating an average value of the initial rPPG signals corresponding to each foot to obtain a target rPPG signal corresponding to each foot; calculating the time difference between a first feature point and a second feature point between all target rPPG signals using a target algorithm to determine multiple bipedal pulse wave arrival time differences; and generating the bipedal time difference wave based on the multiple bipedal pulse wave arrival time differences. In the case where the target algorithm is a peak detection algorithm, the target algorithm is used to calculate the time difference between the first feature point and the second feature point between all target rPPG signals, and the step of determining the arrival time difference of multiple bipedal pulse waves includes: dividing the target rPPG signal corresponding to the left foot based on a preset third sliding time window and a third step size to obtain multiple third left sub-signals; dividing the target rPPG signal corresponding to the right foot based on the third sliding time window and the third step size to obtain multiple third right sub-signals; using the peak detection algorithm to detect the first peak of the target third left sub-signal and the second peak of the target third right sub-signal, wherein the target third left sub-signal is any one of the multiple third left sub-signals, and the target third right sub-signal is a signal at the same time point as the target third left sub-signal in the multiple third right sub-signals; and calculating the time difference corresponding to the target third left sub-signal based on the first peak and the second peak.

7. A computer-readable storage medium, characterized in that Used to store a computer program, the computer program causing a computer to execute a plantar video evaluation method based on the arrival time difference of two-foot pulse waves, the plantar video evaluation method based on the arrival time difference of two-foot pulse waves comprising: obtaining a plantar video of the target user's two feet; extracting a first pixel time domain change signal of each pixel point in the plantar region of the two feet of each frame image according to the plantar video to obtain a pulse wave signal corresponding to each plantar region; determining a plurality of regions of interest in the plantar video based on the pulse wave signals corresponding to each plantar region to obtain a plurality of target regions of interest; extracting a target rPPG signal corresponding to each foot based on the plurality of target regions of interest, and constructing a two-foot time difference wave, the two-foot time difference wave being used to reflect the arrival time difference change of the target rPPG signal corresponding to the two feet; performing evaluation processing based on the two-foot time difference wave to obtain an evaluation result of the plantar video; Extracting target rPPG signals corresponding to each foot based on the multiple target regions of interest, and constructing a bipedal time difference wave, wherein the bipedal time difference wave is used to reflect changes in arrival time differences of the target rPPG signals corresponding to the two feet, comprises the following steps: extracting a second pixel time domain change signal of each pixel point from the multiple target regions of interest in the sole video to obtain an initial rPPG signal corresponding to each foot; calculating an average value of the initial rPPG signals corresponding to each foot to obtain a target rPPG signal corresponding to each foot; calculating the time difference between a first feature point and a second feature point between all target rPPG signals using a target algorithm to determine multiple bipedal pulse wave arrival time differences; and generating the bipedal time difference wave based on the multiple bipedal pulse wave arrival time differences. In the case where the target algorithm is a peak detection algorithm, the target algorithm is used to calculate the time difference between the first feature point and the second feature point between all target rPPG signals, and the step of determining the arrival time difference of multiple bipedal pulse waves includes: dividing the target rPPG signal corresponding to the left foot based on a preset third sliding time window and a third step size to obtain multiple third left sub-signals; dividing the target rPPG signal corresponding to the right foot based on the third sliding time window and the third step size to obtain multiple third right sub-signals; using the peak detection algorithm to detect the first peak of the target third left sub-signal and the second peak of the target third right sub-signal, wherein the target third left sub-signal is any one of the multiple third left sub-signals, and the target third right sub-signal is a signal at the same time point as the target third left sub-signal in the multiple third right sub-signals; and calculating the time difference corresponding to the target third left sub-signal based on the first peak and the second peak.

Images