A non-contact vital sign detection system and method based on video images

Abstract

The present invention discloses a non-contact vital sign detection system and method based on video images, belonging to the field of image processing technology. The system of the present invention includes a multi-core heterogeneous microprocessor, and the multi-core heterogeneous microprocessor includes a programmable control system and a programmable logic system; the programmable control system is composed of a microprocessor, and the programmable logic system is composed of a programmable logic circuit; the method of the present invention adopts a multi-core microprocessor to simplify the image and signal processing algorithms and processes involved in non-contact vital sign detection, and parallelizes and accelerates key algorithms with high computational complexity such as face detection required for non-contact vital sign detection, and reduces the algorithm's computing power requirements and resource overhead while maintaining the detection effect, so that it can run quickly and stably in a low-cost embedded platform, reducing cost, volume and energy consumption, and is highly portable, convenient for users to operate and use, and broadens the application scenarios of non-contact vital sign detection products.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a non-contact vital sign detection system and method based on video images. Background Art

[0002] Video-based non-contact vital sign detection is an emerging physiological signal measurement technology that aims to extract signals related to human vital sign information (such as heart rate, breathing, etc.) from the video data of the person being tested, and realize real-time monitoring of human health status. Compared with other monitoring methods, video-based non-contact vital sign detection technology does not require the subject to wear special sensors, can realize unmanned automatic testing, and is suitable for application scenarios such as health monitoring of infants or the elderly.

[0003] The general process of video-based non-contact vital sign detection technology is as follows: 1) Obtain video data and use image processing algorithms to determine the skin (usually facial) area of ​​interest; 2) Calculate the mean of the three color channels of all pixels in the area of ​​interest respectively, and arrange them into three mean data vectors in the order of video frame acquisition, and the vector length is equal to the number of video frames; 3) Perform signal processing on the three color mean vectors to separate the signal components related to vital signs; 4) Use signal processing algorithms such as FFT to calculate the frequency of vital sign signals and other information to complete the detection of human vital sign information. The "image processing algorithm to determine the facial area of ​​interest" involved in step 1) generally adopts a face detection algorithm based on a sliding window, or a skin segmentation algorithm based on a color model; the "signal processing algorithm to separate the vital sign related component information" involved in step 3) generally adopts an algorithm based on an imaging model or a blind source separation algorithm. The performance of step 3) is very dependent on the recognition accuracy of the facial area of ​​interest in step 1). For the sliding window face detection algorithm, its detection accuracy is high, but the amount of calculation is large. Although there are currently optimized algorithms suitable for embedded environments, due to the limitations of its basic principles, the sliding window method must repeatedly read and write memory data, which is prone to memory bandwidth bottlenecks, affecting the final computing performance and limiting its application in embedded platforms. If a fixed area detection method is used, although a large number of complex calculations and memory bandwidth bottlenecks caused by the sliding window can be avoided, the subject must cooperate and remain still, and the application scenario is limited. The skin segmentation algorithm based on the color model has a small amount of calculation and content overhead, and is suitable for running in an embedded environment. However, this method is easily disturbed by the background environment, the detection effect is poor, and the extraction of vital signs signals is affected. It cannot be used in applications with high detection performance requirements. Therefore, the current embedded non-contact vital sign detection method is limited by the detection algorithm principle and hardware operating conditions, and there is a contradiction between detection performance and operating cost.

[0004] The current non-contact vital signs detection technology has a contradiction between detection performance and equipment cost: if a higher detection accuracy is required, complex image and signal processing algorithms are needed, which have a long calculation time and high computing resource requirements. They require large computing devices such as PCs, which are large in size, high in cost, and have poor portability, making them difficult to carry and use. If a simple algorithm is used to reduce computational complexity and resource overhead so that it can run in a low-cost, portable embedded environment, satisfactory detection results cannot be achieved, especially in applications with low user cooperation such as home health care and sports science research. The performance of non-contact vital signs detection equipment using simple algorithms and embedded platforms cannot meet actual requirements.

[0005] In summary, how to design a high-performance, low-cost non-contact vital signs detection method is a technical problem that needs to be solved urgently. It is also an important factor limiting the promotion of non-contact vital signs detection products in daily health testing, smart healthcare, and motion status monitoring.

[0006] The present invention describes a rapid non-contact vital sign detection method based on a parallel heterogeneous embedded platform, which integrates two design methods, software design and integrated circuit design, to realize key algorithms such as "face region of interest extraction" and "vital sign signal component separation" required for non-contact vital sign detection with high performance and high efficiency, so that it can run effectively in the embedded platform, solve the performance and cost contradiction existing in current similar technical solutions, and provide support for the development of high-performance, low-cost, portable non-contact vital sign detection equipment. Summary of the invention

[0007] In view of the above technical problems existing in the prior art, the present invention proposes a non-contact vital sign detection system and method based on video images, which has a reasonable design, overcomes the shortcomings of the prior art, and has good effects.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A non-contact vital sign detection system based on video images includes a multi-core heterogeneous microprocessor, which includes a programmable control system and a programmable logic system; the programmable control system is composed of a microprocessor including an ARM, and the programmable logic system is composed of a programmable logic circuit including an FPGA.

[0010] In addition, the present invention also mentions a non-contact vital sign detection method based on video images, which adopts the non-contact vital sign detection system based on video images as described above, and specifically includes the following steps:

[0011] Step 1: Use a multi-core heterogeneous microprocessor for video image acquisition;

[0012] Step 2: Pixel format conversion;

[0013] Step 3: Image data cache;

[0014] Step 4: Skin area detection: determining whether the skin area is detected;

[0015] If: the judgment result is that the skin area is detected, then execute step 5;

[0016] Or if the judgment result is that no skin area is detected, then execute step 1;

[0017] Step 5: Coarse detection of face area;

[0018] Step 6: face area precision detection; determine whether the face area is detected;

[0019] If: the judgment result is that the face area is detected, execute step 7;

[0020] Or if the result of the judgment is that no face area is detected, then execute step 1;

[0021] Step 7: Obtaining a PPG signal, i.e., a photoplethysmography signal;

[0022] Step 8: Calculate the result and display it.

[0023] Preferably, in step 2, in step 2, a programmable logic system is used to convert pixel data of each frame of video from a 16-bit RGB565 data format to a 24-bit RGB888 data format by formula (1);

[0024]

[0025] Among them, Pix RGB565 Represents pixel data in format, 16 bits per pixel; Pix R8 、Pix G8 、Pix B8 They represent the R, G, and B channel color data after conversion to RGB888 format. The color data of each channel is 8 bits, for a total of 24 bits.

[0026] Preferably, in step 3, the on-chip storage resources of the programmable logic system are divided into two cache parts, named MEM1 and MEM2, respectively, for storing the converted pixel data and the intermediate results during the operation of the detection algorithm;

[0027] MEM1 is a two-dimensional cache, each cache has L lines, marked as Where r = 1, 2, ..., L, represents the row number, and each row of the cache contains K2 storage units, each with a bit width of 24 bits, used to store a pixel data in RGB888 format. The selection of L and K satisfies L×K 2 =W×H;

[0028] The image data is stored in the cache in blocks. The specific process is as follows:

[0029] The input image is divided into L K×K blocks, marked as B 1,1 , B 1,2 ,……,B i,j ,……,B W / K,H / K , where i and j are the row and column numbers of the image block respectively;

[0030] B 1,1 To B 1,W / K Stored to MEM1 to In the 2,1 To B 2,W / K Stored to MEM1 to In the same way, until B H / K,1 To B H / K,W / K Stored to MEM1 to M1L;

[0031] MEM2 is a two-dimensional cache with a storage unit number of W / K×H / K, marked as Where i = 1, 2, ..., W / K, j = 1, 2, ..., H / K, represents the storage unit number, the bit width of each storage unit is 1, and these storage units are used to store the intermediate results of the algorithm operation process;

[0032] After the image data is cached, each row of the MEM1 cache corresponds one-to-one to each K×K block of the original image, and each unit of the MEM2 cache corresponds one-to-one to each K×K block of the original image.

[0033] Preferably, in step 4, the following steps are specifically included:

[0034] Step 4.1: When K lines of image data are read, the 1st to W / Kth lines of the MEM1 cache are full at the same time, and skin area detection is started at this time; the detection process is performed in units of lines of the MEM1 cache, starting from the first line of the MEM1 cache, i.e., M 1,1 The stored pixel data in RGB888 format are read sequentially, and each image pixel is converted from RGB color space to YCrCb color space using formula (2);

[0035] Step 4.2: Using a programmable logic system, transform the floating-point multiplication operation in formula (2) into an integer multiplication + shift operation, as shown in formula (3);

[0036] Step 4.3: After each pixel is converted, use the on-chip trigger resources of the programmable logic system to construct two 8-bit registers to store the converted Cr and Cb components;

[0037]

[0038] Y=(306*Pix R8 +601*Pix G8 +117*Pix B8 )>>10 (3);

[0039] Step 4.4: Use formula (4) to compare the converted color with the threshold value. The pixels within the threshold range are skin pixels. Use the programmable logic system to build four value comparison circuits to judge the values ​​of Cr and Cb. At the same time, use the counter in the programmable logic system to accumulate the number of pixels that meet the threshold range, i.e., skin pixels. When the detection is completed, 1 All K in 1 row 2 When there are pixels, if the accumulated value of the counter is equal to the total number of pixels K 2 If the ratio exceeds the threshold α, the corresponding The unit is set to 1, otherwise Set to 0; and so on, complete the processing of all the remaining data in the MEM1 cache, and assign the corresponding unit in the MEM2 cache to 1; the processed MEM2 cache divides the original image into L K×K areas, where the unit with a value of 1 marks the area block where the skin may exist in the original image, which serves as the basis for face area detection;

[0040]

[0041] Preferably, in step 5, the following steps are specifically included:

[0042] Step 5.1: Use a square window with a side length of A to open the rectangle of size W / K×H / K represented by the MEM2 cache to remove noise including tiny bridges and isolated points. This is equivalent to filtering out a small area in the original image that is similar to the skin color, leaving a larger adjacent area, which is the candidate face area;

[0043] Step 5.2: Use N pixels of size w1×h1, w2×h2, …, w n ×h nA rectangular window of size W / K×H / K represented by MEM2 is used for sliding summation. The number of cells with a storage value of 1 in the rectangular area covered by the window during sliding is calculated in turn. The window position with a value exceeding the threshold β and with the largest value is taken as the final coarsely detected face area.

[0044] Preferably, in step 5, the module for rough detection of the face area includes a data reading submodule, a sliding summation submodule and a result output submodule;

[0045] The data readout submodule includes W / K memory readout circuits. Each readout circuit reads the address and reads the valid signal under the action of the input clock signal. The clock signal is a square wave with a duty cycle of 50% and a frequency of f cp The read address is a binary signal with a bit width of log2(H / K) that increases by one with the clock signal, and the read valid signal is a 1-bit low-level signal; the W / K data read in parallel by the data reading submodule are sent to the sliding summation submodule;

[0046] The sliding summation submodule is divided into N groups according to the number of sliding windows N and performs simultaneously. The working principle of each group is the same. The working process of the first group is as follows:

[0047] For the first group of sliding window summation submodules, the corresponding sliding window size is w1×h1. The first group of summation modules first constructs (W / K)-w1+1 tree adder groups. Each adder group can simultaneously calculate the sum of w1 rows of data in one working clock cycle. After every h1 working clock cycles, one adder group completes the summation of the number of storage cells with a value of 1 within a sliding window area, and stores the sum results of the (W / K)-w1+1 adder groups in (W / K)-w1+1 registers, respectively, which are recorded as The corresponding sliding window position is stored in another set of registers, recorded as When the summation module completes the summation of the number of storage cells with a value of 1 under the next sliding window area, the new summation result is compared with the previous sliding window summation result, and the larger result is used to update and Similarly, after H / K-h1 working clock cycles, the sliding window summation submodule completes the number of storage cells with an area of ​​w1×h1 and a value of 1 in MEM2, and then obtains (W / K)-w1+1 result registers through the comparator designed in the programmable logic system one of the And its corresponding sliding window position register Complete a set of sliding window summation calculations; finally, send the summation values ​​obtained by the N sets of sliding window summation submodules to the result output submodule;

[0048] The result output submodule makes the final output selection through multiple judgments: judge whether the sum value of the first group is greater than the threshold β; if the sum value of the first group is greater than the threshold β, the sliding window position and window size corresponding to the group are output as the structure, otherwise, continue to judge the sum value of the next group, if the sum value of the next group is greater than the threshold, return the result; if the sum results of all groups are less than the threshold, return the sliding window position and window size corresponding to the last group of sum results;

[0049] The result output submodule includes a multi-level comparator, a priority encoder and an N-to-1 data selector;

[0050] The priority encoder receives the output results of all comparators, and its encoding rules are as follows:

[0051] If the output of the first comparator is 1, the code value "1" is generated regardless of the output of the subsequent comparators; if the output of the first comparator is 0 and the output of the second comparator is 1, the code value "2" is generated regardless of the output of the subsequent comparators; and so on, if only the output of the last comparator is 1, the code value "N" is generated;

[0052] The N-to-1 data selector uses the output value of the priority encoder as a selection signal to select which group of summing registers and sliding window position registers among the N groups of sliding summing submodules are finally output.

[0053] Preferably, in step 6, the following steps are specifically included:

[0054] Step 6.1: Using the window position returned by the rough detection as the detection area, use the face detection method to perform accurate detection of the face area;

[0055] Step 6.2: Detect and output the four boundary coordinates of the upper left, upper right, lower left and lower right of the face area through a programmable control system;

[0056] Step 6.3: Use a fixed-shape template to remove the eyes, nose, mouth, and chin from the face, leaving only the forehead and cheeks as the effective area for detection.

[0057] Preferably, in step 7, the following steps are specifically included:

[0058] Step 7.1: Calculate the pixel mean of the green channel based on the pixels of the forehead and cheek area of ​​the face obtained after fine detection, and arrange them in the time sequence of the video frames to obtain a pixel mean vector G mean =[G1,G2,…,G T ], where T represents the number of video frames;

[0059] Step 7.2: Use a cutoff frequency of fL ~f H The bandpass filter is applied to the pixel mean data vector G mean Perform bandpass filtering to obtain the filtered green channel pixel mean vector

[0060] Preferably, in step 8, the following steps are specifically included:

[0061] Step 8.1: Select pixel mean vector There are p consecutive elements in the , so that their number is an integer power of 2;

[0062] Step 8.2: Perform fast Fourier transform on these pixels and obtain the corresponding spectrum distribution, and find the number of the maximum amplitude component in the spectrum. max , calculate the heart rate HR according to formula (5);

[0063]

[0064] Among them, f s Represents the sampling frequency of Fourier transform, that is, the frame rate of the video.

[0065] Beneficial technical effects brought by the present invention:

[0066] In view of the contradiction between detection performance and equipment cost in current non-contact vital sign detection technology, the present invention designs a fast non-contact vital sign detection method based on a parallel heterogeneous embedded platform, adopts a multi-core microprocessor, simplifies the image and signal processing algorithms and processes involved in non-contact vital sign detection from the perspective of algorithm principle and implementation method, realizes non-contact vital sign detection with high performance and high efficiency, improves detection accuracy, reduces the complexity and resource overhead of the detection method, enables it to run quickly and stably in a low-cost embedded platform, broadens the application scenarios of non-contact vital sign detection products, and is expected to create considerable economic value;

[0067] Compared with the existing non-contact vital sign detection technology, the technical solution disclosed in the present invention adopts a parallel heterogeneous embedded microprocessor to parallelize and accelerate key algorithms with high computational complexity such as face detection required for non-contact vital sign detection. Under the premise of maintaining the detection effect, the algorithm's computing power requirements and resource overhead are reduced, so that it can run quickly and stably in a low-cost embedded platform, greatly reducing the cost, volume and energy consumption of the equipment. It is also highly portable and can be carried around, making it easy for users to operate and use.

[0068] The disclosed non-contact vital sign detection method uses programmable logic devices and hardware circuit acceleration technology to realize key algorithms such as image pixel format conversion, skin area segmentation, and rough detection of face areas. It has high parallelism and running speed. Compared with the face detection technology based on software, it has higher real-time performance and lower computing overhead, which is conducive to the realization of non-contact vital sign detection in a low-cost embedded platform. Color space transformation and threshold segmentation algorithm are used for skin detection. On the one hand, the robustness to lighting conditions is improved to ensure the accuracy and stability of skin detection results; on the other hand, the computational complexity is reduced in principle, which is convenient for running the algorithm in an embedded platform. The idea of ​​block processing is adopted to divide the original high-resolution image into several small areas, and the rough detection of the face area is performed in combination with the skin segmentation method, which greatly reduces the computational complexity of the detection of the facial area of ​​interest. On this basis, the precise detection of the face area within a limited range is performed, which can ensure the accuracy of detection while compressing computing resources, and achieve simultaneous improvement of performance and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a flow chart of the non-contact vital sign detection method of the present invention;

[0070] Figure 2 is a schematic diagram of a pixel data format conversion circuit;

[0071] Figure 3 This is a schematic diagram of the data caching process;

[0072] Figure 4 This is the circuit structure diagram for realizing color space conversion;

[0073] Figure 5 This is the circuit structure diagram for implementing the skin segmentation algorithm;

[0074] Figure 6 This is a diagram of the implementation process of rough face area detection. DETAILED DESCRIPTION

[0075] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0076] Embodiment 1:

[0077] A non-contact vital sign detection system based on video images, including a multi-core heterogeneous microprocessor, which is the core controller. The processor contains two control cores: a programmable control system (hereinafter referred to as the PS part), and a programmable logic system (hereinafter referred to as the PL part). The PS part can be composed of a microprocessor such as ARM, and the software is written in C\C++ language to implement user functions. The PL part can be composed of programmable logic circuits such as FPGA, and the hardware circuit is designed using hardware description languages ​​such as Verilog to implement user functions. This application uses hardware description language to write the PL-side hardware design code, and uses the PL side to control a small CMOS digital camera. The video pixel resolution is W×H, and the frame rate is f s The data format of the pixel output by the CMOS digital camera is RGB565, that is, in each pixel, 5 bits represent red (R), 6 bits represent green (G), and 5 bits represent blue (B).

[0078] Embodiment 2:

[0079] Based on the above embodiment 1, the present invention also provides a non-contact vital sign detection method based on video images (the process is as follows: Figure 1 ), specifically including the following steps:

[0080] Step 1: Video image acquisition;

[0081] The ZYNQ series Z7020 multi-core heterogeneous microprocessor is used as the core controller. The PS part of the processor is Cortex-A9 ARM, and the PL part is a programmable logic device equivalent to the Artix-7 series FPGA. The video pixel resolution is 640×320, and the frame rate is f s is 30Hz.

[0082] Step 2: Pixel format conversion;

[0083] Using the PL side, the pixel data of each frame of video is converted from the 16-bit RGB565 data format to the 24-bit RGB888 data format using formula (1).

[0084]

[0085] Among them, Pix RGB565 Format pixel data (16 bits per pixel), Pix R8 、Pix G8 、Pix B8They represent the R, G, and B color data after conversion to RGB888 format. The color data of each channel is 8 bits, totaling 24 bits. Formula (1) is implemented in hardware and programmed on the PL side using hardware description language. The specific circuit form is as follows Figure 2 shown.

[0086] Step 3: Image data cache;

[0087] The on-chip storage resources of the PL part are used to form two caches, named MEM1 and MEM2, which are used to store the converted pixel data and the intermediate results of the detection algorithm.

[0088] MEM1 is a two-dimensional cache with 200 lines per cache, marked as Where r = 1, 2, ..., 200, represents the row number, each row of the cache contains 32 × 32 = 1024 storage units, each storage unit has a bit width of 24 bits, which is used to store a pixel data in RGB888 format. The data volume of the cache is 200 × 1024 × 24 bits = 640 × 320 × 3 bytes = 600 Kbyte. The image data is stored in the cache in blocks. The specific process is as follows: the input image is divided into 200 32 × 32 blocks, marked as B 1,1 , B 1,2 ,……,B i,j ,……,B 10,20 , where i and j are the row and column numbers of the image block respectively. 1,1 To B 1,20 Stored to MEM1 to In the 2,1 To B 2,20 Stored to MEM1 to In this way, until B 10,1 To B 10,20 Stored to MEM1 to The process is as follows Figure 3 shown.

[0089] MEM2 is a one-dimensional cache with 10×20 storage units, marked as Among them, i=1,2,...,10, j=1,2,...,20, represents the storage unit serial number, the bit width of each storage unit is 1 (the total capacity is 200 bits), and these storage units are used to store the intermediate results of the algorithm running process.

[0090] After image data caching, each row of the MEM1 cache corresponds to each 32×32 block of the original image, and each unit of the MEM2 cache corresponds to each 32×32 block of the original image. This processing method organizes the input image data in a specially designed form, which facilitates the subsequent face area detection algorithm to quickly read data, improve the efficiency of storage access and the speed of algorithm operation.

[0091] Step 4: Skin area detection;

[0092] When K lines of image data are read, the 1st to W / Kth lines of the MEM1 cache are filled at the same time, and the skin area detection operation is started. The detection process is performed in units of lines of the MEM1 cache, starting with the first line of the MEM1 cache (i.e., M 1,1 ) reads the stored pixel data in RGB888 format in sequence, and uses formula (2) to convert each image pixel from RGB color space to YCrCb color space. The conversion process of formula (2) is implemented in hardware. In order to facilitate programming design on the PL side using hardware description language, the floating-point multiplication operation in formula (2) is transformed into integer multiplication + shift operation, as shown in formula (3); the subtraction operation in formula (2) is implemented using a complement adder. After each pixel is converted, the on-chip trigger resources on the PL side are used to construct two 8-bit registers to store the converted Cr and Cb components.

[0093] The specific circuit form is as follows Figure 4 After each pixel is converted, two 8-bit registers are constructed using the on-chip trigger resources at the PL end to store the converted Cr and Cb components.

[0094]

[0095] Y=(306*Pix R8 +601*Pix G8 +117*Pix B8 )>>10 (3);

[0096] After that, the converted color is compared with the threshold value using formula (4), and the pixels within the threshold range are skin pixels. The threshold comparison is implemented in hardware, and the hardware description language is used for programming design on the PL side. By building four value comparison circuits, the values ​​of Cr and Cb are judged; the specific circuit form is shown in the attached figure. Figure 5 At the same time, a counter is used to accumulate the number of pixels (i.e., skin pixels) that meet the threshold range. All 32 in the row 2 When there are 32 pixels, if the counter value is equal to the total number of pixels, 2If the ratio exceeds the threshold α=0.5, the corresponding The unit is set to 1, otherwise Set to 0. Similarly, complete the processing of all the remaining data in the MEM1 cache, and the corresponding unit amplitude in the MEM2 cache. The processed MEM2 cache divides the original image into 200 32×32 areas, where the unit with a value of 1 marks the area block where the skin may exist in the original image, which serves as the basis for face area detection.

[0097]

[0098] The main advantages of the above process are twofold. First, the hardware circuit designed using the resources on the PL side realizes the skin detection function, which has a very high running speed and can complete the skin detection synchronously during the image reading process (with only a delay of 32 lines of data reading time), which can shorten the running time of the algorithm in the embedded environment. Second, the image data is stored and processed in 32×32 blocks, which is equivalent to reducing the dimension of the original image by 32×32 times, which can effectively improve the running speed of subsequent face detection operations and further improve the real-time performance of the algorithm in the embedded environment.

[0099] Step 5: Coarse detection of face area;

[0100] First, a square window with a side length of 2 is used to open the 10×20 rectangle represented by the MEM2 cache to remove noise such as tiny bridges and isolated points. This is equivalent to filtering out small areas in the original image that are close to the skin color, leaving a larger adjacent area, which is the face candidate area.

[0101] Afterwards, three rectangular windows with sizes of 2×2, 4×4, and 8×8 are used to perform a sliding summation operation on the 10×20 rectangle represented by MEM2. The area covered by the window during the sliding process is defined as Where n = {1, 2, 3}, corresponding to three different sliding windows, the maximum range of each window is The number of cells with a storage value of 1 in the rectangular area covered by the window sliding process is calculated in sequence, and the window position with a value exceeding the threshold β=0.7 and the largest value is taken as the final coarsely detected face area.

[0102] The above process uses hardware description language to program the PL side, such as Figure 6As shown in the figure, the hardware module of face area rough detection is divided into three parts: data reading submodule, sliding summation submodule, and result output submodule. The data reading submodule is composed of 10 memory reading circuits. Each reading circuit reads the address and reads the valid signal under the action of the input clock signal. The clock signal is a square wave with a duty cycle of 50% and a frequency of f cp , the read address is a binary signal with a bit width of 5 bits, which is incremented by one with the clock signal, and the read valid signal is a 1-bit low-level signal. The 10 data (i.e., a column of the MEM2 cache) read out by the data reading submodule are sent to the sliding summation submodule. The sliding summation submodule is divided into 3 groups and operates simultaneously. The working principle of each group is the same. The working process is explained by taking the first group as an example. For the first group of sliding window summation submodules, the corresponding sliding window size is 2×2. The first group of summation modules first constructs 9 tree adder groups. Each adder group can calculate the sum of 2 rows of data at the same time in one working clock cycle. After every 2 working clock cycles, one adder group completes the sum of all storage units with a value of 1 within a sliding window area (a total of 2×2=4 units), and the sum results of the 9 adder groups are stored in 9 registers (represented as ), the corresponding sliding window position is stored in another set of registers (denoted as ). When the summation module completes the summation of the number of storage cells with a value of 1 under the next sliding window area, the new summation result is compared with the previous sliding window summation result, and the larger result is used to update and Similarly, after 19 working clock cycles, the sliding window summation submodule completes the number of storage cells with a value of 1 in all 2×2 areas in MEM2, and then obtains 9 result registers through the comparator. one of the And its corresponding sliding window position register Complete a set of sliding window summation calculations.

[0103] Finally, the summed values ​​obtained by the three groups of sliding window summation submodules are sent to the result output submodule, and the final output selection is made through multiple judgments: First, it is judged whether the summed value of the first group is greater than the threshold value 0.7. If it is greater, the sliding window position and window size corresponding to this group are output as the structure. Otherwise, the summed value of the next group is judged. If it is greater than the threshold, the result is returned. If the summed results of all groups are less than the threshold, the sliding window position and window size corresponding to the last group of summed results are returned. This module consists of a multi-level comparator, a priority encoder, and a 3-to-1 data selector. The priority encoder receives the output results of all comparators, and its encoding rules are as follows: if the output of the first comparator is 1, no matter what the output of the subsequent comparator is, the encoding value "1" is generated; if the output of the first comparator is 0 and the output of the second comparator is 1, no matter what the output of the subsequent comparator is, the encoding value "2" is generated, and so on. If only the output of the last comparator is 1, the encoding value "3" is generated. The 3-to-1 data selector uses the output value of the priority encoder as a selection signal to select which group of summing registers and sliding window position registers among the N groups of sliding summing submodules are finally output.

[0104] Step 6: Precision detection of face area;

[0105] The window position returned by the rough detection is used as the detection area, and the classic face detection method can be used to accurately detect the face area. This process is based on the microprocessor on the PS side. C\C++ code is written to implement detection methods such as Haar wavelet features, and the four boundary coordinates of the upper left, upper right, lower left, and lower right of the face area are detected and output. After that, a fixed-shape template is used to remove the eyes, nose, mouth, chin, etc. in the face, leaving only the forehead and cheek parts as the effective area for heart rate detection. The image data required for detection is transmitted through the AXI4 internal high-speed bus on the PL side and the PS side.

[0106] The face region detection implemented by the above method can directly give the possible face region in the thumbnail after dimensionality reduction through rough face detection, so there is no need to perform large-scale sliding window operation on the high-resolution original image, and can realize multi-scale simultaneous detection in parallel. In this implementation, for 320×640 images, 32×32 ratio is used for dimensionality reduction. Compared with the traditional sliding window detection algorithm, the overall calculation time is reduced by nearly 1000 times (the theoretical maximum value of acceleration is 32 2 times), which can greatly improve the execution efficiency of the detection algorithm and is suitable for running on embedded devices with limited computing resources and high real-time requirements.

[0107] Step 7: Get the heart rate signal;

[0108] First, based on the pixels of the forehead and cheek areas of the face obtained after precise detection, the pixel mean of the green channel is calculated and arranged in the time sequence of the video frames to obtain a pixel mean vector G mean =[G1,G2,…,G T ], in this implementation method, T = 150. Then, a bandpass filter with a cutoff frequency of 0.7 Hz to 4 Hz is used to filter the pixel mean data vector G mean Perform bandpass filtering to obtain the filtered green channel pixel mean vector

[0109] Step 8: Calculate the heart rate value;

[0110] Select pixel mean vector Perform fast Fourier transform on the 128 consecutive elements in and obtain the corresponding spectrum distribution, and find the number of the largest amplitude component in the spectrum. max , calculate the heart rate HR according to formula (5).

[0111]

[0112] Among them, f s represents the sampling frequency of Fourier transform, in the present invention, f s =30Hz, which is the frame rate of the video.

[0113] The results of heart rate detection can be dynamically displayed using a small LCD touch screen or display.

[0114] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A non-contact vital sign detection method based on video images, characterized in that: The specific steps include: Step 1: Use a multi-core heterogeneous microprocessor for video image acquisition; Step 2: Pixel format conversion; Step 3: Image data cache; Step 4: Skin area detection: determining whether the skin area is detected; If: the judgment result is that the skin area is detected, then execute step 5; Or if the judgment result is that no skin area is detected, then execute step 1; Step 5: Coarse detection of face area; Step 6: face area precision detection; determine whether the face area is detected; If: the judgment result is that the face area is detected, execute step 7; Or if the result of the judgment is that no face area is detected, then execute step 1; Step 7: Obtaining a PPG signal, i.e., a photoplethysmography signal; Step 8: Calculate and display the results; In step 3, the on-chip storage resources of the programmable logic system are divided into two cache parts, named MEM1 and MEM2, respectively, for storing the converted pixel data and the intermediate results during the operation of the detection algorithm; MEM1 is a two-dimensional cache, each cache has L lines, marked as Where r = 1, 2, ..., L, represents the row number, and each row of the cache contains K 2 storage units, each with a bit width of 24 bits, used to store a pixel data in RGB888 format. The selection of L and K satisfies L×K 2 =W×H; The image data is stored in the cache in blocks. The specific process is as follows: The input image is divided into L K×K blocks, marked as B 1,1 , B 1,2 ,……,B i,j ,……,B W / K,H / K , where i and j are the row and column numbers of the image block respectively; B 1,1 To B 1,W / K Stored to MEM1 to In the 2,1 To B 2,W / K Stored to MEM1 to In the same way, until B H / K,1 To B H / K,W / K Stored to MEM1 to middle; MEM2 is a two-dimensional cache with a storage unit number of W / K×H / K, marked as Where i = 1, 2, ..., W / K, j = 1, 2, ..., H / K, represents the storage unit number, the bit width of each storage unit is 1, and these storage units are used to store the intermediate results of the algorithm operation process; After the image data is cached, each row of the MEM1 cache corresponds one-to-one to each K×K block of the original image, and each cell of the MEM2 cache corresponds one-to-one to each K×K block of the original image; The rough detection of the face area in step 5 is implemented by a rough detection module of the face area, and the rough detection module of the face area includes a data reading submodule, a sliding summation submodule and a result output submodule; The data readout submodule includes W / K memory readout circuits. Each readout circuit reads the address and the valid signal under the action of the input clock signal. The clock signal is a square wave with a duty cycle of 50% and a frequency of f cp The read address is a binary signal with a bit width of log2(H / K) that increases by one with the clock signal, and the read valid signal is a 1-bit low-level signal; the W / K data read in parallel by the data reading submodule are sent to the sliding summation submodule; The sliding summation submodule is divided into N groups according to the number of sliding windows N and performs simultaneously. The working principle of each group is the same. The working process of the first group is as follows: For the first group of sliding window summation submodules, the corresponding sliding window size is w1×h1. The first group of summation modules first constructs (W / K)-w1+1 tree adder groups. Each adder group can simultaneously calculate the sum of w1 rows of data in one working clock cycle. After every h1 working clock cycles, one adder group completes the summation of the number of storage cells with a value of 1 within a sliding window area, and stores the summation results of the (W / K)-w1+1 adder groups in (W / K)-w1+1 registers, respectively, which are recorded as The corresponding sliding window position is stored in another set of registers, recorded as When the summation module completes the summation of the number of storage cells with a value of 1 under the next sliding window area, the new summation result is compared with the previous sliding window summation result, and the larger result is used to update and Similarly, after H / K-h1 working clock cycles, the sliding window summation submodule completes the number of storage cells with an area of ​​w1×h1 and a value of 1 in MEM2, and then obtains (W / K)-w1+1 result registers through the comparator designed in the programmable logic system one of the And its corresponding sliding window position register Complete a set of sliding window summation calculations; finally, send the summation values ​​obtained by the N sets of sliding window summation submodules to the result output submodule; represents the i-th result register among (W / K)-w1+1 result registers; represents the i-th position register in (W / K)-w1+1 sliding window position registers, where the value range of i is 1 to (W / K)-w1+1; The result output submodule makes the final output selection through multiple judgments: judge whether the sum value of the first group is greater than the threshold β; if the sum value of the first group is greater than the threshold β, the sliding window position and window size corresponding to the group are output as the structure, otherwise, continue to judge the sum value of the next group, if the sum value of the next group is greater than the threshold, return the result; if the sum results of all groups are less than the threshold, return the sliding window position and window size corresponding to the last group of sum results; The result output submodule includes a multi-level comparator, a priority encoder and an N-to-1 data selector; The priority encoder receives the output results of all comparators, and its encoding rules are as follows: If the output of the first comparator is 1, the code value "1" is generated regardless of the output of the subsequent comparators; if the output of the first comparator is 0 and the output of the second comparator is 1, the code value "2" is generated regardless of the output of the subsequent comparators; and so on, if only the output of the last comparator is 1, the code value "N" is generated; The N-to-1 data selector uses the output value of the priority encoder as a selection signal to select which group of summing registers and sliding window position registers among the N groups of sliding summing submodules are finally output.

2. The non-contact vital sign detection method based on video images according to claim 1 is characterized in that: In step 2, in step 2, using a programmable logic system, the pixel data of each frame of video is converted from a 16-bit RGB565 data format to a 24-bit RGB888 data format through formula (1); Among them, Pix RGB565 Represents pixel data in format, 16 bits per pixel; Pix R8 、Pix G8 、Pix B8 They represent the R, G, and B channel color data after conversion to RGB888 format. The color data of each channel is 8 bits, for a total of 24 bits.

3. The non-contact vital sign detection method based on video images according to claim 1 is characterized in that: In step 4, the following steps are specifically included: Step 4.1: When K lines of image data are read, the 1st to W / Kth lines of the MEM1 cache are full at the same time, and skin area detection is started at this time; the detection process is performed in units of lines of the MEM1 cache, starting from the first line of the MEM1 cache, i.e., M 1 1, sequentially read the stored pixel data in RGB888 format, and use formula (2) to convert each image pixel from RGB color space to YCrCb color space; Step 4.2: Using a programmable logic system, transform the floating-point multiplication operation in formula (2) into an integer multiplication + shift operation, as shown in formula (3); Step 4.3: After each pixel is converted, use the on-chip trigger resources of the programmable logic system to construct two 8-bit registers to store the converted Cr and Cb components; And=(306*Pix R8 +601*Pix G8 +117*Pix B8 )>>10 (3); Step 4.4: Use formula (4) to compare the converted color with the threshold value. The pixels within the threshold range are skin pixels. Use the programmable logic system to build four value comparison circuits to judge the values ​​of Cr and Cb. At the same time, use the counter in the programmable logic system to accumulate the number of pixels that meet the threshold range, i.e., skin pixels. When the detection is completed, All K in the row 2 When there are pixels, if the accumulated value of the counter is equal to the total number of pixels K 2 If the ratio exceeds the threshold α, the corresponding The unit is set to 1, otherwise Set to 0; and so on, complete the processing of all the remaining data in the MEM1 cache, and assign the corresponding unit in the MEM2 cache to 1; the processed MEM2 cache divides the original image into L K×K areas, where the unit with a value of 1 marks the area block where the skin may exist in the original image, which serves as the basis for face area detection; 4. The non-contact vital sign detection method based on video images according to claim 1 is characterized in that: In step 5, the following steps are specifically included: Step 5.1: Use a square window with a side length of A to open the rectangle of size W / K×H / K represented by the MEM2 cache to remove noise including tiny bridges and isolated points. This is equivalent to filtering out a small area in the original image that is similar to the skin color, leaving a larger adjacent area, which is the candidate face area; Step 5.2: Use N pixels of size w1×h1, w2×h2, …, w n ×h n A rectangular window of size W / K×H / K represented by MEM2 is used for sliding summation. The number of cells with a storage value of 1 in the rectangular area covered by the window during sliding is calculated in turn. The window position with a value exceeding the threshold β and with the largest value is taken as the final coarsely detected face area.

5. The non-contact vital sign detection method based on video images according to claim 1, characterized in that: In step 6, the following steps are specifically included: Step 6.1: Using the window position returned by the rough detection as the detection area, use the face detection method to perform accurate detection of the face area; Step 6.2: Detect and output the four boundary coordinates of the upper left, upper right, lower left and lower right of the face area through a programmable control system; Step 6.3: Use a fixed-shape template to remove the eyes, nose, mouth, and chin from the face, leaving only the forehead and cheeks as the effective area for detection.

6. The non-contact vital sign detection method based on video images according to claim 1, characterized in that: In step 7, the following steps are specifically included: Step 7.1: Calculate the pixel mean of the green channel based on the pixels of the forehead and cheek area of ​​the face obtained after fine detection, and arrange them in the time sequence of the video frames to obtain a pixel mean vector G mean =[G1,G2,…,G T ], where T represents the number of video frames; Step 7.2: Use a cutoff frequency of f L ~f H The bandpass filter of the pixel mean data vector G mean Perform bandpass filtering to obtain the filtered green channel pixel mean vector 7. The non-contact vital sign detection method based on video images according to claim 1, characterized in that: In step 8, the following steps are specifically included: Step 8.1: Select the filtered green channel pixel mean vector There are p consecutive elements in the , so that their number is an integer power of 2; Step 8.2: Perform fast Fourier transform on these pixels and obtain the corresponding spectrum distribution, and find the number of the maximum amplitude component in the spectrum. max , calculate the heart rate HR according to formula (5); Among them, f s Represents the sampling frequency of Fourier transform, that is, the frame rate of the video.

Images