Information output method, device, equipment, medium and program product

Abstract

The present invention provides an information output method, device, equipment, medium and program product, which are applied to the field of information processing technology. The information output method includes: photographing the facial area of the object to be measured to obtain multiple frames of images to be processed; obtaining a first pulse wave signal corresponding to a first area and a second pulse wave signal corresponding to a second area based on the multiple frames of images to be processed; performing cross-correlation calculation on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal; splitting the target pulse wave signal to obtain m sub-calculated pulse wave signals, and performing combined calculation on every two adjacent sub-calculated pulse wave signals to obtain m-1 sub-target pulse wave signals; obtaining a heart rate calculation result of the target object based on the m-1 sub-target pulse wave signals, and outputting the heart rate calculation result through a preset terminal; reducing the influence of the abnormality of a single area on the final result, reducing data incoherence, and improving the accuracy of the heart rate calculation result.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and more particularly to an information output method, apparatus, device, medium and program product. Background Art

[0002] The heart rate state is very important for detecting the psychological and physiological states of an individual. Traditional heart rate state measurements are often contact-based, requiring the tester to wear specific devices for measurement. However, contact-based measurement methods are difficult to apply in some scenarios. Therefore, a remote heart rate measurement device can be used to obtain features related to the heart rate state for analysis.

[0003] In the related art, a remote heart rate measurement device often needs to perform band-pass filtering to remove noise in the collected heart rate signal. However, band-pass filtering also filters out some heart rate signals, resulting in low accuracy of the final heart rate calculation result. At the same time, affected by rapid movement and emotional fluctuations, the heart rate waveform will show irregular changes, which will also cause a large error in the heart rate calculation result. Summary of the Invention

[0004] In view of the above problems, the present invention provides an information output method, apparatus, device, medium and program product.

[0005] According to a first aspect of the present invention, an information output method is provided, including: photographing a facial area of a subject to be measured to obtain multiple frames of images to be processed, where the facial area includes a first area and a second area located at different positions; obtaining a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area based on the multiple frames of images to be processed; performing cross-correlation calculation on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal; splitting the target pulse wave signal into m sub-calculated pulse wave signals, and performing combined calculation on every two adjacent sub-calculated pulse wave signals to obtain m - 1 sub-target pulse wave signals, where m is a positive integer greater than 0; obtaining a heart rate calculation result of the target object based on the m - 1 sub-target pulse wave signals, and outputting the heart rate calculation result through a preset terminal.

[0006] According to an embodiment of the present invention, obtaining a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area based on the multiple frames of images to be processed includes: obtaining a first pixel mean value array corresponding to the first area and a second pixel mean value array corresponding to the second area based on the multiple frames of images to be processed; obtaining the first pulse wave signal based on the first pixel mean value array; obtaining the second pulse wave signal based on the second pixel mean value array.

[0007] According to an embodiment of the present invention, obtaining the first pulse wave signal based on the above-mentioned first pixel mean array includes: normalizing the first pixel mean array to obtain a normalized first pixel mean array; separating pixel channels of the normalized first pixel mean array to obtain a preset number of pixel channel mean arrays; using a preset signal processing algorithm to obtain an initial pulse wave signal based on the preset number of pixel channel mean arrays; and using an autocorrelation algorithm to obtain the first pulse wave signal based on the initial pulse wave signal.

[0008] According to an embodiment of the present invention, the multi-frame images to be processed are collected by an imaging device. Obtaining the heart rate calculation result of the target object based on the above-mentioned m-1 sub-target pulse wave signals includes: for a sub-target pulse wave signal, obtaining the number of peaks and the signal duration of the sub-target pulse wave signal; and obtaining the heart rate calculation result of the target object based on the frame rate of the imaging device, the above-mentioned m-1 numbers of peaks, and the above-mentioned m-1 signal durations.

[0009] According to an embodiment of the present invention, obtaining the first pixel mean array corresponding to the first region and the second pixel mean array corresponding to the second region based on the multi-frame images to be processed includes: using a preset key point detection model to extract multi-frame first images to be processed corresponding to the first region and multi-frame second images to be processed corresponding to the second region from the multi-frame images to be processed; obtaining the first pixel mean array based on the multi-frame first images to be processed; and obtaining the second pixel mean array based on the multi-frame second images to be processed.

[0010] According to an embodiment of the present invention, the method further includes: denoising the heart rate calculation result by using a preset interference frame number threshold and a preset amplitude threshold to obtain a target heart rate calculation result.

[0011] The second aspect of the present invention provides an information output device, including: a photographing module for photographing the facial area of the object to be measured to obtain multiple frames of images to be processed, where the facial area includes a first area and a second area located at different positions; a pulse wave signal acquisition module for obtaining a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area based on the multiple frames of images to be processed; a cross-correlation module for performing cross-correlation calculation on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal; a splitting module for splitting the target pulse wave signal into m sub-calculated pulse wave signals, and performing combined calculation on every two adjacent sub-calculated pulse wave signals to obtain m-1 sub-target pulse wave signals, where m is a positive integer greater than 0; an output module for obtaining a heart rate calculation result of the target object based on the m-1 sub-target pulse wave signals and outputting the heart rate calculation result through a preset terminal.

[0012] The third aspect of the present invention provides an electronic device, including: one or more processors; a memory for storing one or more computer programs, where the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0013] The fourth aspect of the present invention further provides a computer-readable storage medium, on which a computer program or instruction is stored, and when the computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0014] The fifth aspect of the present invention further provides a computer program product, including a computer program or instruction, and when the computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0015] According to the embodiments of the present invention, by photographing the first area and the second area of the area of the object to be measured and highlighting the pulse wave waveform based on the multiple self-correlation algorithm, a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area are obtained. Then, cross-correlation processing is performed on the first pulse wave signal and the second pulse wave signal, reducing the influence of a single area anomaly on the final result. Further, the target pulse wave is split into signals, and every two adjacent sub-calculated pulse waves are combined and calculated. Furthermore, the heart rate calculation result is obtained according to the combined sub-target pulse wave data, which can reduce data incoherence, correlate the data before and after, and improve the accuracy of the heart rate calculation result. Description of the Drawings

[0016] Through the following description of the embodiments of the present invention with reference to the drawings, the above content and other objects, features, and advantages of the present invention will become clearer.

[0017] Figure 1Shows an application scenario diagram of an information output method and apparatus according to an embodiment of the present invention.

[0018] Figure 2 Shows a flowchart of an information output method according to an embodiment of the present invention.

[0019] Figure 3 Shows a schematic diagram of non-contact heart rate acquisition principle according to an embodiment of the present invention.

[0020] Figure 4 Shows a schematic diagram of a sliding window according to an embodiment of the present invention.

[0021] Figure 5 Shows a schematic diagram of the principle of obtaining a pulse wave signal based on a facial image according to an embodiment of the present invention.

[0022] Figure 6 Shows a graph of peak heart rate calculation results according to an embodiment of the present invention.

[0023] Figure 7 Shows a graph of key points in a facial area according to an embodiment of the present invention.

[0024] Figure 8 Shows a schematic diagram of the principle of obtaining multiple different facial areas according to an embodiment of the present invention.

[0025] Figure 9 Shows a heart rate measurement diagram of a person to be measured sitting under a tree shade under field sunlight according to an embodiment of the present invention.

[0026] Figure 10 Shows a heart rate measurement diagram of a person to be measured sitting on a step under field sunlight according to an embodiment of the present invention.

[0027] Figure 11 Shows a heart rate measurement diagram of a person to be measured standing under field sunlight according to an embodiment of the present invention.

[0028] Figure 12 Shows a heart rate measurement diagram of a person to be measured sitting in a pavilion under field sunlight according to an embodiment of the present invention.

[0029] Figure 13 Shows a structural block diagram of an information output apparatus according to an embodiment of the present invention.

[0030] Figure 14 Shows a block diagram of an electronic device suitable for implementing an information output method according to an embodiment of the present invention. Detailed implementation manners

[0031] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In the following detailed description, for the sake of explanation, numerous specific details are set forth to provide a comprehensive understanding of the embodiments of the present invention. However, obviously, one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts of the present invention.

[0032] The terms used herein are merely for describing specific embodiments and are not intended to limit the present invention. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0033] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0034] In the case of using expressions such as "at least one of A, B, and C, etc.", generally, it should be interpreted according to the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include, but not be limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0035] The heart rate state is very important for monitoring a person's mental and physical conditions. At the same time, heart rate data can also be used for emotional state analysis. Traditional heart rate state measurements are all contact-based, such as electrode electrocardiogram and photoplethysmogram and other heart rate measurement methods. Although photoplethysmogram is more convenient than electrode electrocardiogram, it also requires contact for measurement. However, in some special cases, wearable devices are prone to cause skin infections. Thus, remote heart rate measurement has emerged, which analyzes features related to heart rate obtained through non-contact devices such as cameras, radars, and other sensors to obtain the heart rate state.

[0036] In related technologies, remote heart rate measurement devices focus on the processing of pulse wave signals. For the extracted pulse wave signals, detrending, band-pass filtering, or wavelet transform is performed to remove interference in the noise signals, and the pulse wave signals are framed by state to calculate the heart rate state by obtaining the number of peaks through fast Fourier transform. Although the final result of this solution is relatively stable, since band-pass filtering directly removes signals outside the normal heart rate range, it means that some abnormal heart rate signals are also filtered out. At the same time, in the case of rapid movement or drastic emotional fluctuations, the heart rate state will not be a good frequency waveform, which will also lead to large errors in the calculation results.

[0037] In view of this, embodiments of the present invention provide an information output method, which includes: photographing the facial area of a subject to be measured to obtain multiple frames of images to be processed, where the facial area includes a first area and a second area located at different positions in the facial area; obtaining a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area based on the multiple frames of images to be processed, and the first pulse wave signal and the second pulse wave signal are subjected to multiple autocorrelation processes; performing cross-correlation calculation on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal; splitting the target pulse wave signal into m sub-calculated pulse wave signals, and performing combined calculation on every two adjacent sub-calculated pulse wave signals to obtain m - 1 sub-target pulse wave signals, where m is a positive integer greater than 0; obtaining a heart rate calculation result of the target object based on the m - 1 sub-target pulse wave signals, and outputting the heart rate calculation result to a preset terminal.

[0038] Figure 1 The application scenario diagram of the information output method and device according to the embodiment of the present invention is shown.

[0039] As Figure 1 shown, the application scenario according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0040] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0041] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop computers, desktop computers, and the like.

[0042] The server 105 can be a server that provides various services. For example, it can be a background management server (only for example) that supports the websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server can analyze and process data such as user requests received, and feedback the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0043] It should be noted that the information output method provided in the embodiments of the present invention can generally be executed by the server 105. Correspondingly, the information output device provided in the embodiments of the present invention can generally be set in the server 105. The information output method provided in the embodiments of the present invention can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the information output device provided in the embodiments of the present invention can also be set in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0044] It should be understood that Figure 1 the numbers of the terminal devices, the network, and the server in

[0045] are merely illustrative. According to the implementation requirements, there can be any number of terminal devices, networks, and servers.

[0045] Based on the scenario described below Figure 1 through Figures 2 to 12 the information output method of the embodiments of the present invention will be described in detail.

[0046] Figure 2 shows a flowchart of the information output method according to the embodiments of the present invention.

[0047] As Figure 2 shown, the information output method of this embodiment includes operations S210 to S250.

[0048] In operation S210, the facial area of the object to be measured is photographed to obtain multiple frames of images to be processed.

[0049] Among them, the facial area includes a first area and a second area located at different positions.

[0050] In operation S220, a first pulse wave signal corresponding to a first region and a second pulse wave signal corresponding to a second region are obtained based on multiple frames of images to be processed.

[0051] Among them, the first pulse wave signal and the second pulse wave signal are subjected to multiple autocorrelation processes.

[0052] Figure 3 The schematic diagram of non-contact heart rate acquisition according to an embodiment of the present invention is shown.

[0053] As Figure 3 shown, there are specular reflection and diffuse reflection when the light source irradiates the skin. The video acquisition device receives the change in the skin surface pixel value caused by the venous artery contraction caused by the heartbeat, and then extracts the features of the diffuse reflection containing the change features to calculate the heart rate signal.

[0054] According to an embodiment of the present invention, the heart rate can be estimated by analyzing the minute changes in the skin color through video-based remote photoplethysmography, obtaining a first candidate pulse wave signal corresponding to a first region and a second candidate pulse wave signal corresponding to a second region. The first candidate pulse wave signal is subjected to multiple autocorrelations to obtain the above-mentioned first pulse wave signal, and the second candidate pulse wave signal is subjected to multiple autocorrelations to obtain the above-mentioned second pulse wave signal. The autocorrelation calculation of the above-mentioned first candidate pulse wave signal can be performed by the following formula (1), or the autocorrelation calculation of the above-mentioned second candidate pulse wave signal can be performed by using the following formula (1). As shown in formula (1):

[0055] (1);

[0056] Among them, is the pulse wave signal obtained after autocorrelation, n is the shift amount of the candidate pulse wave signal, m is the summation index variable. When using the above formula (1) to calculate the first pulse wave signal, x is the first candidate pulse wave signal; when using the above formula (1) to calculate the second pulse wave signal, x is the second candidate pulse wave signal.

[0057] In operation S230, the cross-correlation calculation is performed on the first pulse wave signal and the second pulse wave signal to obtain the target pulse wave signal.

[0058] The cross-correlation calculation of the first pulse wave signal and the second pulse wave signal can be performed by the following formula (2).

[0059] (2);

[0060] Among them, represents the target pulse wave signal, is the summation index variable, x represents the first pulse wave signal, and y represents the second pulse wave signal. It represents the time delay amount.

[0061] In operation S240, the target pulse wave signal is split into m sub-computed pulse wave signals, and for every two adjacent sub-computed pulse wave signals, a combined calculation is performed to obtain m - 1 sub-target pulse wave signals.

[0062] Where m is a positive integer greater than 0.

[0063] Figure 4 It shows a schematic diagram of a sliding window according to an embodiment of the present invention.

[0064] As Figure 4 shown, splitting the target pulse signal can obtain 4 sub-computed pulse wave signals, namely sub-computed pulse wave signal 1, sub-computed pulse wave signal 2, sub-computed pulse wave signal 3, and sub-computed pulse wave signal 4. A combined calculation is performed on two adjacent sub-computed pulse wave signals, that is, sub-computed pulse wave signal 1 and sub-computed pulse wave signal 2 are combined into sub-target pulse wave signal A, sub-computed pulse wave signal 2 and sub-computed pulse wave signal 3 are combined into sub-target pulse wave signal B, and sub-computed pulse wave signal 3 and sub-computed pulse wave signal 4 are combined into sub-target pulse wave signal C.

[0065] According to an embodiment of the present invention, when the frame rate of the acquisition device is 30 fps (Frames Per Second), 20 s of the target pulse wave signal can be stably selected for splitting and calculation. By splitting and reorganizing the target pulse wave signal, when calculating the heart rate calculation result, the time length during which the facial pixel values collected due to severe jitter or light change change drastically can be reduced, and the anomaly is limited to a single sub-computed pulse wave, thereby reducing the probability of abnormal results.

[0066] In operation S250, a heart rate calculation result of the target object is obtained based on the m - 1 sub-target pulse wave signals, and the heart rate calculation result is output through a preset terminal.

[0067] It should be noted that the heart rate calculation result obtained according to the embodiment of the present invention is only a reference intermediate value result, rather than a medical diagnosis result.

[0068] According to an embodiment of the present invention, by photographing a first region and a second region of a region of an object to be measured, and highlighting a pulse wave waveform based on a multiple autocorrelation algorithm, a first pulse wave signal corresponding to the first region and a second pulse wave signal corresponding to the second region are obtained. Then, cross-correlation processing is performed on the first pulse wave signal and the second pulse wave signal, reducing the influence of a single-region anomaly on the final result. Further, the target pulse wave is signal-split, and adjacent two sub-calculated pulse waves are combined for calculation. Furthermore, a heart rate calculation result is obtained based on the combined sub-target pulse wave data, which can reduce data incoherence, correlate the data before and after, and improve the accuracy of the heart rate calculation result.

[0069] According to an embodiment of the present invention, obtaining the first pulse wave signal corresponding to the first region and the second pulse wave signal corresponding to the second region based on multiple frames of images to be processed includes: obtaining a first pixel mean value array corresponding to the first region and a second pixel mean value array corresponding to the second region based on the multiple frames of images to be processed; obtaining the first pulse wave signal based on the first pixel mean value array; and obtaining the second pulse wave signal based on the second pixel mean value array.

[0070] According to an embodiment of the present invention, obtaining the first pulse wave signal based on the first pixel mean value array includes: normalizing the first pixel mean value array to obtain a normalized first pixel mean value array; separating pixel channels of the normalized first pixel mean value array to obtain a preset number of pixel channel mean value arrays; using a preset signal processing algorithm to obtain an initial pulse wave signal based on the preset number of pixel channel mean value arrays; and using an autocorrelation algorithm to obtain the first pulse wave signal based on the initial pulse wave signal.

[0071] Figure 5 A schematic diagram of the principle of obtaining a pulse wave signal based on a facial image according to an embodiment of the present invention is shown.

[0072] As Figure 5 shown, where R n (t) represents the red pulse wave sub-signal obtained by color channel separation through the first pixel mean value, G n (t) represents the green pulse wave sub-signal obtained by color channel separation through the first pixel mean value, B n (t) represents the blue pulse wave sub-signal obtained by color channel separation through the first pixel mean value. The middle three-dimensional graph indicates the distribution of pulse intensity, which is obtained by mapping the signals of the three channels into a three-dimensional space through remote photoplethysmography. Each axis represents the signal intensity of a color channel. z1, z2, and z3 are feature vectors extracted from the sphere, which are used to capture the main changes of the pulse wave signal. s1(t), s2(t), and s3(t) represent the obtained initial pulse wave signals.

[0073] According to an embodiment of the present invention, the above preset signal processing algorithm may be, for example, the rPPG (Remote Photoplethysmography) algorithm.

[0074] According to an embodiment of the present invention, by extracting the pixel mean array of a specific region from multiple frames of images, performing normalization and pixel separation, the change of the pulse signal can be captured from different angles, thereby improving the accuracy of analysis. Further, by using the preset signal processing algorithm to reduce the pixels to a three-channel pixel channel mean array and performing feature extraction to obtain the initial pulse wave signal, the processing flow of the pulse wave signal can be simplified.

[0075] According to an embodiment of the present invention, the multiple frames of images to be processed are collected by a camera device, and the heart rate calculation result of the target object is obtained based on m - 1 sub-target pulse wave signals, including: for a sub-target pulse wave signal, obtaining the number of peaks and the signal duration of the sub-target pulse wave signal; and obtaining the heart rate calculation result of the target object based on the frame rate of the camera device, the m - 1 numbers of peaks and the m - 1 signal durations.

[0076] The heart rate calculation result of the target object can be calculated by the following formula (3).

[0077] (3);

[0078] Wherein, represents the heart rate calculation result of the target object, represents the frame rate of the shooting device, n represents the number of peaks, and L represents the signal duration.

[0079] According to an embodiment of the present invention, the above method further includes: using a preset interference frame number threshold and a preset amplitude threshold to denoise the heart rate calculation result to obtain the target heart rate calculation result.

[0080] Figure 6 Fig. shows the peak heart rate calculation result diagram according to an embodiment of the present invention.

[0081] According to an embodiment of the present disclosure, in the normal heart beat state, it is 40 - 150 BPM (Beats Per Minute), then the interval between every two heart beats is 0.4 s - 1.5 s. For a 30 fps camera, the corresponding data frames are 12 - 45 frames. The preset interference frame number threshold can be set to 12 frames to remove the interference peaks. The peak heart rate calculation result diagram obtained after removing the interference peaks is as shown in Figure 6 shown, Figure 6 where the abscissa and ordinate are not shown, Figure 6 only for showing the waveform shape, and not strictly indicating the waveform size.

[0082] According to an embodiment of the present invention, the target heart rate calculation result can be obtained by screening the peaks based on the heart rate property. For the sub-target pulse wave signal, the heartbeat amplitude is greater than the interference wave amplitude. The preset amplitude threshold can be set as the upper limit of the interference wave amplitude, so as to screen out the peak data that does not meet the conditions by setting the preset amplitude threshold, and finally calculate the target heart rate calculation result.

[0083] According to an embodiment of the present invention, obtaining the first pixel mean array corresponding to the first region and the second pixel mean array corresponding to the second region based on multiple frames of images to be processed includes: using a preset key point detection model to extract multiple frames of first images to be processed corresponding to the first region and multiple frames of second images to be processed corresponding to the second region from the multiple frames of images to be processed; obtaining the first pixel mean array based on the multiple frames of first images to be processed; obtaining the second pixel mean array based on the multiple frames of second images to be processed.

[0084] According to an embodiment of the present invention, key points of the facial region can be collected through a computer vision application tool (i.e., the preset key point detection model). The above computer vision application tool collects 468 key points on the face, and accurately frames the facial region through the key points to obtain regions such as the lower face, head, left face, and right face. Among them, the large region contains more key points, which can include [266, 352, 433, 364, 379, 400, 152, 150, 172, 137, 123, 36] and other key points. The small region is a small region that does not include regions such as the mouth and nose. The above first region can be selected as the lower face region, and the second region can be selected as the right face, which mainly contains [109, 10, 338, 337, 151, 108] and other key points, which can make the obtained results avoid the interference of activities such as breathing and speaking.

[0085] Figure 7 Shows the key point map of the facial region according to an embodiment of the present invention. Figure 8 Shows the schematic diagram of obtaining multiple different facial regions according to an embodiment of the present invention.

[0086] It is possible to obtain Figure 7 the key points located in different regions in Figure 8 the images of multiple different facial regions in Figure 7 Including: The full face image 810 can be obtained according to the global key points in Figure 7 the chin image 820 can be obtained according to the mouth key points and chin key points in Figure 7 the lower face image 830 can be obtained according to the nose key points and chin key points in Figure 7 the right face image 840 can be obtained according to the right corner of the mouth key point and the right chin key point in Figure 7The key point of the left corner of the mouth and the key point of the left chin are used to obtain the left face image 850.

[0087] According to the embodiments of the present invention, for different scenarios and different regions, different combinations can be selected for the first region and the second region. When measuring for a long time and there are changes in the external light, more expressions can be obtained from the lower face and the right face. When there are more expressions, it is better to select the head region less affected by expressions. By comprehensively processing multiple regions, relatively stable signals can be obtained, reducing the influence of multiple blood vessels in a single region on the result and the influence of abnormalities in a single region on the final result.

[0088] According to the embodiments of the present invention, by framing the facial region based on the obtained key points, it is avoided that the framed region includes additional environmental regions outside the skin region, which may cause drastic changes in the pixels of the facial region used for calculation, resulting in a large error in the calculation result. At the same time, by setting the framed region in the lower face and the right face, it can be avoided that the obtained results are unstable due to facial occlusion (such as hair, glasses, etc.).

[0089] Figures 9 to 12 Schematic diagrams of heart rate measurement results in different scenarios according to the embodiments of the present invention are shown.

[0090] Among them, Figure 9 A heart rate measurement graph of the person to be measured sitting under a tree shade under field sunlight according to the embodiments of the present invention is shown. Figure 10 A heart rate measurement graph of the person to be measured sitting on a step under field sunlight according to the embodiments of the present invention is shown. Figure 11 A heart rate measurement graph of the person to be measured standing under field sunlight according to the embodiments of the present invention is shown; Figure 12 A heart rate measurement graph of the person to be measured sitting in a pavilion under field sunlight according to the embodiments of the present invention is shown.

[0091] As Figures 9 to 12 shown, where the abscissa represents time and the ordinate represents signal intensity. In different test environments, for the same person to be measured, the obtained heart rate measurement graphs have significant differences. In Figure 9 , the true heart rate value is 96, and the calculated heart rate result measured by the method according to the embodiments of the present invention is 95.04; in Figure 10 , the true heart rate value is 93, and the calculated heart rate result measured by the method according to the embodiments of the present invention is 95.04; in Figure 11 , the true heart rate value is 122, and the calculated heart rate result measured by the method according to the embodiments of the present invention is 123.39; in Figure 12 , the true heart rate value is 103, and the calculated heart rate result measured by the method according to the embodiments of the present invention is 102.6. It can be seen that the method proposed by the present invention can accurately measure the heart rate value in multiple environments.

[0092] Based on the above information output method, the present invention also provides an information output device. The following will be combined with Figure 13 to describe this device in detail.

[0093] Figure 13 Fig. shows a structural block diagram of an information output device according to an embodiment of the present invention.

[0094] As Figure 13 shown, the information output device 1300 of this embodiment includes a shooting module 1310, a pulse wave signal acquisition module 1320, a cross-correlation module 1330, a splitting module 1340, and an output module 1350.

[0095] The shooting module 1310 is configured to shoot the facial area of the object to be measured to obtain multiple frames of images to be processed. The facial area includes a first area and a second area located at different positions. In one embodiment, the shooting module 1310 may be configured to perform the operation S210 described above, which will not be elaborated here.

[0096] The pulse wave signal acquisition module 1320 is configured to obtain a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area based on the multiple frames of images to be processed. In one embodiment, the pulse wave signal acquisition module 1320 may be configured to perform the operation S220 described above, which will not be elaborated here.

[0097] The cross-correlation module 1330 is configured to perform cross-correlation calculation on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal. In one embodiment, the cross-correlation module 1330 may be configured to perform the operation S230 described above, which will not be elaborated here.

[0098] The splitting module 1340 is configured to split the target pulse wave signal into m sub-calculated pulse wave signals, and perform combined calculation on every two adjacent sub-calculated pulse wave signals to obtain m - 1 sub-target pulse wave signals, where m is a positive integer greater than 0. In one embodiment, the splitting module 1340 may be configured to perform the operation S240 described above, which will not be elaborated here.

[0099] The output module 1350 is configured to obtain a heart rate calculation result of the target object based on the m - 1 sub-target pulse wave signals, and output the heart rate calculation result through a preset terminal. In one embodiment, the output module 1350 may be configured to perform the operation S250 described above, which will not be elaborated here.

[0100] According to an embodiment of the present invention, any multiple of the photographing module 1310, the pulse wave signal acquisition module 1320, the cross-correlation module 1330, the splitting module 1340, and the output module 1350 may be combined and implemented in one module, or any one of them may be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules may be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the photographing module 1310, the pulse wave signal acquisition module 1320, the cross-correlation module 1330, the splitting module 1340, and the output module 1350 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or may be implemented by any other reasonable means such as hardware or firmware for integrating or packaging circuits, or may be implemented in any one of the three implementation manners of software, hardware, and firmware, or in any appropriate combination of several of them. Alternatively, at least one of the photographing module 1310, the pulse wave signal acquisition module 1320, the cross-correlation module 1330, the splitting module 1340, and the output module 1350 may be at least partially implemented as a computer program module, and when the computer program module is run, it can perform corresponding functions.

[0101] Figure 14 FIG. shows a block diagram of an electronic device suitable for implementing the information output method according to an embodiment of the present invention.

[0102] As Figure 14 shown, the electronic device according to an embodiment of the present invention includes a processor 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage section 1408 into a random access memory (RAM) 1403. The processor 1401 may include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application specific integrated circuit (ASIC)), etc. The processor 1401 may also include on-board memory for caching purposes. The processor 1401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0103] In the RAM 1403, various programs and data required for the operation of the electronic device 1400 are stored. The processor 1401, the ROM 1402, and the RAM 1403 are connected to each other via a bus 1404. The processor 1401 performs various operations of the method flow according to the embodiments of the present invention by executing the programs in the ROM 1402 and / or the RAM 1403. It should be noted that the programs can also be stored in one or more memories other than the ROM 1402 and the RAM 1403. The processor 1401 can also perform various operations of the method flow according to the embodiments of the present invention by executing the programs stored in the one or more memories.

[0104] According to an embodiment of the present invention, the electronic device 1400 may further include an input / output (I / O) interface 1405, and the input / output (I / O) interface 1405 is also connected to the bus 1404. The electronic device 1400 may further include one or more of the following components connected to the input / output (I / O) interface 1405: an input part 1406 including a keyboard, a mouse, etc.; an output part 1407 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage part 1408 including a hard disk, etc.; and a communication part 1409 including a network interface card such as a LAN card, a modem, etc. The communication part 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the input / output (I / O) interface 1405 as needed. A removable medium 1411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1410 as needed so that a computer program read from it can be installed into the storage part 1408 as needed.

[0105] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of the present invention is implemented.

[0106] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present invention, the computer-readable storage medium may be any tangible medium that contains or stores a program, and this program can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 1402 and / or RAM 1403 described above and / or one or more memories other than the ROM 1402 and RAM 1403.

[0107] An embodiment of the present invention further includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product runs in a computer system, the program code is used to enable the computer system to implement the information output method provided by the embodiment of the present invention.

[0108] When the computer program is executed by the processor 1401, it executes the above functions defined in the system / apparatus of the embodiment of the present invention. According to an embodiment of the present invention, the above-described systems, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0109] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and be downloaded and installed through the communication part 1409, and / or be installed from the removable medium 1411. The program code contained in the computer program can be transmitted by any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0110] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1409, and / or be installed from the removable medium 1411. When the computer program is executed by the processor 1401, it executes the above functions defined in the system of the embodiment of the present invention. According to an embodiment of the present invention, the above-described systems, devices, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0111] In accordance with embodiments of the present invention, program code for executing the computer programs provided by the embodiments of the present invention may be written in any combination of one or more programming languages. Specifically, these computing programs may be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, such as Java, C++, Python, the "C" language, or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computing device (e.g., by connecting through the Internet using an Internet service provider).

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the above-mentioned module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and combinations of blocks in the block diagram or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0113] Those skilled in the art can understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present invention.

[0114] The above describes the embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although the embodiments are described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Without departing from the scope of the present invention, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications should fall within the scope of the present invention.

Claims

1. An information output method, characterized in that, The method includes: Taking pictures of the facial area of the object to be measured to obtain multiple frames of images to be processed, where the facial area includes a first area and a second area located at different positions; Based on the multiple frames of images to be processed, obtaining a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area; Performing cross-correlation calculation on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal; Splitting the target pulse wave signal into m sub-calculated pulse wave signals, and performing merging calculation on every two adjacent sub-calculated pulse wave signals to obtain m - 1 sub-target pulse wave signals, where m is a positive integer greater than 0; Based on the m - 1 sub-target pulse wave signals, obtaining a heart rate calculation result of the target object, and outputting the heart rate calculation result through a preset terminal.

2. The method according to claim 1, characterized in that The obtaining the first pulse wave signal corresponding to the first area and the second pulse wave signal corresponding to the second area based on the multiple frames of images to be processed includes: Based on the multiple frames of images to be processed, obtaining a first pixel mean value array corresponding to the first area and a second pixel mean value array corresponding to the second area; Based on the first pixel mean value array, obtaining the first pulse wave signal; Based on the second pixel mean value array, obtaining the second pulse wave signal.

3. The method according to claim 2, characterized in that The obtaining the first pulse wave signal based on the first pixel mean value array includes: Normalizing the first pixel mean value array to obtain a normalized first pixel mean value array; Separating pixel channels of the normalized first pixel mean value array to obtain a preset number of pixel channel mean value arrays; Using a preset signal processing algorithm, obtaining an initial pulse wave signal based on the preset number of pixel channel mean value arrays; Using an autocorrelation algorithm, obtaining the first pulse wave signal based on the initial pulse wave signal.

4. The method according to claim 1, wherein The multiple frames of images to be processed are collected by an imaging device, and the obtaining the heart rate calculation result of the target object based on the m - 1 sub-target pulse wave signals includes: For a sub-target pulse wave signal, obtaining the number of peaks and the signal duration of the sub-target pulse wave signal; Based on the frame rate of the imaging device, the m - 1 numbers of peaks, and the m - 1 signal durations, obtaining the heart rate calculation result of the target object.

5. The method according to claim 2, characterized in that, The obtaining the first pixel mean value array corresponding to the first area and the second pixel mean value array corresponding to the second area based on the multiple frames of images to be processed includes: Using a preset key point detection model to extract multiple frames of first images to be processed corresponding to the first area and multiple frames of second images to be processed corresponding to the second area from the multiple frames of images to be processed; Based on the multiple frames of first images to be processed, obtaining the first pixel mean value array; Based on the multiple frames of second images to be processed, obtaining the second pixel mean value array.

6. The method according to claim 4, wherein The method further includes: Using a preset interference frame number threshold and a preset amplitude threshold to denoise the heart rate calculation result to obtain a target heart rate calculation result.

7. An information output device, characterized in that, The device includes: A photographing module, configured to photograph a facial area of an object to be measured, so as to obtain multiple frames of images to be processed, where the facial area includes a first area and a second area located at different positions; A pulse wave signal acquisition module, configured to obtain a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area based on the multiple frames of images to be processed; A cross-correlation module, configured to perform cross-correlation calculation on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal; A splitting module, configured to split the target pulse wave signal to obtain m sub-computed pulse wave signals, and perform combined calculation on every two adjacent sub-computed pulse wave signals to obtain m - 1 sub-target pulse wave signals, where m is a positive integer greater than 0; An output module, configured to obtain a heart rate calculation result of a target object based on the m - 1 sub-target pulse wave signals, and output the heart rate calculation result through a preset terminal.

8. An electronic device, comprising: One or more processors; A memory, configured to store one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images