Information output method, device, equipment, medium and program product

Abstract

The invention provides an information output method, device and equipment, a medium and a program product, and is applied to the technical field of information processing.The information output method comprises the steps that a face area of a to-be-detected object is shot, and multiple frames of to-be-processed images are obtained; obtaining a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area based on the multiple frames of to-be-processed images; performing cross-correlation calculation on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal; splitting the target pulse wave signal to obtain m sub-calculation pulse wave signals, and performing merging calculation on every two adjacent sub-calculation pulse wave signals to obtain m-1 sub-target pulse wave signals; obtaining a heart rate calculation result of the target object based on the m-1 sub-target pulse wave signals, and outputting the heart rate calculation result through a preset terminal; the influence of single area abnormity on a final result is reduced, data incontinuity is reduced, and the accuracy of a heart rate calculation result is improved.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and more specifically to an information output method, device, equipment, medium and program product. Background Art

[0002] Heart rate status is very important for detecting a person's psychological and physiological state. Traditional heart rate status measurement is often contact-based, requiring the tester to wear specific equipment for measurement. However, contact-based measurement methods are difficult to apply in some scenarios. Therefore, a remote heart rate measurement device can be used to obtain heart rate status-related features for analysis.

[0003] In related technologies, remote heart rate measurement devices often need to use bandpass filtering to remove noise from collected heart rate signals. However, bandpass filtering will also filter out part of the heart rate signal, resulting in low accuracy in the final heart rate calculation results. At the same time, affected by rapid movement and emotional fluctuations, the heart rate waveform will change irregularly, which will also lead to large errors in the heart rate calculation results. Summary of the invention

[0004] In view of the above problems, the present invention provides an information output method, apparatus, device, medium and program product.

[0005] According to a first aspect of the present invention, there is provided an information output method, comprising: photographing a facial area of ​​an object to be measured to obtain a plurality of frames of images to be processed, wherein the facial area comprises a first area and a second area located at different positions; obtaining a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area based on the plurality of frames of images to be processed; performing cross-correlation calculation on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal; splitting the target pulse wave signal to obtain m sub-calculated pulse wave signals, performing merging calculation on each two adjacent sub-calculated pulse wave signals to obtain m-1 sub-target pulse wave signals, wherein m is a positive integer greater than 0; obtaining a heart rate calculation result of the target object based on the m-1 sub-target pulse wave signals, and outputting the heart rate calculation result through a preset terminal.

[0006] According to an embodiment of the present invention, the obtaining of the first pulse wave signal corresponding to the first area and the second pulse wave signal corresponding to the second area based on the multiple frames of images to be processed includes: obtaining a first pixel mean array corresponding to the first area and a second pixel mean array corresponding to the second area based on the multiple frames of images to be processed; obtaining the first pulse wave signal based on the first pixel mean array; and obtaining the second pulse wave signal based on the second pixel mean array.

[0007] According to an embodiment of the present invention, the above-mentioned obtaining the above-mentioned first pulse wave signal based on the above-mentioned first pixel mean array includes: normalizing the above-mentioned first pixel mean array to obtain a normalized first pixel mean array; performing pixel channel separation on the above-mentioned normalized first pixel mean array to obtain a preset number of pixel channel mean arrays; using a preset signal processing algorithm to obtain an initial pulse wave signal based on the above-mentioned preset number of pixel channel mean arrays; using an autocorrelation algorithm to obtain the above-mentioned first pulse wave signal based on the above-mentioned initial pulse wave signal.

[0008] According to an embodiment of the present invention, the above-mentioned multiple frames of images to be processed are collected by a camera device, and the above-mentioned heart rate calculation result of the above-mentioned target object is obtained based on the above-mentioned m-1 sub-target pulse wave signals, including: for one sub-target pulse wave signal, obtaining the number of peaks and signal duration of the sub-target pulse wave signal; obtaining the heart rate calculation result of the above-mentioned target object based on the frame rate of the above-mentioned camera device, the m-1 above-mentioned number of peaks and the m-1 above-mentioned signal duration.

[0009] According to an embodiment of the present invention, the above-mentioned first pixel mean array corresponding to the above-mentioned first area and the second pixel mean array corresponding to the above-mentioned second area are obtained based on the above-mentioned multiple frames of images to be processed, including: using a preset key point detection model to extract multiple frames of first images to be processed corresponding to the above-mentioned first area and multiple frames of second images to be processed corresponding to the above-mentioned second area from the above-mentioned multiple frames of images to be processed; obtaining the above-mentioned first pixel mean array based on the above-mentioned multiple frames of first images to be processed; obtaining the above-mentioned second pixel mean array based on the above-mentioned multiple frames of second images to be processed.

[0010] According to an embodiment of the present invention, the method further includes: denoising the heart rate calculation result using a preset interference frame number threshold and a preset amplitude threshold to obtain a target heart rate calculation result.

[0011] The second aspect of the present invention provides an information output device, comprising: a shooting module, used to shoot a facial area of ​​a subject to be tested to obtain multiple frames of images to be processed, wherein the facial area includes a first area and a second area located at different positions; a pulse wave signal acquisition module, used to obtain a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area based on the multiple frames of images to be processed; a cross-correlation module, used to perform cross-correlation calculation on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal; a splitting module, used to split the target pulse wave signal to obtain m sub-calculated pulse wave signals, and merge and calculate each two adjacent sub-calculated pulse wave signals to obtain m-1 sub-target pulse wave signals, wherein m is a positive integer greater than 0; an output module, used to obtain a heart rate calculation result of the target subject based on the m-1 sub-target pulse wave signals, and output the heart rate calculation result through a preset terminal.

[0012] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0013] The fourth aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0014] The fifth aspect of the present invention also provides a computer program product, including a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0015] According to an embodiment of the present invention, a first area and a second area of ​​an area of ​​an object to be measured are photographed, and a pulse wave waveform is highlighted based on a multiple autocorrelation algorithm to obtain a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area. Then, cross-correlation processing is performed on the second pulse wave signal of the first pulse wave signal, thereby reducing the impact of a single area abnormality on the final result. Furthermore, the target pulse wave is signal-splitting, and two adjacent sub-calculated pulse waves are merged and calculated, and then the heart rate calculation result is obtained based on the merged sub-target pulse wave data, which can reduce data incoherence, correlate the data front and back, and improve the accuracy of the heart rate calculation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings.

[0017] Figure 1A diagram showing an application scenario of the information output method and device according to an embodiment of the present invention is shown.

[0018] Figure 2 A flow chart of an information output method according to an embodiment of the present invention is shown.

[0019] Figure 3 A schematic diagram of a non-contact heart rate acquisition principle according to an embodiment of the present invention is shown.

[0020] Figure 4 A schematic diagram of a sliding window according to an embodiment of the present invention is shown.

[0021] Figure 5 A schematic diagram showing the principle of obtaining a pulse wave signal based on a facial image according to an embodiment of the present invention is shown.

[0022] Figure 6 A graph showing peak heart rate calculation results according to an embodiment of the present invention is shown.

[0023] Figure 7 A facial region key point map according to an embodiment of the present invention is shown.

[0024] Figure 8 A schematic diagram of acquiring multiple different facial regions according to an embodiment of the present invention is shown.

[0025] Fig. 9 A heart rate measurement diagram of a person to be measured sitting in the shade of a tree under sunlight in the wild according to an embodiment of the present invention is shown.

[0026] Fig.10 A heart rate measurement diagram of a person to be measured sitting on steps under sunlight in the wild according to an embodiment of the present invention is shown.

[0027] Fig.11 A heart rate measurement diagram of a person to be measured standing in the wild under sunlight according to an embodiment of the present invention is shown.

[0028] Fig.12 A heart rate measurement diagram of a person under test sitting in a pavilion under sunlight in the wild according to an embodiment of the present invention is shown.

[0029] Fig.13 A structural block diagram of an information output device according to an embodiment of the present invention is shown.

[0030] Fig.14 A block diagram of an electronic device suitable for implementing an information output method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0031] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0032] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0033] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0034] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0035] Heart rate status is very important for monitoring a person's psychological and physiological condition. At the same time, heart rate data can also be used to analyze emotional status. Traditional heart rate status measurements are all contact-based, such as electrode electrocardiogram and photoplethysmography. Although photoplethysmography is more convenient than electrode electrocardiogram, it also requires contact for measurement. However, in some special cases, wearable devices are prone to cause skin infections. Thus, remote heart rate measurement was born. It uses non-contact devices such as cameras, radars and other sensors to obtain and analyze heart rate-related features to obtain heart rate status.

[0036] In related technologies, remote heart rate measurement devices focus on the processing of pulse wave signals, and perform detrending, bandpass filtering or wavelet transform on the extracted pulse wave signals to remove interference in the noise signal, and perform state box selection on the pulse wave signal to slowly obtain the number of peaks through fast Fourier to calculate the heart rate state. Although the final result of this solution is relatively stable, since the bandpass filter directly removes the signal outside the normal heart rate range, it means that part of the abnormal heart rate signal is also filtered out. At the same time, in the case of rapid exercise or drastic emotional fluctuations, the heart rate state will not be a good frequency waveform, which will also lead to large errors in the calculation results.

[0037] In view of this, an embodiment of the present invention provides an information output method, which includes: photographing a facial area of ​​an object to be tested to obtain multiple frames of images to be processed, wherein the facial area includes a first area and a second area located at different positions of the facial area; obtaining a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area based on the multiple frames of images to be processed, and the first pulse wave signal and the second pulse wave signal are subjected to multiple autocorrelation processing; performing cross-correlation calculation on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal; splitting the target pulse wave signal to obtain m sub-calculated pulse wave signals, and merging and calculating each two adjacent sub-calculated pulse wave signals to obtain m-1 sub-target pulse wave signals, wherein m is a positive integer greater than 0; obtaining a heart rate calculation result of the target object based on the m-1 sub-target pulse wave signals, and outputting the heart rate calculation result to a preset terminal.

[0038] Figure 1 A diagram showing an application scenario of the information output method and device according to an embodiment of the present invention is shown.

[0039] like Figure 1 As shown, the application scenario according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0040] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0041] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0042] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0043] It should be noted that the information output method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the information output device provided in the embodiment of the present invention can generally be set in the server 105. The information output method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the information output device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0044] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0045] The following will be based on Figure 1 The scene described by Figure 2 to Figure 12 The information output method of the embodiment of the present invention is described in detail.

[0046] Figure 2 A flow chart of an information output method according to an embodiment of the present invention is shown.

[0047] like Figure 2 As shown, the information output method of this embodiment includes operations S210 to S250.

[0048] In operation S210, a facial region of an object to be detected is photographed to obtain a plurality of frames of images to be processed.

[0049] The facial area includes a first area and a second area located at different positions.

[0050] In operation S220, a first pulse wave signal corresponding to the first region and a second pulse wave signal corresponding to the second region are obtained based on the plurality of frames of images to be processed.

[0051] The first pulse wave signal and the second pulse wave signal are processed by multiple autocorrelation processes.

[0052] Figure 3 A schematic diagram of a non-contact heart rate acquisition principle according to an embodiment of the present invention is shown.

[0053] like Figure 3 As shown, when the light source is irradiated on the skin, there is specular reflection and diffuse reflection. The video acquisition device receives the change of pixel value on the skin surface caused by the contraction of veins and arteries caused by the heartbeat, and then extracts the feature of the diffuse reflection containing the change feature to calculate the heart rate signal.

[0054] According to an embodiment of the present invention, the heart rate can be estimated by analyzing the slight changes in skin color in the video through video-based remote photoplethysmography, and a first candidate pulse wave signal corresponding to the first area and a second candidate pulse wave signal corresponding to the second area are obtained. The first candidate pulse wave signal is obtained after multiple autocorrelations, and the second candidate pulse wave signal is obtained after multiple autocorrelations. The first candidate pulse wave signal can be autocorrelated by the following formula (1), or the second candidate pulse wave signal can be autocorrelated by the following formula (1). As shown in formula (1):

[0055] (1);

[0056] in, is the pulse wave signal obtained after autocorrelation, n is the shift amount of the candidate pulse wave signal, m is the index variable for summation, when the above formula (1) is used to calculate the first pulse wave signal, x is the first candidate pulse wave signal; when the above formula (1) is used to calculate the second pulse wave signal, x is the second candidate pulse wave signal.

[0057] In operation S230, a cross-correlation calculation is performed on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal.

[0058] The cross-correlation calculation of the first pulse wave signal and the second pulse wave signal can be performed using the following formula (2).

[0059] (2);

[0060] in, Indicates the target pulse wave signal, is the index variable for summation, represents the first pulse wave signal, y represents the second pulse wave signal, Indicates the amount of time delay.

[0061] In operation S240, the target pulse wave signal is split into m sub-calculated pulse wave signals, and a combination calculation is performed on every two adjacent sub-calculated pulse wave signals to obtain m-1 sub-target pulse wave signals.

[0062] Wherein, m is a positive integer greater than 0.

[0063] Figure 4 A schematic diagram of a sliding window according to an embodiment of the present invention is shown.

[0064] like Figure 4 As shown, the target pulse signal can be split into four sub-calculated pulse wave signals, namely sub-calculated pulse wave signal 1, sub-calculated pulse wave signal 2, sub-calculated pulse wave signal 3 and sub-calculated pulse wave signal 4. Two adjacent sub-calculated pulse wave signals are merged and calculated, that is, sub-calculated pulse wave signal 1 and sub-calculated pulse wave signal 2 are merged into sub-target pulse wave signal A, sub-calculated pulse wave signal 2 and sub-calculated pulse wave signal 3 are merged into sub-target pulse wave signal B, and sub-calculated pulse wave signal 3 and sub-calculated pulse wave signal 4 are merged into sub-target pulse wave signal C.

[0065] According to an embodiment of the present invention, when the frame rate of the acquisition device is 30fps (Frames Per Second), a 20s target pulse wave signal can be stably selected for splitting and calculation. By splitting and reorganizing the target pulse wave signal, when calculating the heart rate calculation result, the length of time when the collected facial pixel values ​​change dramatically due to severe shaking or illumination changes can be reduced, and the abnormal changes can be limited to a single sub-calculated pulse wave, thereby reducing the probability of abnormal results.

[0066] In operation S250, a heart rate calculation result of the target object is obtained based on the m-1 sub-target pulse wave signals, and the heart rate calculation result is output through a preset terminal.

[0067] It should be noted that the heart rate calculation result calculated according to the embodiment of the present invention is only a reference intermediate value result, and is not a medical diagnosis result.

[0068] According to an embodiment of the present invention, a first area and a second area of ​​an area of ​​an object to be measured are photographed, and a pulse wave waveform is highlighted based on a multiple autocorrelation algorithm to obtain a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area. Then, cross-correlation processing is performed on the second pulse wave signal of the first pulse wave signal, thereby reducing the impact of a single area abnormality on the final result. Furthermore, the target pulse wave is signal-splitting, and two adjacent sub-calculated pulse waves are merged and calculated, and then the heart rate calculation result is obtained based on the merged sub-target pulse wave data, which can reduce data incoherence, correlate the data front and back, and improve the accuracy of the heart rate calculation result.

[0069] According to an embodiment of the present invention, the above-mentioned obtaining a first pulse wave signal corresponding to a first area and a second pulse wave signal corresponding to a second area based on multiple frames of images to be processed includes: obtaining a first pixel mean array corresponding to the first area and a second pixel mean array corresponding to the second area based on multiple frames of images to be processed; obtaining the first pulse wave signal based on the first pixel mean array; and obtaining the second pulse wave signal based on the second pixel mean array.

[0070] According to an embodiment of the present invention, obtaining a first pulse wave signal based on a first pixel mean array includes: normalizing the first pixel mean array to obtain a normalized first pixel mean array; performing pixel channel separation on the normalized first pixel mean array to obtain a preset number of pixel channel mean arrays; using a preset signal processing algorithm to obtain an initial pulse wave signal based on a preset number of pixel channel mean arrays; and using an autocorrelation algorithm to obtain a first pulse wave signal based on the initial pulse wave signal.

[0071] Figure 5 A schematic diagram showing the principle of obtaining a pulse wave signal based on a facial image according to an embodiment of the present invention is shown.

[0072] like Figure 5 As shown, R n (t) represents the red pulse wave signal obtained by color channel separation through the first pixel mean, G n (t) represents the green pulse wavelength signal obtained by color channel separation through the first pixel mean, B n (t) represents the blue pulse wave signal obtained by color channel separation by the first pixel mean. The three-dimensional graph in the middle indicates the distribution of pulse intensity. The signals of the three channels are mapped into a three-dimensional space by remote photoplethysmography. Each axis represents the signal intensity of a color channel. z1, z2, and z3 are feature vectors extracted from the sphere, which are used to capture the main changes of the pulse wave signal. 1 (t)、s 2 (t)、s3 (t) represents the initial pulse wave signal obtained.

[0073] According to an embodiment of the present invention, the preset signal processing algorithm may be, for example, a rPPG (Remote Photoplethysmography) algorithm.

[0074] According to an embodiment of the present invention, by extracting a pixel mean array of a specific area from multiple frames of images, and performing normalization and pixel separation, changes in the pulse signal can be captured from different angles, thereby improving the accuracy of the analysis. Further, by using a preset signal processing algorithm to reduce the pixels into a three-channel pixel channel mean array, and performing feature extraction to obtain an initial pulse wave signal, the processing flow of the pulse wave signal can be simplified.

[0075] According to an embodiment of the present invention, multiple frames of images to be processed are collected by a camera device, and a heart rate calculation result of the target object is obtained based on m-1 sub-target pulse wave signals, including: for a sub-target pulse wave signal, obtaining the number of peaks and signal duration of the sub-target pulse wave signal; and obtaining the heart rate calculation result of the target object based on the frame rate of the camera device, the number of m-1 peaks and the m-1 signal duration.

[0076] The heart rate calculation result of the target object can be calculated by the following formula (3).

[0077] (3);

[0078] in, Indicates the heart rate calculation result of the target object. It represents the frame rate of the shooting device, n represents the number of peaks, and L represents the signal duration.

[0079] According to an embodiment of the present invention, the method further includes: denoising the heart rate calculation result using a preset interference frame number threshold and a preset amplitude threshold to obtain a target heart rate calculation result.

[0080] Figure 6 A graph showing peak heart rate calculation results according to an embodiment of the present invention is shown.

[0081] According to the embodiment of the present disclosure, the normal heartbeat state is 40~150BPM (Beats Per Minute), and the interval between two heartbeats is 0.4s~1.5s. For a 30fps camera, the corresponding data frames are 12~45 frames. The preset interference frame number threshold can be set to 12 frames to remove the interference peak. The peak heart rate calculation result obtained after removing the interference peak is shown in the figure Figure 6 As shown, Figure 6 The horizontal and vertical axes are not shown. Figure 6 It is only to show the waveform shape and does not strictly indicate the waveform size.

[0082] According to an embodiment of the present invention, the peak values ​​can be screened by the heart rate properties to obtain the target heart rate calculation result. For the sub-target pulse wave signal, the heartbeat amplitude is greater than the interference wave amplitude. The preset amplitude threshold can be set as the upper limit of the interference wave amplitude, so that the peak data that does not meet the conditions can be screened out by setting the preset amplitude threshold, and finally the target heart rate calculation result can be calculated.

[0083] According to an embodiment of the present invention, a first pixel mean array corresponding to a first area and a second pixel mean array corresponding to a second area are obtained based on multiple frames of images to be processed, including: using a preset key point detection model to extract multiple frames of first images to be processed corresponding to the first area and multiple frames of second images to be processed corresponding to the second area from multiple frames of images to be processed; obtaining a first pixel mean array based on the multiple frames of first images to be processed; and obtaining a second pixel mean array based on the multiple frames of second images to be processed.

[0084] According to an embodiment of the present invention, a computer vision application tool (i.e., a preset key point detection model) can be used to collect key points of a facial area. The computer vision application tool collects 468 key points from the face, and the key points are used to accurately select the facial area to obtain areas such as the lower face, head, left face, and right face. A large area contains more key points, such as [266, 352, 433, 364, 379, 400, 152, 150, 172, 137, 123, 36]. A small area does not include areas such as the mouth and nose. The first area can be selected as the lower face area, and the second area can be selected as the right face, which mainly includes key points such as [109, 10, 338, 337, 151, 108]. This can prevent the obtained results from being disturbed by activities such as breathing and talking.

[0085] Figure 7 A facial region key point map according to an embodiment of the present invention is shown. Figure 8 A schematic diagram of acquiring multiple different facial regions according to an embodiment of the present invention is shown.

[0086] Can be based on Figure 7 The key points located in different areas are obtained Figure 8 Including: Figure 7 The global key points in the full face image 810 can be obtained according to Figure 7 The mouth key points and the chin key points in the image 820 can be obtained according to Figure 7 The nose key points and chin key points in the image 830 of the lower face can be obtained; Figure 7The right face image 840 can be obtained by the key points of the right corner of the mouth and the right chin; Figure 7 The left mouth corner key point and the left chin key point obtain the left face image 850.

[0087] According to an embodiment of the present invention, different combinations of the first area and the second area can be selected for different scenes and different areas. When long-term stable measurements are performed and there are changes in external lighting, more expressions can be obtained from the lower face and the right face. When there are many expressions, it is better to select the head area that is less affected by the expressions. By performing comprehensive processing on multiple areas, a more stable signal can be obtained, reducing the impact of multiple blood vessels in a single area on the results and the impact of abnormalities in a single area on the final results.

[0088] According to an embodiment of the present invention, by selecting the facial area based on the acquired key points, it is avoided that the selected area includes additional environmental areas outside the skin area, which would cause drastic changes in the pixels of the facial area used for calculation, resulting in large errors in the calculation results; at the same time, the selected area is set on the lower face and the right face, which can avoid unstable acquisition results caused by facial occlusion (such as hair, glasses, etc.).

[0089] Figure 9~Figure 12 A schematic diagram of heart rate measurement results in different scenarios according to an embodiment of the present invention is shown.

[0090] in, Fig. 9 A heart rate measurement diagram of a person to be measured sitting in the shade of a tree under sunlight in the wild according to an embodiment of the present invention is shown. Fig.10 A heart rate measurement diagram of a person to be measured sitting on steps under sunlight in the wild according to an embodiment of the present invention is shown. Fig.11 A heart rate measurement diagram of a person to be measured standing under sunlight in the wild according to an embodiment of the present invention is shown; Fig.12 A heart rate measurement diagram of a person under test sitting in a pavilion under sunlight in the wild according to an embodiment of the present invention is shown.

[0091] like Figure 9~Figure 12 As shown in the figure, the horizontal axis represents time and the vertical axis represents signal strength. Under different test environments, for the same person under test, the heart rate measurement graphs obtained have significant differences. Fig. 9 The actual heart rate value is 96, and the heart rate calculated by the method of the embodiment of the present invention is 95.04; Fig.10 In , the real heart rate value is 93, and the heart rate calculated by the method of the embodiment of the present invention is 95.04; in Fig.11 In , the real heart rate value is 122, and the heart rate calculated by the method of the embodiment of the present invention is 123.39; Fig.12In the example, the real heart rate value is 103, and the heart rate calculated by the method according to the embodiment of the present invention is 102.6. It can be seen that the method proposed by the present invention can accurately measure the heart rate value in a variety of environments.

[0092] Based on the above information output method, the present invention also provides an information output device. Fig.13 The device is described in detail.

[0093] Fig.13 A structural block diagram of an information output device according to an embodiment of the present invention is shown.

[0094] like Fig.13 As shown, the information output device 1300 of this embodiment includes a shooting module 1310 , a pulse wave signal acquisition module 1320 , a cross-correlation module 1330 , a splitting module 1340 and an output module 1350 .

[0095] The photographing module 1310 is used to photograph the facial region of the object to be measured to obtain multiple frames of images to be processed, and the facial region includes a first region and a second region located at different positions. In one embodiment, the photographing module 1310 can be used to perform the operation S210 described above, which will not be repeated here.

[0096] The pulse wave signal acquisition module 1320 is used to obtain a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area based on multiple frames of images to be processed. In one embodiment, the pulse wave signal acquisition module 1320 can be used to perform the operation S220 described above, which is not repeated here.

[0097] The cross-correlation module 1330 is used to perform cross-correlation calculation on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal. In one embodiment, the cross-correlation module 1330 can be used to perform the operation S230 described above, which will not be described in detail here.

[0098] The splitting module 1340 is used to split the target pulse wave signal into m sub-calculated pulse wave signals, and merge and calculate each two adjacent sub-calculated pulse wave signals to obtain m-1 sub-target pulse wave signals, where m is a positive integer greater than 0. In one embodiment, the splitting module 1340 can be used to perform the operation S240 described above, which will not be repeated here.

[0099] The output module 1350 is used to obtain the heart rate calculation result of the target object based on the m-1 sub-target pulse wave signals, and output the heart rate calculation result through a preset terminal. In one embodiment, the output module 1350 can be used to perform the operation S250 described above, which will not be repeated here.

[0100] According to an embodiment of the present invention, any multiple modules among the shooting module 1310, the pulse wave signal acquisition module 1320, the cross-correlation module 1330, the splitting module 1340 and the output module 1350 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the shooting module 1310, the pulse wave signal acquisition module 1320, the cross-correlation module 1330, the splitting module 1340 and the output module 1350 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware and firmware or in a proper combination of any of them. Alternatively, at least one of the shooting module 1310, the pulse wave signal acquisition module 1320, the cross-correlation module 1330, the splitting module 1340 and the output module 1350 may be at least partially implemented as a computer program module, which may perform a corresponding function when executed.

[0101] Fig.14 A block diagram of an electronic device suitable for implementing an information output method according to an embodiment of the present invention is shown.

[0102] like Fig.14 As shown, the electronic device according to an embodiment of the present invention includes a processor 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage part 1408 into a random access memory (RAM) 1403. The processor 1401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1401 may also include an onboard memory for caching purposes. The processor 1401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0103] In RAM 1403, various programs and data required for the operation of electronic device 1400 are stored. Processor 1401, ROM 1402 and RAM 1403 are connected to each other via bus 1404. Processor 1401 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 1402 and / or RAM 1403. It should be noted that the program can also be stored in one or more memories other than ROM 1402 and RAM 1403. Processor 1401 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0104] According to an embodiment of the present invention, the electronic device 1400 may further include an input / output (I / O) interface 1405, which is also connected to the bus 1404. The electronic device 1400 may further include one or more of the following components connected to the input / output (I / O) interface 1405: an input portion 1406 including a keyboard, a mouse, etc.; an output portion 1407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1408 including a hard disk, etc.; and a communication portion 1409 including a network interface card such as a LAN card, a modem, etc. The communication portion 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the input / output (I / O) interface 1405 as needed. A removable medium 1411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1410 as needed, so that a computer program read therefrom is installed into the storage portion 1408 as needed.

[0105] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0106] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 1402 and / or RAM 1403 described above and / or one or more memories other than ROM 1402 and RAM 1403.

[0107] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the information output method provided by the embodiment of the present invention.

[0108] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 1401. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0109] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1409, and / or installed from the removable medium 1411. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0110] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1409, and / or installed from the removable medium 1411. When the computer program is executed by the processor 1401, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0111] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0112] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0114] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. An information output method, characterized in that: The method comprises: Shooting a facial region of the subject to be tested to obtain a plurality of frames of images to be processed, wherein the facial region includes a first region and a second region located at different positions; Obtaining a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area based on the multiple frames of images to be processed; Performing cross-correlation calculation on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal; The target pulse wave signal is split into m sub-calculated pulse wave signals, and each two adjacent sub-calculated pulse wave signals are combined and calculated to obtain m-1 sub-target pulse wave signals, where m is a positive integer greater than 0; A heart rate calculation result of the target object is obtained based on the m-1 sub-target pulse wave signals, and the heart rate calculation result is output through a preset terminal.

2. The method according to claim 1, characterized in that The obtaining, based on the multiple frames of images to be processed, a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area comprises: Obtaining a first pixel mean value array corresponding to the first area and a second pixel mean value array corresponding to the second area based on the multiple frames of images to be processed; Obtaining the first pulse wave signal based on the first pixel mean value array; The second pulse wave signal is obtained based on the second pixel mean value array.

3. The method according to claim 2, characterized in that The obtaining the first pulse wave signal based on the first pixel mean value array includes: Normalizing the first pixel mean value array to obtain a normalized first pixel mean value array; Performing pixel channel separation on the normalized first pixel mean value array to obtain a preset number of pixel channel mean value arrays; Using a preset signal processing algorithm, an initial pulse wave signal is obtained based on the preset number of pixel channel mean arrays; The first pulse wave signal is obtained based on the initial pulse wave signal by using an autocorrelation algorithm.

4. The method according to claim 1, characterized in that: The multiple frames of images to be processed are collected by a camera device, and the heart rate calculation result of the target object is obtained based on the m-1 sub-target pulse wave signals, including: For a sub-target pulse wave signal, obtaining the number of peaks and signal duration of the sub-target pulse wave signal; The heart rate calculation result of the target object is obtained based on the frame rate of the camera device, the m-1 number of peaks and the m-1 signal durations.

5. The method according to claim 2, characterized in that: The step of obtaining a first pixel mean value array corresponding to the first area and a second pixel mean value array corresponding to the second area based on the multiple frames of images to be processed includes: Using a preset key point detection model, extracting a plurality of frames of first images to be processed corresponding to the first area and a plurality of frames of second images to be processed corresponding to the second area from the plurality of frames of images to be processed; Obtaining the first pixel mean value array based on the multiple frames of first to-be-processed images; The second pixel mean array is obtained based on the multiple frames of second images to be processed.

6. The method according to claim 4, characterized in that The method further comprises: The heart rate calculation result is denoised using a preset interference frame number threshold and a preset amplitude threshold to obtain a target heart rate calculation result.

7. An information output device, characterized in that: The device comprises: A shooting module, used for shooting a facial area of ​​the object to be tested to obtain a plurality of frames of images to be processed, wherein the facial area includes a first area and a second area located at different positions; A pulse wave signal acquisition module, used for obtaining a first pulse wave signal corresponding to the first area and a second pulse wave signal corresponding to the second area based on the multiple frames of images to be processed; A cross-correlation module, used for performing cross-correlation calculation on the first pulse wave signal and the second pulse wave signal to obtain a target pulse wave signal; A splitting module, used for splitting the target pulse wave signal into m sub-calculated pulse wave signals, and merging and calculating each two adjacent sub-calculated pulse wave signals to obtain m-1 sub-target pulse wave signals, wherein m is a positive integer greater than 0; The output module is used to obtain the heart rate calculation result of the target object based on the m-1 sub-target pulse wave signals, and output the heart rate calculation result through a preset terminal.

8. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images