A breathing rate detection method and device based on drone video

Abstract

The present invention discloses a respiratory rate detection method and device based on drone video, belonging to the technical field of drone image processing. The method obtains a video stream containing the respiratory action signs of the target object, selects the chest and abdomen of the target object in the video stream as the region of interest; grayscale processing is performed on the region of interest, the chest and abdomen vibration region of the target object is taken as the object, and the respiratory vibration region is subjected to time phase processing to extract the original signal; since the original signal is mixed with the interference signal caused by the drone jitter, the original signal is used to obtain the candidate respiratory modal signal by modal decomposition, the interference signal caused by the drone shaking is removed, the noise interference is suppressed, and finally a good respiratory rate waveform fitting is obtained, and at the same time, the spatial domain weighting of the amplitude information to the phase information can improve the signal-to-noise ratio of the respiratory signal, and realize the dynamic detection of accurate respiratory rate.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle image processing, and in particular, relates to a breathing rate detection method, device, electronic device and storage medium based on unmanned aerial vehicle video stream. Background Art

[0002] The search and rescue of people in disaster-stricken areas is often delayed due to road congestion, insufficient search and rescue personnel, etc., resulting in casualties. With the rapid development of drone applications, people have gradually begun to use drones equipped with cameras to detect vital signs, making it possible to use drones to assist in search and rescue of people and to assist in detecting people's vital signs.

[0003] During the search and rescue process, the respiratory rate is one of the important indicators reflecting the health status of the target object. The traditional respiratory rate detection method requires attaching an impedance sensor to the target object and using the change signal of the sensor voltage to detect the respiratory rate. It is not suitable for respiratory rate detection in the above scenario.

[0004] At present, people have begun to use drones to measure human respiratory rate. For example, by photographing the human face, measuring blood volume pulse (BVP), or photographing the respiratory vibration of the human abdomen, etc., the respiratory rate is calculated. However, when the drone is hovering to shoot video, it will be disturbed by airflow and motors, causing its own vibration. Because the interference caused by the drone's vibration will have a serious impact on the signal detection of the video, there are problems such as poor accuracy in identifying and detecting the respiratory rate and interference from false target signals. Summary of the invention

[0005] In view of the problem that the interference caused by the jitter of the drone during the above-mentioned breathing rate detection process will have a serious impact on the signal of the video, the present invention proposes a breathing rate detection method based on the drone video stream, considering that the time-phase signal is mixed with the interference signal caused by the jitter of the drone, the original signal is used to obtain the candidate breathing mode signal by modal decomposition, and the intrinsic mode with the smallest variance is selected to realize breathing rate detection, remove the interference signal caused by the shaking of the drone, suppress noise interference, and finally obtain a good breathing rate waveform fitting, and at the same time, the spatial domain weighting of the amplitude information to the phase information can improve the signal-to-noise ratio of the breathing signal, thereby realizing accurate dynamic detection of the breathing rate.

[0006] To solve the above-mentioned purpose, the present invention adopts the following technical solution.

[0007] A first aspect of the present invention provides a breathing rate detection method based on a drone video stream, the method comprising:

[0008] Obtain a video stream containing the target's respiratory movement signs, and select the target's chest and abdomen in the video stream as the region of interest;

[0009] grayscale processing is performed on the region of interest, the chest and abdomen respiratory vibration region of the target object is taken as the object, time phase processing is performed on the respiratory vibration region, and the original signal is extracted;

[0010] Separating a plurality of eigenmodes of the original signal by a modal decomposition method, and fitting the plurality of eigenmodes with a respiratory waveform to obtain a respiratory mode signal as a respiratory signal;

[0011] The respiratory signal is extracted, and the respiratory rate signal is calculated using a preset respiratory rate calculation method to obtain the respiratory rate.

[0012] As an example, the step of performing time phase processing on the respiratory vibration region and extracting the original signal includes:

[0013] Determine the number of layers of the complex controllable pyramid filter group according to the size of the video stream, and construct a complex controllable filter group in multiple scale directions;

[0014] A complex controllable pyramid filter bank is used to decompose each frame of the video into multi-scale and multi-directional local phase information and amplitude information.

[0015] According to the respiratory vibration area information being concentrated at the edge of the vibration area or the edge of the image texture, the phase information is weighted in the spatial domain using the amplitude information to improve the signal-to-noise ratio;

[0016] The phase sequences of multiple scales and multiple directions of each frame are superimposed to obtain a composite signal of phase and time as the original signal, in which an interference signal is mixed.

[0017] As an example, the step of separating multiple eigenmodes of the original signal by using a modal decomposition method, and fitting the multiple eigenmodes with a respiratory waveform to obtain a respiratory mode signal as a respiratory signal includes:

[0018] Decomposing the original signal using a variational mode decomposition algorithm to obtain a plurality of eigenmodes, and fitting the plurality of eigenmodes with a respiratory waveform as candidate respiratory mode signals;

[0019] Remove the interference signal from the original signal and select the breathing mode signal with the smallest variance as the breathing signal.

[0020] As an example, the preset breathing rate algorithm includes:

[0021] Obtaining a fast Fourier transform analysis power spectrum, and determining that the frequency corresponding to the maximum power in the power spectrum is the respiratory rate; and / or

[0022] The respiratory rate signal is calculated using a Fourier iterative interpolation algorithm to obtain the corresponding respiratory rate.

[0023] As an example, the step of calculating the respiratory rate signal using the Fourier iterative interpolation algorithm to obtain the corresponding respiratory rate includes:

[0024] Calculate the power spectrum P of the respiratory signal with N samples and the position T corresponding to the maximum power;

[0025] According to the Fourier series iterative interpolation calculation, the Fourier coefficient deviation E that tends to be stable after multiple iterations is determined q ;

[0026] The calculation formula for determining the respiratory rate RR is as follows:

[0027]

[0028] Among them, f s is the video sampling rate, N is the number of samples, E q is the Fourier coefficient deviation, and N is the number of samples.

[0029] As an example, in the step of using amplitude information to perform spatial domain weighting on phase information to improve the signal-to-noise ratio, the spatial domain weighted calculation formula is:

[0030] B i (x,y,t)=B i (x,y,t)·A i (x,y,t)

[0031] Among them, the phase subband sequence is B i (x, y, t), x and y represent the horizontal and vertical positions of the pixels in the video image, t represents the tth frame image in the video stream, and the amplitude subband sequence is A i (x,y,t).

[0032] A second aspect of the present invention provides a breathing rate detection device based on drone video stream, comprising:

[0033] A video receiving module is used to obtain a video stream containing respiratory movement signs of the target object, and select the chest and abdomen of the target object in the video stream as a region of interest;

[0034] A signal extraction module is used to perform grayscale processing on the region of interest, take the chest and abdomen respiratory vibration region of the target object as the object, perform time phase processing on the respiratory vibration region, and extract the original signal;

[0035] A modal decomposition module, which is used to separate multiple eigenmodes of the original signal by using a modal decomposition method, and fit the multiple eigenmodes with a respiratory waveform to obtain a respiratory modal signal as a respiratory signal;

[0036] The calculation module is used to extract the respiratory signal and calculate the respiratory rate signal using a preset respiratory rate calculation method to obtain the respiratory rate.

[0037] A third aspect of the present invention provides a method for detecting vital signs, the method comprising the following steps:

[0038] The drone is equipped with a camera to detect and search the target area;

[0039] Use visual positioning technology to adjust the UAV flight parameters so that the UAV flies around the target object;

[0040] The drone is equipped with a camera to capture the chest and abdomen respiratory movement signs of the target object to obtain a video stream, and the above method is used to obtain the target object's respiratory rate.

[0041] A fourth aspect of the present invention provides an electronic device, comprising a processor, an input device, an output device and a memory, wherein the processor, input device, output device and memory are connected in sequence, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the above method.

[0042] A fifth aspect of the present invention provides a readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the above method.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) The example of the present invention proposes a breathing rate detection method based on a drone video stream. Considering that the time-phase signal is mixed with interference signals caused by the jitter of the drone, the original signal is decomposed using the modal method to obtain candidate breathing mode signals, and the intrinsic mode with the smallest variance is selected to realize breathing rate detection, thereby removing the interference signals caused by the shaking of the drone and suppressing noise interference. Finally, a good breathing rate waveform fitting is obtained. At the same time, the spatial domain of the amplitude information and the phase information are weighted, which can improve the signal-to-noise ratio of the breathing signal and realize accurate dynamic detection of the breathing rate.

[0045] (2) The present invention adopts an iterative Fourier coefficient interpolation method to analyze the respiratory signal, determines the Fourier coefficient deviation that tends to be stable after multiple iterations, and then calculates the respiratory rate, thereby achieving a balance between detection accuracy and time cost, and effectively improving the accuracy and robustness of respiratory rate detection in drone scenarios.

[0046] (3) In order to solve the problems that traditional contact respiratory rate detection is complicated to operate and not suitable for actual scenarios, the present invention proposes a method for detecting vital signs based on drones. This technology can realize respiratory rate detection by only using a drone camera to shoot a breathing video of the target object, thereby achieving high-efficiency, wide-range, and anti-drone jitter interference respiratory rate detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and other purposes, features and advantages of the present application will become more apparent by describing the embodiments of the present application in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps. In the accompanying drawings:

[0048] Figure 1 A flow chart of a breathing rate detection method based on drone video stream provided as an example of the present invention;

[0049] Figure 2 A schematic diagram of a drone detecting a target object provided as an example of the present invention;

[0050] Figure 3 A block diagram of a breathing rate detection device based on drone video stream provided as an example of the present invention;

[0051] Figure 4 A block diagram of an electronic device according to an embodiment of the present application is illustrated;

[0052] Figure 5 VMD modal decomposition diagram provided for the example of the present invention;

[0053] Figure 6 This is a respiratory waveform diagram taken out as an example of the present invention. DETAILED DESCRIPTION

[0054] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0055] Exemplary Methods

[0056] like Figure 1 As shown, this embodiment provides a breathing rate detection method based on drone video stream, and the method includes the following steps:

[0057] S110: Acquire a video stream containing respiratory movement signs of the target object, and select the chest and abdomen of the target object in the video stream as a region of interest.

[0058] Specifically, the target object in this example refers to a human body. Since the abdomen of a human body rises and falls when breathing, the chest and abdomen of the target object in the video stream are taken as the region of interest, and the rise and fall of the abdomen is regarded as one breath. It should be understood that the target object here can also be other animals, such as dogs, pigs, or cows.

[0059] like Figure 2 As shown, the video stream here is obtained by controlling the drone to hover around the target object, and using the camera carried by the drone to collect the abdominal movement signs of the target object. The parameter settings of the drone camera can be set to 1280*1080 resolution, frame rate 60 / second, and the generated video image is an RGB image. It should be understood that the video stream here can also be obtained in non-real time, for example, receiving video images pre-stored in a server, or receiving video images transmitted from other devices.

[0060] S120: grayscale processing is performed on the region of interest, the chest and abdomen vibration regions of the target object are taken as objects, time phase processing is performed on the respiratory vibration region, and the original signal is extracted.

[0061] Specifically, since this example records one rise and fall of the chest and abdomen of the human body in the region of interest as one breath, which is reflected as a grayscale change in the image, in order to highlight the respiratory movement information and accelerate the calculation, the color space of the acquired video is converted from the RGB domain to the Gray domain. The conversion formula is:

[0062] I gray =0.299*R+0.587*G+0.114*B (1)

[0063] In the formula, R, G, and B represent the red, green, and blue color channel matrices respectively, and I gray Represents the converted grayscale matrix.

[0064] After grayscale processing of the region of interest, spatial filtering is performed, and the chest and abdomen vibration areas of the human body are taken as the object, and the respiratory vibration area is processed for time phase. Since the drone is affected by airflow and motor vibration when hovering, the obtained time-phase signal is mixed with interference signals caused by the drone's jitter

[0065] In one embodiment, step S120 includes the following steps:

[0066] S121: Determine the number of layers of the complex controllable pyramid filter group according to the size of the video stream, and construct a complex controllable filter group of multiple scales and multiple directions.

[0067] Specifically, in this example, the complex controllable pyramid is used to perform multi-resolution processing on the image in the frequency domain. Its essence is to perform frequency domain filtering processing on the original image by constructing a set of frequency domain filters of different scales and directions.

[0068] S122: Using a complex controllable pyramid filter group, each frame image of the video is decomposed into multi-scale and converted into multi-scale and multi-directional local phase information and amplitude information.

[0069] Specifically, each frame of the video is decomposed into a complex controllable pyramid to obtain a set of frequency bands of different scales and directions. Then, each frequency band is inverse Fourier transformed and returned to the spatial domain. At this time, the pixel value obtained will become a complex number, and the spatial phase can be extracted.

[0070] For example, for a frame of image, the steps of obtaining its multi-scale and multi-directional phase spectrum baseband through the complex controllable pyramid are as follows:

[0071] (1) If the width of the input image is w, the height is h, and floor is rounded down, then the total number of scale decomposition layers N is:

[0072]

[0073] (2) The bandpass template of the complex controllable pyramid is multiplied with each directional template to obtain a layer of directional filter. The bandpass template is reduced to 1 / 4 of the original one and then multiplied with the directional template to obtain the next layer of directional filter, finally obtaining a multi-layer multi-directional filter group.

[0074] The bandpass filter of the complex controllable pyramid is obtained by multiplying the high-pass filter and the low-pass filter. The implementation formula of the high-pass filter HiMask is:

[0075]

[0076] The implementation formula of the low-pass filter LoMask is:

[0077]

[0078] Where rad represents the polar diagram, r has a value range of r∈2^[-MaxHeight:0], the step size is 1, which is used to control the effective range of the filter, MaxHeight is the number of pyramid layers, and twidt is artificially set to 1 to adjust the invalid frequency band range of the high-pass filter and the effective frequency band range of the low-pass filter.

[0079] (3) The input image is filtered using the obtained N-layer vertical filter group to obtain N-layer multi-directional frequency bands, and after inverse Fourier transform, N-layer subband sequences containing phase information and amplitude subband sequences are obtained.

[0080] S123: according to the respiratory vibration area information being concentrated on the image texture or the vibration edge, using the amplitude information to perform spatial domain weighting on the phase information to improve the signal-to-noise ratio;

[0081] Specifically, after spatial filtering, N layers of subband sequences containing phase information are obtained. For the t-th frame image of the i-th layer, the corresponding phase spectrum width is w i , height h i , the phase subband sequence is B i (x,y,t), the amplitude subband sequence is A i (x, y, t), then the phase spectrum spatial domain weighted calculation formula is:

[0082] B i (x,y,t)=B i (x,y,t)·A i (x,y,t) (4)

[0083] S124: superimposing the phase sequences of multiple scales and multiple directions of each frame to obtain a composite signal of phase and time as the original signal, wherein the original signal is mixed with an interference signal.

[0084] Specifically, the average phase signal of the 1st to Fth frames of the i-th layer subband sequence in the above steps is:

[0085]

[0086] Finally, the average phase signal of N layers is averaged to obtain the original signal, where t = 1, 2, ..., F.

[0087]

[0088] S130: separating multiple eigenmodes of the original signal by modal decomposition, and fitting the multiple eigenmodes with the respiratory waveform to obtain a respiratory mode signal as a respiratory signal;

[0089] Specifically, the original signal obtained in the above steps is first decomposed into a variational modal signal. In this example, variational modal signal decomposition is an adaptive, completely non-recursive modal variation and signal processing method. This method has the advantage of determining the number of modal decompositions. Its adaptability is reflected in determining the number of modal decompositions of a given sequence according to actual conditions. In the subsequent search and solution process, it can adaptively match the optimal center frequency and finite bandwidth of each mode, and realize the effective separation of inherent modal components and the frequency domain division of signals, thereby obtaining the effective decomposition components of a given signal, and finally obtaining the optimal solution to the variational problem. Figure 5 , where the VMD algorithm can be used to decompose the original signal extracted in the previous step into several intrinsic mode components.

[0090] Then, the respiratory signal is fitted. The eigenmode components extracted in the above steps are fitted with the general respiratory signal one by one. Different signals are uniformly normalized and initially phased. The eigenmode corresponding to the minimum variance is selected as the selected respiratory signal P(t). Its waveform is as follows: Figure 6 shown.

[0091] S140: extracting the respiratory signal, and calculating the respiratory rate signal using a preset respiratory rate algorithm to obtain the respiratory rate.

[0092] Specifically, a fast Fourier transform power spectrum is obtained, and the frequency corresponding to the maximum power in the power spectrum is the respiratory rate; and / or the respiratory rate signal is calculated using a Fourier iterative interpolation algorithm to obtain the corresponding respiratory rate.

[0093] As a preferred implementation, the respiratory rate signal is calculated using a Fourier iterative interpolation algorithm, and the steps of obtaining the corresponding respiratory rate include:

[0094] First, calculate the power spectrum P of the respiratory signal with N samples and the position T corresponding to the maximum power;

[0095] Then perform Fourier series iterative interpolation calculation: Initialize the starting deviation e 0 =0 and the number of iterations Q = 50, it is found from statistics that the Fourier coefficient deviation tends to be stable after 50 iterations.

[0096]

[0097] e k =e k-1 +r(e k-1 ) (8)

[0098] Where, d = ±0.5, e k is the Fourier series deviation after k iterations, S d Expressed as calculating e k The intermediate variable P(t) represents the breathing signal; e k-1 represents the Fourier series deviation after iteration k-1 times; j represents the imaginary unit.

[0099] The final calculated respiratory rate RR is:

[0100]

[0101] Among them, f s is the video sampling rate, N is the number of samples, E q is the Fourier coefficient deviation, N is the number of samples, e Q Represents the Fourier series deviation at 50 iterations.

[0102] The experimental results are as follows:

[0103] The human breathing rate recorded by the YX-1207 piezoelectric sensor was used as the true value, and an experimental comparison analysis was conducted with the method of extracting the breathing rate from the human face based on video. A total of 55 sets of experimental test videos were shot in the experiment, with a video length of 1 minute. The experimental settings are as follows: Figure 2 As shown, the experiment is based on the average error M e It is a performance evaluation standard, and its unit is the number of breaths per minute. It is calculated as follows:

[0104]

[0105] Where Sum is the total number of test samples, RP cal is the number of human breathing times detected by the respiratory rate detection method within the duration of the i-th video test sample, RP real It is the actual number of breaths of the human body. e It represents the error between the measured value and the true value of the respiratory rate detection method, M e The smaller the value, the higher the accuracy of the respiratory rate detection method. Comparative Example 1 uses the method disclosed in CN113052115A to extract the respiratory rate from the human face. The experimental results are shown in Table 1:

[0106] Table 1

[0107]

[0108]

[0109] It can be seen from the above table that the method of the present invention reduces the average error compared to extracting the respiratory rate from the human face, and realizes a more accurate human respiratory rate detection method.

[0110] The present invention also provides a method for detecting vital signs, the method comprising the following steps:

[0111] The drone is equipped with a camera to detect and search the target area; it mainly searches for or targets, such as human bodies.

[0112] Use visual positioning technology to adjust the flight parameters of the drone, such as visual SLAM, or use satellite navigation conditions to make the drone fly around the target object; use the camera carried by the drone to shoot the chest and abdominal respiratory movement signs of the target object to obtain a video stream, and use the above method to obtain the target object's respiratory rate.

[0113] Exemplary Devices

[0114] like Figure 3As shown, based on the above exemplary method, a breathing rate detection device based on drone video stream is also provided, comprising:

[0115] A video receiving module 20, which is used to obtain a video stream containing respiratory movement signs of the target object, and select the chest and abdomen of the target object in the video stream as the region of interest;

[0116] A signal extraction module 30 is used to perform grayscale processing on the region of interest, take the chest and abdomen vibration regions of the target object as the object, perform time phase processing on the respiratory vibration region, and extract the original signal;

[0117] A modal decomposition module 40, which is used to separate multiple eigenmodes of the original signal by modal decomposition, and fit the multiple eigenmodes with the respiratory waveform to obtain a respiratory modal signal as a respiratory signal;

[0118] The calculation module 50 is used to extract the respiratory signal and calculate the respiratory rate signal using a preset respiratory rate algorithm to obtain the respiratory rate.

[0119] Exemplary Electronic Devices

[0120] Below, reference Figure 4 The electronic device according to the embodiment of the present application is described. The electronic device can be the mobile device itself, or a stand-alone device independent of the mobile device, which can communicate with the mobile device to receive the collected input signals from the mobile device and send the selected target decision behavior to the mobile device.

[0121] Figure 4 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0122] like Figure 4 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0123] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0124] The memory 12 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the decision-making behavior decision-making method of each embodiment of the present application described above and / or other desired functions.

[0125] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). For example, the input device 13 may include various devices such as an onboard diagnostic system (OBD), a unified diagnostic service (UDS), an inertial measurement unit (IMU), a camera, a laser radar, a millimeter wave radar, an ultrasonic radar, and vehicle-to-everything (V2X). The input device 13 may also include, for example, a keyboard, a mouse, and the like. The output device 14 may include, for example, a display, a speaker, a printer, a communication network, and a remote output device connected thereto, and the like.

[0126] Of course, to simplify, Figure 4 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0127] Exemplary computer program products and computer-readable storage media

[0128] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the decision-making behavior decision-making method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0129] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0130] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the decision-making behavior decision-making method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0131] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0132] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0133] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0134] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0135] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0136] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A breathing rate detection method based on drone video stream, characterized in that: The method comprises: Obtain a video stream containing the target's respiratory movement signs, and select the target's chest and abdomen in the video stream as the region of interest; grayscale processing is performed on the region of interest, the chest and abdomen respiratory vibration region of the target object is taken as the object, time phase processing is performed on the respiratory vibration region, and the original signal is extracted; Specifically, it includes: recording one rise and fall of the chest and abdomen of the human body in the region of interest as one breath, which is reflected as a grayscale change in the image. In order to highlight the respiratory movement information and accelerate the calculation, the color space of the acquired video is converted from the RGB domain to the Gray domain; after the grayscale processing of the region of interest, spatial filtering processing is performed, and the vibration area of ​​the chest and abdomen of the human body is taken as the object, and the time phase processing of the respiratory vibration area is performed; because the drone will be affected by the airflow and motor vibration when hovering and shooting, the obtained time-phase signal is mixed with the interference signal caused by the jitter of the drone; Separating a plurality of eigenmodes of the original signal by a modal decomposition method, and fitting the plurality of eigenmodes with a respiratory waveform to obtain a respiratory mode signal as a respiratory signal; Extracting the respiratory signal, and calculating the respiratory rate signal using a preset respiratory rate calculation method to obtain the respiratory rate; The step of performing time phase processing on the respiratory vibration region and extracting the original signal comprises: Determine the number of layers of the complex controllable pyramid filter group according to the size of the video stream, and construct a complex controllable filter group in multiple scale directions; A complex controllable pyramid filter bank is used to decompose each frame of the video into multi-scale and multi-directional local phase information and amplitude information. According to the respiratory vibration area information being concentrated at the edge of the vibration area or the edge of the texture, the phase information is weighted in the spatial domain using the amplitude information to improve the signal-to-noise ratio; The phase sequences of multiple scales and multiple directions of each frame are superimposed to obtain a composite signal of phase and time as the original signal, wherein the original signal is mixed with an interference signal; The spatial domain weighted calculation formula in the step of using amplitude information to perform spatial domain weighting on phase information to improve the signal-to-noise ratio is: B i (x,y,t)=B i (x,y,t)·A i (x,y,t) Among them, the phase subband sequence is B i (x, y, t), x and y represent the horizontal and vertical positions of the pixels in the video image, t represents the tth frame image in the video stream, and the amplitude subband sequence is A i (x, y, t); after spatial filtering, we get N layers of subband sequences containing phase information. For the tth frame image of the i-th layer, the corresponding phase spectrum width is w i , height h i , the phase subband sequence is B i (x,y,t), the amplitude subband sequence is A i (x,y,t); The phase sequences of multiple scales and multiple directions of each frame are superimposed to obtain a composite signal of phase and time as the original signal, and the step of mixing the interference signal in the original signal comprises: The average phase signal of the 1st to Fth frames of the i-th layer subband sequence in the above steps is: Finally, the average phase signal of N layers is averaged to obtain the original signal, where t = 1, 2, ..., F; 2. A breathing rate detection method based on drone video stream according to claim 1, characterized in that: The step of separating multiple eigenmodes of the original signal by using a modal decomposition method, and fitting the multiple eigenmodes with a respiratory waveform to obtain a respiratory mode signal as a respiratory signal comprises: Decomposing the original signal using a variational mode decomposition algorithm to obtain a plurality of eigenmodes, and fitting the plurality of eigenmodes with a respiratory waveform as candidate respiratory mode signals; Remove the interference signal from the original signal and select the breathing mode signal with the smallest variance as the breathing signal.

3. A breathing rate detection method based on drone video stream according to claim 1, characterized in that: The preset breathing rate algorithm includes: Obtaining a fast Fourier transform analysis power spectrum, and determining that the frequency corresponding to the maximum power in the power spectrum is the respiratory rate; and / or The respiratory rate signal is calculated using a Fourier iterative interpolation algorithm to obtain the corresponding respiratory rate.

4. A breathing rate detection method based on drone video stream according to claim 2, characterized in that: The step of calculating the respiratory rate signal by using the Fourier iterative interpolation algorithm to obtain the corresponding respiratory rate comprises: Calculate the power spectrum P of the respiratory signal with a sample number of N1 and the position T corresponding to the maximum power; According to the Fourier series iterative interpolation calculation, the Fourier coefficient deviation E that tends to be stable after multiple iterations is determined q ; The calculation formula for determining the respiratory rate RR is as follows: Among them, f s is the video sampling rate, N1 is the number of samples, E q is the Fourier coefficient deviation.

5. A breathing rate detection device based on drone video stream, characterized in that: include: A video receiving module, which is used to obtain a video stream containing respiratory movement signs of the target object, and select the chest and abdomen of the target object in the video stream as a region of interest; A signal extraction module is used to perform grayscale processing on the region of interest, take the chest and abdomen respiratory vibration region of the target object as the object, perform time phase processing on the respiratory vibration region, and extract the original signal; Specifically, it includes: recording one rise and fall of the chest and abdomen of the human body in the region of interest as one breath, which is reflected as a grayscale change in the image. In order to highlight the respiratory movement information and accelerate the calculation, the color space of the acquired video is converted from the RGB domain to the Gray domain; after the grayscale processing of the region of interest, spatial filtering processing is performed, and the vibration area of ​​the chest and abdomen of the human body is taken as the object, and the time phase processing of the respiratory vibration area is performed; because the drone will be affected by the airflow and motor vibration when hovering and shooting, the obtained time-phase signal is mixed with the interference signal caused by the jitter of the drone; A modal decomposition module, which is used to separate multiple eigenmodes of the original signal by using a modal decomposition method, and fit the multiple eigenmodes with a respiratory waveform to obtain a respiratory modal signal as a respiratory signal; A calculation module, which is used to extract the respiratory signal, calculate the respiratory rate signal using a preset respiratory rate calculation method, and obtain the respiratory rate; The step of performing time phase processing on the respiratory vibration region and extracting the original signal comprises: Determine the number of layers of the complex controllable pyramid filter group according to the size of the video stream, and construct a complex controllable filter group in multiple scale directions; A complex controllable pyramid filter bank is used to decompose each frame of the video into multi-scale and multi-directional local phase information and amplitude information. According to the respiratory vibration area information being concentrated at the edge of the vibration area or the edge of the texture, the phase information is weighted in the spatial domain using the amplitude information to improve the signal-to-noise ratio; The phase sequences of multiple scales and multiple directions of each frame are superimposed to obtain a composite signal of phase and time as the original signal, wherein the original signal is mixed with an interference signal; The spatial domain weighted calculation formula in the step of using amplitude information to perform spatial domain weighting on phase information to improve the signal-to-noise ratio is: B i (x,y,t)=B i (x,y,t)·A i (x,y,t) Among them, the phase subband sequence is B i (x, y, t), x and y represent the horizontal and vertical positions of the pixels in the video image, t represents the tth frame image in the video stream, and the amplitude subband sequence is A i (x, y, t); after spatial filtering, we get N layers of subband sequences containing phase information. For the tth frame image of the i-th layer, the corresponding phase spectrum width is w i , height h i , the phase subband sequence is B i (x,y,t), the amplitude subband sequence is A i (x,y,t); The phase sequences of multiple scales and multiple directions of each frame are superimposed to obtain a composite signal of phase and time as the original signal, and the step of mixing the interference signal in the original signal comprises: The average phase signal of the 1st to Fth frames of the i-th layer subband sequence in the above steps is: Finally, the average phase signal of N layers is averaged to obtain the original signal, where t = 1, 2, ..., F; 6. A method for detecting vital signs, characterized in that: The method comprises the following steps: The drone is equipped with a camera to detect and search the target area; Use visual positioning technology to adjust the UAV flight parameters so that the UAV flies around the target object; The target object's chest and abdominal respiratory movement signs are photographed by a camera carried by an unmanned aerial vehicle to obtain a video stream, and the target object's respiratory rate is obtained using any one of the methods of claims 1-4.

7. An electronic device, characterized in that: The method comprises a processor, an input device, an output device and a memory, wherein the processor, the input device, the output device and the memory are connected in sequence, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 4.

8. A readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 4.

Images