Lung monitoring system

Abstract

A method and apparatus (100, fig. 2) is provided for tracking a user’s chest movement and comprises receiving (e.g. pre-processed) video data S102 showing the user breathing, the data comprising image

Description

TECHNICAL FIELD

[001] The invention relates to a method and device for monitoring a patient’s lung, particularly using a portable device. BACKGROUND

[002] Respiratory patterns are a tool in primary and secondary care for patients. Respiration patterns can be used to determine whether a patient is sick. For example, Figure 1a compares the breathing patterns of a normal person with those of a sick patient and a severely sick patient. As a patient’s condition deteriorates, the rate of breathes per minute increases (e.g. from 12 breaths to 30 breaths per minute) and the volume of air inspired also increases from (e.g. 6 litres to over 25 litres per minute). Spirometry is a known technique which analyses respiration lung flow and volume to determine whether patients have specific lung conditions. Examples of the patterns associated with some major illnesses are shown in Figure 1b. However, analysing these sounds often requires one-to-one and / or face-to-face presence to listen to sounds from the lungs.

[003] US2017055878A1 discloses a method and corresponding apparatus for monitoring breathing by computing a calibration signal from a first sequence of images of a user's chest to produce a calibration model. The calibration signal is representative of movement of the user's chest during a first time period during which the user is using an incentive spirometer (IS). The first sequence of images corresponds to the first time period. A method and corresponding apparatus employ the calibration model to produce a breathing information estimate about the user's breathing from a second sequence of images of the user's chest corresponding to a second time period during which the user is not using the commercially-available IS.

[004] US20140163405A1 discloses a physiological information measurement system including at least one video capture unit which captures at least one video. A calculating unit measures physiological information according to the video. The display unit shows the physiological information.

[005] US11363990B2 discloses a system and method for monitoring one or more physiological parameters of a subject. The system includes a camera configured to capture and record a video sequence including at least one image frame of at least one region of interest (ROI) of the subject's body. A computer in signal communication with the camera to receive signals transmitted by the camera representative of the video sequence includes a processor configured to process the signals associated with the video sequence recorded by the camera and a display configured to display data associated with the signals.

[006] The present applicant has recognised the need for an improved device for sensing patient information. SUMMARY

[007] According to the present invention there is provided an apparatus, system and method as set forth in the appended claims. Other features of the invention will be apparent from the dependent claims, and the description which follows.

[008] In a first approach of the present techniques there is described a computer-implemented method for tracking a user’s chest movement, the method comprising: receiving video data of the user’s chest showing the user breathing, the video data comprising a plurality of images showing the user’s chest; defining a plurality of template tracking areas in a first image of the plurality of images, each template tracking area comprising a plurality of pixels; and for each other image in the plurality of images, locating a plurality of corresponding tracking areas in the image, wherein each corresponding tracking area corresponds to one of the plurality of template tracking areas; and for each of the plurality of corresponding tracking areas, determining a difference in location between the corresponding tracking area in the image and a previous image from the plurality of images. The method comprises determining, using the determined location differences, movement of the user’s chest; and outputting information on the determined movement of the user’s chest.

[009] The plurality of images may be images that are taken consecutively in time. Thus, the difference in location between tracking areas may be determined between consecutive images in the plurality of images. The plurality of images may be taken over a time frame which includes at least one breath of the user.

[0010] Each tracking area comprises a plurality of pixels in the image and the number of pixels defines the size of the tracking area, for example there may be between 100 to 400 pixels. Each tracking area may have a square shape, such as 10x10 pixels or 20x20 pixels. Alternatively, the tracking areas may have any suitable shape such as an approximately circular shape, or a rectangular shape, or an approximately elliptical shape.

[0011] Defining the plurality of template tracking areas may comprise defining the location of each template tracking area within the first image. This may comprise defining a target area in the first image. Using a plurality of tracking areas to track a user’s chest movement allows tracking of individual areas of a user’s chest. The target area may thus be a substantial portion of a user’s chest or a specific area within a user’s chest to more accurately target a part of a user’s lung. The method may further comprise determining whether each tracking area moves synchronously with all other tracking areas and this may allow determination of lung function of or irregularities originating in different parts of the lung.

[0012] Defining the plurality of template tracking areas may further comprise defining a grid of tracking areas. The grid may be arranged to minimise overlap between tracking areas, for example by specifying a minimum number of pixels between adjacent tracking areas in the grid. There may be at least four tracking areas, for example there may be a grid of 4x4 or 8x8 tracking areas. The distance between adjacent tracking areas may be the same so that the tracking areas are evenly spaced within the grid.

[0013] Locating a plurality of corresponding tracking areas in each of the other images may comprise, for each other image in the plurality of images, placing a search area at each location in the other image corresponding to a location of one of the plurality of template tracking areas. The search area may be placed at a location so that an indicative (or initial) tracking area within the search area is at the same location as the corresponding template tracking area. The location of the search area which may be determined by co-ordinates of at least one pixel within the search area or within template tracking area. The search areas may be located at the same locations in each image, or may be updated after a fixed or predetermined number of images have been processed (e.g. 30 or so images).

[0014] The tracking area is likely to have moved from one image to the next and thus each search area comprises a peripheral area around an indicative tracking area, wherein the peripheral area comprises a fixed number of pixels in each direction around the indicative tracking area. The indicative tracking area has the same size and shape as the corresponding template tracking area and is located at the same location. The search area thus comprises the indicative tracking area and peripheral area. The fixed number may be the same (e.g. five) in each direction or different in different directions. To locate the tracking area within the search area, the method may comprise defining multiple subareas within the search area, wherein each subarea has the same size and shape as the corresponding template tracking area; calculating, for each of the defined subareas, a difference score between pixel values of the template tracking area and pixel values of each defined subarea; and selecting, as the corresponding tracking area, the subarea which most closely matches the template tracking area using the calculated difference score. For example, the subarea having the lowest difference score may be selected as the tracking area.

[0015] The difference score may be calculated using any suitable method, for example using: A / ^\R’i-Rd + + l^-Fd) t = o where Ri, Gi and Bi correspond to the red, blue and green colour channels of i-th pixel within the template tracking area, Ri’, Gi’ and Bi’ to to the red, blue and green colour channels of i-th pixel within each subarea, and N is the total number of pixels within the template tracking area or subarea.

[0016] The video data may comprise all three of the RGB (red, green, blue) colour channels, or it may comprise only one colour channel. Additionally or alternatively, the video data may be infrared video data, i.e. the video may have been recorded at infrared wavelengths. A pixel value may be a value based on one or more colour channels, or based on a value of the infrared videodata. It will be appreciated that the difference score may also be calculated using infrared values. The difference score may be alternatively termed a similarity score.

[0017] The method may comprise determining a confidence level in the located plurality of corresponding tracking areas before determining a difference in location between the corresponding tracking area in the image and a previous image (and doing any further processing). Confidence in the located plurality of corresponding tracking areas may be quantified in an appropriate way. For example, when difference scores are used as described above, the difference scores of the tracking areas between subsequent images may be monitored and compared to a typical difference in pixel values. Only when the confidence level is above a confidence threshold, is further processing for that image continued, e.g. continuing with the determining a difference step. When the confidence is low, the image is rejected and the next, subsequent image in time, is processed.

[0018] The use of search areas which are larger than the tracking areas helps to compensate for movement by the user because such movement means that the physical location on a user’s chest of the tracking areas (and hence in the images) is also likely to have moved. It will be appreciated that updating the search areas after a certain number of images have been processed may provide further compensation to correct for user movement. Merely as an illustrative example, the user may have moved towards the top left direction of the image and thus the subarea that most closely matches the template tracking area is always located in the top left corner of the search area. By recentring the search area on the tracking area, i.e. by moving the search area towards the left corner of the image, the search areas are more likely to include the tracking area which matches the template tracking area.

[0019] In addition to, or alternative to updating the search areas, the method may further comprise updating the template tracking areas when a fixed number of images in the plurality of images have been processed. Updating the template tracking areas can be important in order to ensure that the template tracking areas adapt to changing image conditions, such as, for example, a change in lighting or any other changes. The method may comprise determining that corresponding tracking areas have been located in a threshold number of other images; calculating a set of pixel values for pixels within each corresponding tracking area and updating the plurality of template tracking areas based on the calculated set of pixel values for the corresponding tracking areas. Updating the plurality of template tracking areas may comprise taking a weighted average of pixel values for each template tracking area and the calculated sets of pixel values. The average may be weighted such that the current template tracking areas are awarded a greater weight than the recently located tracking areas. In this way, if the pixel values of the tracking areas that have just been found deviate largely from the pixel values of the current template tracking areas, this will only have a small impact on the pixel values of the template tracking areas.

[0020] Determining a difference in location between the corresponding tracking area in the image and a previous image comprises calculating a difference in position between at least one pixel in each tracking area.

[0021] Receiving video data of a user breathing may comprise receiving pre-processed video data of a user breathing. The video data may be pre-processed using at least one of bandpass filtering and smoothing. The bandpass filtering may be done using any suitable filter, e.g. a second order Butterworth filter. Similarly, the smoothing may be done using any suitable technique, e.g. a Savitzky-Golay filter.

[0022] The method may further comprise using the determined information on the movement of the user’s chest to extract information about the user’s breathing and / or lung function. The method may further comprise outputting a loop diagram of the user’s breathing, the loop diagram representing a user’s lung function. Lung function may be expressed using any suitable metric, including at least one or more of respiratory rate, respiratory pattern (e.g. over a fixed time frame such as a minute), forced expiratory volume (measured as FEV1, FEV2, FEV3 and / or FEV6), forced vital capacity, displacement loop volume and displacement loop times.

[0023] Outputting a loop diagram may further comprise: obtaining a breathing signal based on the determined information on the movement of the user’s chest; determining stationary and zero crossing points of the breathing signal; computing a gradient signal from the breathing signal; determining stationary and zero crossing points of the gradient signal; analysing the stationary and zero crossing points of the breathing signal and the gradient signal to identify each individual breath cycle; and plotting each complete breath cycle as a loop diagram. The stationary points (e.g. the peaks and troughs) of the gradient signal correspond to the points of maximum positive velocity and the points of maximum negative velocity in the breathing signal. Similarly, the zero crossing points in the gradient signal correspond to the peaks and troughs in the breathing signal. The method may further comprise rejecting any implausible stationary and zero crossing points. Implausible stationary and zero crossing points may be adjacent points with the same label (e.g. two peaks adjacent each other) or may be points which are too close to one another.

[0024] In a second approach to the present techniques, there is described an apparatus for tracking a user’s chest movement, the apparatus comprising: an image capture device for capturing video data of the user’s chest that shows the user breathing; and at least one processor, coupled to memory, and arranged for: carrying out the method described above. The apparatus may further comprise a user interface for directing the user to position the image capture device for capturing the video of the user’s chest, e.g. in the target area as defined above.

[0025] The apparatus may be a user device, such as a smartphone, tablet computer, smartwatch, computer, laptop or any other suitable user device which comprises both the image capture device and a processor for locally processing the captured video data. Alternatively, the apparatus may comprise a user device which comprises the image capture device and a communication module to send the captured video data to a server or any other suitable device comprising a processor. For example, the apparatus may be integrated into other devices and / or vehicles, such as cars, lorries, vans, busses, or any other vehicle that can travel across roads, water or in the air.

[0026] In a further approach to the present techniques, there is described a non-transitory carrier carrying a computer program which when implemented on a computer, e.g. the user device and / or remote processing device, causes the computer to implement the method described above. BRIEF DESCRIPTION OF DRAWINGS

[0027] For a better understanding of the invention, and to show how embodiments of the same may be carried into effect, reference will now be made, by way of example only, to the accompanying diagrammatic drawings in which:

[0028] Figure 1a is a graph comparing lung performance in healthy and unhealthy patients;

[0029] Figure 1b is a graph showing lung characteristics in a healthy patient and in various patients having different lung conditions;

[0030] Figure 2 is a block diagram of an apparatus for tracking a user’s chest movement;

[0031] Figure 3 is a flowchart showing a pipeline for analysing a video showing a user breathing;

[0032] Figure 4 is a diagram showing how the user’s chest movement is tracked using a grid of tracking areas;

[0033] Figure 5 is a flowchart showing how the grid of tracking areas is used to determine the user’s chest movement;

[0034] Figure 6a shows a guide outline in an application interface which is used to direct the user to position an image capture device of the apparatus to capture the video data of the user’s chest;

[0035] Figures 6b to 6e illustrate how the location of a tracking area is determined from one image of the video to the next to determine the user’s chest movement;

[0036] Figure 7 is a flowchart showing how the detected movement can be transformed into output breathing information;

[0037] Figure 8a shows a breathing signal of the user plotting movement against time;

[0038] Figure 8b shows a smoothed breathing signal derived from Figure 8a;

[0039] Figure 8c shows a gradient signal of the smoothed breathing signal of Figure 8b;

[0040] Figures 9a to 9e show loop diagrams corresponding to individual breaths of a user which may be extracted from a gradient signal such as that shown in Figure 8b;

[0041] Figures 10a to 10c are raw breathing signal, smoothed breathing signal and a gradient signal for a user performing slow spirometry;

[0042] Figure 10d is a loop diagram based on Figure 10c;

[0043] Figures 11a to 11c are raw breathing signal, smoothed breathing signal and a gradient signal for a user performing pursed lip breathing. DETAILED DESCRIPTION OF DRAWINGS

[0044] Figure 2 is a block diagram of the components of a system which may be used to track a user’s chest movement and hence determine lung function. The system comprises a user device 100, e.g. a smart phone, mobile phone, tablet or similar computing or electronic device which is typically handheld. The user device 100 comprises the standard components such as a processor 102, a user interface 104, storage 106 and a communications interface 108. The user interface 104 may be any standard interface, including a touch sensitive display screen, voice input, keyboard input etc. Similarly, the storage 106 may be conventional memory which may include RAM and / ROM. The user device 100 comprises an image capture device 110 which may be any standard image capture device, such as a camera.

[0045] The video data may be analysed by the processor 102 on the user device 100. Alternatively, the raw video data may be transferred, e.g. using the communications interface 108 to a processing module 120. For example, the communications interface 108 may connect via a wireless connection, e.g. via the Internet. The processing module 120 may be remote from the user device, e.g. located in a different location from the user device or may be at least partially local to the user device (e.g. located within the user device). Using a remote processing module 120 may allow access to more powerful resources so that the raw data may be processed more quickly. A local processor has the advantage that it can be used in areas without any wireless connections. Accordingly, both local and remote processing may occur. The processing module 120 may be implemented in hardware e.g. as a computing device such as a server or may be implemented in the cloud 132. As schematically illustrated in Figure 2, the processing module 120 comprises a processor 122, at least one filter 124 and a communications interface 126 via which the processing module 120 can receive data from the user device 100 and / or send messages to the user device 100.

[0046] As shown in Figure 2, the system may also comprise a repository 130 into which raw and / or processed data can be stored. The repository 130 may be a secure database for the video files. Data may be transferred from the processing module 120 to the repository 130 or from the cloud 132. Further processing may be done in the cloud 132 and thus the processing module 120 is able to communicate with the cloud 132. The repository 130 may include a secure personalised data store in which the uploaded data may only be accessed by use of a key or other personalised identifier.

[0047] In some instances, a trained machine learning, ML, model (not shown) may optionally be used to analyse the video data. The video data may be analysed to extract information about a user’s breathing. The information about a user’s breathing may further be used to determine if the user’s breathing is within a normal range, and information about the user’s breathing may be output to the user using the user interface 104. The methods used to analyse the video data are described in more detail below.

[0048] The user device 100 may also be any other device suitable for capturing video data of a user’s chest and using the captured video data to track a user’s chest movement. For example, the user device 100 could be a vehicle, such as a car, bus, van, lorry, aircraft, boat or any other vehicle that is equipped with an image capture device. In this example, the image capture device of the vehicle may capture video data of a driver or other occupant and analyse this video data. The vehicle may comprise a processor coupled to memory, capable of doing this analysis. The processor may analyse the video data to track the driver’s or other occupant’s chest movement. This analysis may then be used to infer information about the driver’s or other occupant’s breathing, which may in turn be used to deduce physiological information about the driver or other occupant, such as whether they are tired or not. The user device 100, when part of a vehicle, may then additionally be configured to alert the driver or other occupant if an unfavourable physiological state is detected. For example, if the system detects that the driver is tired and should take a break, it can alert the driver and / or other occupants of the vehicle of this fact.

[0049] Figure 3 is a flowchart showing a pipeline for analysing a video showing a user breathing. The method shown in Figure 3 may be implemented by the apparatus of Figure 2. In a first step, video data of the user’s chest showing the user breathing is received S102. The video data may comprise a plurality of images showing the user’s chest. The plurality of images may be images that are taken consecutively in time, such that they show at least one breath of the user over time. The video data may comprise all three of the RGB (red, green, blue) colour channels, as is conventional. Alternatively, only one of the colour channels may be used in the following analysis. For example, the G (green) colour channel may carry more relevant information than the other two colour channels and thus the following analysis may be performed on the G channel of the video data only.

[0050] In a next step, a plurality of tracking areas are defined in the first image of the plurality of images S104 and their location in the first image may be determined. Each tracking area may comprise a plurality of pixels in the image. The location may be defined by the pixel row and column location of at least one of the outermost pixels within the tracking area. For example, the location may be defined by the row and column location of the top-left pixel of each tracking area. The tracking areas may have any suitable size, for example, there may be between 100 to 400 pixels in each tracking area. The shape of each tracking area may be defined by the arrangement of the pixels. For example, the pixels may be arranged in a grid (e.g. 20x20 pixels or 10x10 pixels) and thus each tracking area may have a square shape. Alternatively, the tracking areas may have any suitable shape such as an approximately circular shape, or a rectangular shape, or an approximately elliptical shape.

[0051] The plurality of tracking areas may be arranged in any suitable pattern, or may be arranged randomly. The tracking areas may be arranged to avoid overlap between the tracking areas, e.g, by defining a minimum number of pixels between each tracking area. Figure 4 is a schematic diagram of a user 200 showing a grid of tracking areas 206 which are arranged across a target area, in this case a user’s chest 202. In this example, there are sixteen tracking areas 206 arranged in a 4x4 grid 204 but it will be appreciated that this is just one example and a grid of 8x8 tracking areas, or any other suitable grid arrangement may be used. In this example, the tracking areas 206 are spaced equally across the user’s chest area, although this is not necessary and may change if a particular area of the user’s chest is being monitored.

[0052] It will be appreciated that the size of the grid may be determined by a number of factors, such as desired accuracy, or resource constraints of the apparatus on which the method may be implemented. For example, a grid comprising a larger number of tracking areas may correspond to greater accuracy but may also require more computational resources. Thus, the size and nature of the grid may be adapted to achieve a desired accuracy, or to function within the limitations of a resource constrained device.

[0053] It will also be appreciated that there is a relationship between the size and nature of the grid and the size and spacing of the tracking areas within the grid. For example, for efficient detection of tracking areas in subsequent images, it may be necessary for the tracking areas to comprise a certain number of pixel values. If the grid contains a larger number of tracking areas, this may mean that each tracking area contains fewer pixels to avoid any overlap between the tracking areas. This might make detecting tracking areas in a subsequent image more difficult and less accurate. Thus, it will be appreciated that there may be a balance to be struck between higher resolution (i.e. a larger number of tracking areas) and maintaining accuracy.

[0054] Returning to Figure 3, the next step S106 is to determine a difference in location for each tracking area in subsequent images. The difference in location of tracking areas between two consecutive images of the plurality of images corresponds to movement of the user’s chest during the time frame in which the images were captured. By tracking the locations of the plurality of tracking areas across the plurality of images comprising the video data, the user’s chest movement may be determined.

[0055] The next step S108 is to determine the movement of a user’s chest using the determined location differences. It will be appreciated that any movement determined from a two-dimensional image is limited to two-dimensional movement, for example in the x and y directions using standard coordinates. Chest movement due to breathing corresponds to three-dimensional movement with the movement in two dimensions (x and y) being captured directly in the images and the movement of the chest in the perpendicular z direction (third dimension) being approximated using the two-dimensional movement captured in the image. It will be appreciated that the z-direction carries the most information, followed by one of the dimensions in the x-y plane which may carry more information on the perpendicular movement than the other dimension. For example, the y-axis may be defined as an axis between a user’s head and feet and the x-axis may be defined as an axis from one shoulder to the other. In this example, the y-direction may carry more information on the three-dimensional movement than components of the movement in the x-direction. This is because as the lung inflates, any point on the surface of the lung moves both outwards and upwards, similar to an inflating balloon. Thus, movement in the y-direction is linked to movement along the z-axis.

[0056] Two different types of chest movement may be determined. The first is a localised measurement which tracks the movement over time separately for each of the tracking areas and hence separately for each area of the chest covered by a tracking area. The second is a global measurement which is an average measurement of the whole chest over time that may be calculated by taking an average of the localised measurements of the tracking areas for each image. That is, the global measurement provides information on how the whole of the user’s chest has moved overtime.

[0057] Both of these output measurements can provide valuable insight into a user’s breathing. The global measurement (which may also be termed average displacement) allows determination of a user’s breathing rate and general respiratory function. The localised measurements for the tracking areas enable detailed tracking of a user’s breathing and lung function with respect to different portions of a user’s lung. That is, the localised measurements enable comparison between the breathing pattern that is observed for different parts of the lung. For example, the localised measurements can be compared to determine whether the separate tracking areas are moving synchronously or asynchronously. One or more tracking areas which are moving asynchronously may be indicative of irregularities originating in different parts of the lung, for example one part of a lung not inflating. In a final step S110, information on the determined movement of the user’s chest for the plurality of images comprising the video data is output.

[0058] Figure 5 is a flowchart giving more detail on how the grid of tracking areas may be defined and then refined. As described above, the tracking areas may be equal in size and may be spaced equally across the user’s chest to form a grid of tracking areas. When tracking a user’s chest, the first step S202 is to determine which area in the first image shows the target area, i.e. the user’s chest. This may be done automatically, e.g. using any suitable technique such as edge detection. Alternatively, guidance may be given to the user to ensure the correct area is tracked.

[0059] Figure 6a is a schematic illustration of an application interface on the user device which is used to direct the user to correctly position an image capture device of the apparatus to capture the video data of the user’s chest. A guide outline 604 is shown on the user interface of the apparatus 600. The guide outline can be any useful outline which ensures that the target area 606 (in this case targeting the user’s chest) is captured by the image capture device. In this example, the guide outline 604 shows a user’s head, shoulder and arm outline and when the user positions the apparatus 600 such that the guide outline 604 is aligned with the user’s head and arms, the user’s chest area will be positioned inside the target area 606.

[0060] Returning to Figure 5, the tracking areas in the first image may then be defined at step S204 based on the alignment of the target area with the first image. Each tracking area may be positioned such that it covers a subarea of the area showing the user’s chest to obtain as much information as possible on the user’s breathing. As described above in relation to step S104 of Figure 3, an image is typically a grid of pixels and each tracking area, at least initially, may be defined by the location of the outermost pixels.

[0061] The tracking areas defined in the first image may then be saved as template tracking areas S206. The template tracking areas may be used to determine a location of a tracking area in a subsequent image. In a subsequent image, pixel values of the template tracking areas may be compared with pixel values in the subsequent image to determine the movement of a user’s chest.

[0062] Figure 5 describes one method for determining the difference in location of the tracking areas in each image in the video data (i.e. the step of S106 of Figure 3). First, the locations of the tracking areas of a previous image are used to find the tracking areas in a current (i.e. a later or subsequent) image.

[0063] Figures 6b to 6e illustrate how the location of a tracking area is determined from one image of the video to the next to determine the user’s chest movement. Figure 6b shows the location 704 of a tracking area 702 of a first (or previous) image 700. As shown, the location is indicated by the coordinates of the top-left pixel in the tracking area. In Figure 6c, the corresponding tracking area 702’ is at the same location 704’ in the next image (i.e. the current image) 700’. In practice, the tracking areas are likely to move from one image to the next, For example due to movement by the user. The area covered by the located tracking areas 702’ may thus merely be an indicative or initial tracking area which is not the best match for the original tracking area.

[0064] Returning to Figure 5, the next step is to locate search areas based on these initial tracking areas to ensure that there is consistent and accurate mapping from one image to a subsequent image in the video. Such a search area 706 is shown in Figure 6d. Each search area 706 covers the initial tracking area (i.e. the expected location of the tracking area) together with a peripheral area 708 around the initial tracking area. Thus, each search area comprises pixel values at the locations of pixels in and around the corresponding tracking area in the previous image. The search areas 706 may be placed at the same locations in each image for a fixed or predetermined number of images, merely as an example around 30 images. After the fixed number of images have been processed, the locations of the search areas and / or the initial tracking areas may be updated.

[0065] Each peripheral area 708 may be a uniform area around each initial tracking area. In other words, each search area may have an area which is greater than the initial tracking area by a fixed number, e.g. five pixels or any other number, for example, between five to ten pixels or ten to fifty pixels in each direction around each initial tracking area, as is shown in Figure 6d. It will also be appreciated that the number of pixels may depend on the absolute size of an image that is being analysed. That is, the absolute pixel size of the search area may be greater for an image of higher resolution, and smaller for an image of lower resolution. Alternatively, the search area may be non-uniform in size and shape.

[0066] Returning to Figure 5, in a next step S210, each search area is searched by defining each subarea which is within the search area and which is the same size and shape as the template tracking area 710 so that accurate comparisons can be made. At step S212. the pixel values of the pixels within each subarea are compared to the pixel values of the pixels within the corresponding template tracking areas 710 to select a subarea 708 of each search area that most closely matches the corresponding template tracking area. The subarea 708 that most closely matches the template tracking area 710 may be termed the final tracking area and may be determined using any suitable method.

[0067] For example, the final tracking area may be found by assigning a distance score to each subarea within the search area. The distance score may be a measure of similarity between each subarea in the search area and the template tracking area. The distance score may be computed as the intensity difference of each R, G and B channel values between the subarea and the template tracking area. That is, a difference in pixel value for each channel used may be calculated. The subarea with the smallest difference between these pixel values may then be selected as the final tracking area. For example, the difference score may be calculated as: N ^Ri-Rd + IC / -GJ + i = o where R, G and B correspond to the red, blue and green colour channels of the template tracking area, R’, G’ and B’ to values of corresponding pixels in each subarea, i is the i-th pixel within the template tracking area or subarea, and N is the total number of pixels within the template tracking area or subarea.

[0068] Additionally or alternatively, the final tracking area may be found by calculating, for each combination of the subarea 708 and template tracking area 710, a cross-correlation value. The crosscorrelation value is a measure of which subarea most closely matches the template tracking area. Hence, the cross-correlation value also is a measure of the most likely location of each final tracking area within the search area. It will be appreciated that selecting an appropriate size and shape for the search area helps to ensure that the final tracking areas which are found are relevant.

[0069] Step 212 of Figure 5 is illustrated in Figure 6e. The comparison of each subarea 708 in the search area 706 may, for example, be achieved by “moving” the subarea 708 across the whole of the search area 706 one pixel location at a time and comparing each of these subareas to the template tracking area 710.

[0070] Returning to Figure 5, the next step S214 is to determine whether there is sufficient confidence in the corresponding tracking areas which have been detected in the current image for subsequent calculations to be meaningful. The tracking areas may be the tracking areas which were located in step S208 or the refined final tracking areas which were output from step S212. When there is confidence that the tracking areas can be used, the method proceeds to further process the current image at step S216. Otherwise the current image is rejected at step S218.

[0071] Confidence in the detected tracking area may be quantified in an appropriate way. For example, the difference score calculated above may be monitored and compared to a typical difference in pixel values. A typical difference in pixel values may, for example, be on the order of 101. If the difference that is detected is on the order of 103 instead, the outlier image may be discarded and not processed any further. Alternatively, an average of all previous difference scores may be calculated (i.e. for all final tracking areas in all images before the image currently being processed) to determine a threshold for discarding an image. Alternatively, the average difference score may be calculated from a fixed number of previous images (e.g. five). The difference score of each final tracking area or an average difference score for all of the final tracking areas of the current image may then be compared with the average difference score and the current image may be rejected when the compared difference scores are sufficiently different, e.g. the difference score each final tracking area of the current image or the average difference score is twice as large as the average difference score.

[0072] When there is confidence in the detected tracking areas, a difference in location of each tracking area between the two consecutive images is calculated at step S216. The difference in location may be the distance between the location of a tracking area in the current image and the location of the corresponding tracking area in the earlier image. For example, the difference in location may be calculated by calculating a difference in location between the pixel at the centre of the tracking area of the current and previous images. Alternatively, the difference in location may be calculated using any other suitable method, for example, by comparing the locations of any other pixels, such as the top-left or top-right pixel of the two tracking areas.

[0073] Optionally, the template tracking areas may be updated at step S220. Such an update may be done after each image is processed as described above or more efficiently after multiple images have been processed. This may be done by including information about the tracking areas of the recently processed images in the template tracking areas. For example, each template tracking area may be updated by taking a weighted average of the pixel values for the current saved template tracking area and the pixel values for the corresponding tracking areas in each of the images since the last update was performed. The average may be weighted such that the current template tracking areas are awarded a greater weight than the recently located tracking areas. In this way, if the pixel values of the tracking areas that have just been found deviate largely from the pixel values of the current template tracking areas, this will only have a small impact on the pixel values of the template tracking areas. This will minimise the effect of the update on finding tracking areas in subsequent images. However, updating the template tracking areas can be important in order to ensure that the template tracking areas adapt to changing image conditions, such as, for example, a change in lighting or any other changes.

[0074] At step S222, there is a determination as to whether there are further images in the video data to process. Steps S208 to S220 above are repeated for each image in the plurality of images until the last image is processed. Once all images have been processed, the movement of a user’s chest may be determined at step S224. In other words, the method has reached step S108 of Figure 3.

[0075] Figure 7 is a flowchart showing how the detected movement can be transformed into output breathing information. In a first step S300, a raw breathing signal is obtained, for example as described above.

[0076] An example breathing signal is plotted in Figure 8a which plots the average movement in the y direction over time wherein the average movement is calculated across all tracking areas in each image which is not rejected. Thus, the breathing signal may also be termed a movement signal. Time is a continuous number from 0 and may be calculated from the number of frames (i.e. images) which are included divided by the number of frames per second. The signal measuring the y-direction movement is typically inverted (i.e. negative values as shown) so that inhalation results in an increase in the signal strength and exhalation results in a decrease. Figure 8a shows examples of individual breaths 502, 504, 506, 508 of a user and the fifth breath is significantly deeper than other breaths.

[0077] Returning to Figure 7, the next step S302 is to apply one or more bandpass filters, e.g. by using appropriate high and / or low band filters such as second order Butterworth filter and / or any other suitable filters. The input may be signal such as shown in Figure 8a which covers multiple frequency bands and the output signal covers a reduced set of frequency bands. For example, only frequencies between 0.1 and 2Hz may be let through this way. After applying a filter, the signal may be smoothed as shown at step S304, for example using a Savitzky-Golay filter, or any other suitable filter such as a second order polynomial filter. The inputs to the smoothing may be the filtered signal and a window length (measured in frames per second fps).

[0078] Figure 8b shows a smoothed signal derived from the raw movement signal in Figure 8a and thus the same symbols are used. The first few individual breaths are 502’, 504’, 506’, 508’. In addition to being smoothed to reduce noise in the signal, the values of the y movement are shifted so that not all values are negative. The zero value is set at a value which is approximately the median or average of the values of the peaks and troughs.

[0079] Returning to Figure 7, the next step S306 is to calculate a gradient of the filtered, smoothed signal. The inputs to the smoothing may be the smoothed signal, a window length (measured in frames per second fps - this may be a 1 second window) and the details of the method used. There are various options for calculating the gradient, including use a median filter, specifying how many frames to use and a specifying a blanking time (i.e. minimum distance between detected peaks). An example process may be to invert the input signal, e.g. by subtracting it from 255, then applying a three-frame median filter and finally computing the gradient for each frame (i.e. each image) by using a least squares fit for the frames data. The output is the gradient signal and the first few frames will be zero.

[0080] Returning to Figure 7, the next step S308 is to apply temporal smoothing to the gradient of the breathing signal, for example by applying a Savitzky-Golay filter as described above. In a next step S310, the stationary and zero crossing points of the gradient signal are found. The stationary points are the peaks and troughs in the gradient signal. The peaks may be found from the gradient signal and the troughs may be found as the peaks in the inverted gradients using standard techniques. Similarly, the zero crossing points may be found using standard techniques. Each of the peaks, troughs and zero crossing points may be labelled with an index which indicates their location in the gradient signal. These indices may be the sorted in ascending order (by time). The stationary points and the zero crossing points of the gradient signal are the used to locate the zero crossing and stationary points in the breathing signal at step S312. The output from this step may be an array of frame indices indicating the location of the peaks, troughs and zero crossing points in the gradient signal which may be termed the “gradjndices”, an array of frame indices indicating the location of the peaks, troughs and zero crossing points in the original signal which may be termed the “rawjndices”, and an array of the labels for each index which may be termed “labels”.

[0081] Figure 8c shows a gradient of the smoothed breathing signal in Figure 8b. The first few individual breaths are 502”, 504”, 506”, 508”. Stationary points are indicated by red star symbols, peaks by orange dot symbols and troughs by green triangle symbols. The corresponding location of these symbols are also shown in Figures 8a and 8b which show that the red star symbols are the peaks and troughs of Figures 8a and 8b, the orange dot symbols are the points of maximum positive velocity of Figures 8a and 8b and the green triangle symbols are the points of maximum negative velocity. In other words, the peaks and troughs of the gradient signal correspond to the points of maximum positive velocity and the points of maximum negative velocity in the breathing signal. Similarly, the zero crossing points in the gradient signal correspond to the peaks and troughs in the breathing signal.

[0082] Returning to Figure 7, the next step S314 is to reject any implausible stationary and zero crossing points. Implausible stationary and zero crossing points may be adjacent points with the same label (e.g. two peaks adjacent each other) or may be points which are too close to one another. The inputs may include the breathing signal output, the gradient signal, the “gradjndices”, the “rawjndices” and the “labels” returned from step S312, the original labels used in steps S310 and S312 for the “peak”, “trough” or “zero crossing point” and the distance (i.e. number of frames) which was used in the gradient calculation.

[0083] This step involves analysing the pattern of stationary and zero crossing points found in the gradient signal and in the breathing signal. For example, when there are two adjacent zero crossing points in the gradient signal, the corresponding stationary points in the breathing signal are analysed. If both the stationary points are peaks, the peak with the highest value will be retained in the breathing signal and the corresponding zero crossing point in the gradient signal is retained. Similarly, if both the stationary points are troughs, the trough with the lowest gradient value will be retained. When there two adjacent peaks which are not separated by the normal pattern of other stationary and zero crossing points, e.g. peak - crossing point - trough - crossing point - peak, the peak with the highest gradient value will be retained and the corresponding zero crossing point in the breathing signal is retained. Similarly, when consecutive troughs are not separated by the normal pattern, the trough with the lowest gradient value will be retained.

[0084] The output from step S314 will typically be a pattern of stationary and zero crossing points which follow the normal pattern, e.g. zero to peak to zero to trough to zero. The output may also include the “grad_indices”, the “rawjndices” and the “labels” to assist in further analysis. The pattern of stationary and zero crossing points may then be analysed to detect or isolate “complete” breaths within the breathing signal at step S316. This may be done by identifying each complete breath cycle as one which has the pattern zero to peak to zero to trough to zero and rejected other patterns. Any breath cycles which are longer than a threshold time (i.e. width) may be rejected. The output may be an array of indices indicating the temporal location of the start of each breathing cycle.

[0085] Based on the detected breaths, a user’s respiratory rate may then be calculated at step S318. The respiratory rate simply is the number of breaths in a minute. The respiratory rate may thus be calculated by counting how many breath cycles have been detected and by computing the total duration as the time of the first detected breath cycle to the end of the last detected breath cycle. The total duration is then divided by the number of breath cycles to obtain the respiratory rate. The final step S320 is to generate the desired outputs from the information calculated above.

[0086] Examples of such outputs are shown in Figures 9a to 9e which are loop diagrams also known as displacement loops, corresponding to individual breaths of the breathing signal of a user. A loop diagram is traditionally obtained by performing spirometry and provides a measure of a user’s lung function. A loop diagram may be generated by using a smoothed gradient of an individual breath, as shown in Figure 8c. The x-axis in a loop diagram represents time and the y-axis represents velocity. An inhalation period (period between minimum and maximum of breathing signal) of an individual breath is plotted on the negative side of the y-axis and the smoothed gradient breathing signal is inverted as necessary, so that it has negative values. The inhalation period is plotted in negative time direction, i.e. such that the last point of the inhalation period is plotted at the origin. An exhalation period (period between maximum and minimum of breathing signal) of an individual breath is plotted on the positive side of the y-axis and the smoothed gradient signal is inverted as necessary, so that it has positive values. The loop diagrams of Figures 9a to 9d are equivalent to loop diagrams obtained using conventional spirometry such as the loop diagram of Figure 1b.

[0087] Different sectors of the loop diagrams may be indicators for different aspects of lung function and thus the loop diagrams may be analysed to extract information about each breath. For example, the loop diagram may be divided into quadrants to analyse lung function as illustrated in Figure 9e. The duration (time) between consecutive fiducial points (i.e. consecutive stationary or zero crossing points - zero to peak, peak to zero etc) may be calculated. The volume (V1, V2, V3, V4) between consecutive fiducial points may additionally or alternatively be calculated as the trapezoidal area between the gradient and time axes. The output may comprise an array of the timings between consecutive fiducial points and / or an array of the volumes between consecutive fiducial points. Additionally, or alternatively, the shape of the positive section of the loop diagram, close to the origin may provide an indication of the severity of COPD.

[0088] While taking a video of themselves breathing, the user may be instructed to perform specific breathing exercises or breathe in a specific pattern to obtain a loop diagram that is of maximum diagnostic use. For example, the above mentioned methods may be used to obtain the Force Expiratory Volume (FEV1) which can indicate how severe a patient’s chronic obstructive pulmonary disease (COPD) is. FEV1 may be obtained by instructing the patient to perform maximum voluntary ventilation (i.e. a patient breathing as deeply and quickly as possible for about 12 seconds). Similarly, the above techniques may be used to obtain the Force Volume Capacity (FEC). Additionally, having a patient perform a set of breathing manoeuvres may help obtain accurate data, as there is less movement that is not due to breathing. Hence, the problem of having to filter out non-breathing related movement can be minimised.

[0089] As examples of the output from different types of breathing being performed by the user, Figure 10a shows the raw breathing signal from a user who is performing slow spirometry. The raw signal of Figure 10a has been processed as described above to generate the filtered, smoothed breathing signal shown in Figure 10b and the gradient signal shown in Figure 10c. An example loop diagram is shown in Figure 10d.

[0090] As an alternative example, Figure 11a shows the raw breathing signal from a user who is performing pursed lip breathing. The raw signal of Figure 11a has been processed as described above to generate the filtered, smoothed breathing signal shown in Figure 11 b and the gradient signal shown in Figure 11c.

[0091] The diagrams shown in the above mentioned figures may be processed using a machine learning, ML, or Artificial Intelligence, Al, algorithm. These algorithms may be used to perform an analysis of a patient’s breathing pattern and evaluate their lung function.

[0092] Additionally, the user’s face may also be detected, and the recording used to obtain a value for the user’s heart rate. To detect changes in blood volume in a user’s face, it is important to choose an area of the face that allows for easy monitoring of blood flow, i.e. that is transparent enough for blood flow to be visible in this area. This can be done by detecting blood volume changes in, for example, the cheek and forehead areas.

[0093] At least some of the example embodiments described herein may be constructed, partially or wholly, using dedicated special-purpose hardware. Terms such as ‘component’, ‘processor’, ‘module’ or ‘unit’ used herein may include, but are not limited to, a hardware device, such as circuitry in the form of discrete or integrated components, a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), which performs certain tasks or provides the associated functionality. In some embodiments, the described elements may be configured to reside on a tangible, persistent, addressable storage medium and may be configured to execute on one or more processors. These functional elements may in some embodiments include, by way of example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Although the example embodiments have been described with reference to the components, modules and units discussed herein, such functional elements may be combined into fewer elements or separated into additional elements.

[0094] Various combinations of optional features have been described herein, and it will be appreciated that described features may be combined in any suitable combination. In particular, the features of any one example embodiment may be combined with features of any other embodiment, as appropriate, except where such combinations are mutually exclusive. Throughout this specification, the term “comprising” or “comprises” means including the component(s) specified but not to the exclusion of the presence of others.

[0095] Attention is directed to all papers and documents which are filed concurrently with or previous to this specification in connection with this application and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference.

[0096] All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and / or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and / or steps are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.

[0097] Although a few preferred embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes and modifications might be made without departing from the scope of the invention, as defined in the appended claims. The invention is not restricted to the details of the foregoing embodiment(s). The invention extends to any novel one, or any novel combination, of the features disclosed in this specification (including any accompanying claims, abstract and drawings), or to any novel one, or any novel combination, of the steps of any method or process so disclosed.

Claims

1. A computer-implemented method for tracking a user’s chest movement, the method comprising:5 receiving video data of the user’s chest showing the user breathing, the video data comprisinga plurality of images showing the user’s chest;defining a plurality of template tracking areas in a first image of the plurality of images, each template tracking area comprising a plurality of pixels;for each other image in the plurality of images:10 locating a plurality of corresponding tracking areas in the image, wherein eachcorresponding tracking area corresponds to one of the plurality of template tracking areas;determining that corresponding tracking areas have been located in a threshold number of other images, and updating the template tracking areas in the first image by:calculating a set of pixel values for pixels within each corresponding tracking area; 15 andupdating the plurality of template tracking areas in the first image based on the calculated set of pixel values for the corresponding tracking areas; andfor each of the plurality of corresponding tracking areas, determining a difference in location between the corresponding tracking area in the image and a previous image from the 20 plurality of images;determining, using the determined location differences, movement of the user’s chest; and outputting information on the determined movement of the user’s chest.

2. The method as claimed in claim 1, wherein defining the plurality of template tracking areas 25 comprisesdefining a target area in the first image; anddefining a grid of tracking areas across the target area,wherein the grid specifies a minimum number of pixels between adjacent tracking areas in the grid.

303. The method as claimed in claim 2, wherein the template tracking areas have a uniform size and are uniformly arranged in the grid.

4. The method as claimed in any one of the preceding claims, wherein locating a plurality of 35 corresponding tracking areas in each of the other images comprises, for each other image in the plurality of images,placing a search area at each location in the other image corresponding to a location of one of the plurality of template tracking areas and for each search area:23 08 24defining multiple subareas within the search area, wherein each subarea has the same size and shape as the corresponding template tracking area;calculating, for each of the defined subareas, a difference score between pixel values of the template tracking area and pixel values of each defined subarea; and5 selecting, as the corresponding tracking area, the subarea which most closely matchesthe template tracking area using the calculated difference score.

5. The method as claimed in claim 4, wherein the search area comprises a peripheral area around an indicative tracking area having the same size and shape as the corresponding template tracking 10 area, wherein the peripheral area comprises a fixed number of pixels in each direction around the indicative tracking area.

6. The method as claimed in claim 4 or claim 5, wherein the difference score is calculated using: N+ + ^-B^i = 015 where Ri, Gi and Bi correspond to the red, blue and green colour channels of i-th pixel within the template tracking area, Ri’, Gi’ and Bi’ to to the red, blue and green colour channels of i-th pixel within each subarea, and N is the total number of pixels within the template tracking area or subarea.

7. The method of any one of the preceding claims, comprising before determining a difference in 20 location between the corresponding tracking area in the image and a previous image, determining a confidence level in the located plurality of corresponding tracking areas; and only when the confidence level is above a confidence threshold, continuing with the determining a difference step.25 8. The method as claimed in any preceding claim, wherein updating the plurality of templatetracking areas comprises taking a weighted average of pixel values for each template tracking area and the calculated sets of pixel values.

9. The method as claimed in any preceding claim, wherein determining a difference in location 30 between the corresponding tracking area in the image and a previous image comprises calculating a difference in position between at least one pixel in each tracking area.

10. The method as claimed in any preceding claim, wherein receiving video data of a user breathing comprises receiving pre-processed video data of a user breathing.3511. The method as claimed in any preceding claim, wherein outputting information on the determined movement of the user’s chest comprises outputting a loop diagram of the user’s breathing, wherein the loop diagram representing a user’s lung function.

12. The method as claimed in claim 11, wherein outputting a loop diagram further comprises: obtaining a breathing signal based on the determined information on the movement of the user’s chest;5 determining stationary and zero crossing points of the breathing signal;computing a gradient signal from the breathing signal;determining stationary and zero crossing points of the gradient signal;analysing the stationary and zero crossing points of the breathing signal and the gradient signal to identify each individual breath cycle; and10 plotting each complete breath cycle as a loop diagram.

13. An apparatus for tracking a user’s chest movement, the apparatus comprising: an image capture device for capturing video data of the user’s chest that shows the user breathing; and15 at least one processor which is configured to track a user’s chest movement using the capturedvideo data as set out in any one of the preceding claims.

14. The apparatus of claim 13, further comprising a user interface for directing the user to position the image capture device to capture a video of the user’s chest;2015. A non-transitory carrier carrying a computer program which when implemented on a computer, causes the computer to implement the method of any one of claims 1 to 12.23 08 24