Assessment of subject's areas of interest
Patent Information
- Application Number
- JP2023530193
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-11-20
- Filing Date
- 2021-11-18
- Publication Date
- 2024-11-21
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing methods for analyzing skin conditions require expert intervention and are time-consuming and costly, lacking accuracy when using artificial intelligence techniques.
A system that allows users to annotate and measure their own skin regions of interest using sensors and image capture devices, associating sensor data with user-provided information to facilitate self-analysis.
Enables accurate and efficient skin analysis without expert intervention, reducing time and cost by allowing users to identify and label their skin features, with potential for real-time feedback and treatment adjustments.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to assessing a region of interest in a subject, and more particularly to assessing a region of interest in a subject using sensor data relating to the region of interest. [Background technology]
[0002] Many personal care and health activities are performed by people at home. In some cases, such activities may be performed on a person's skin and may involve obtaining data indicative of skin parameters for further analysis. Analysis of such data may be performed automatically using artificial intelligence techniques, including identifying specific areas of a person's skin from which to obtain sensor data and analyzing the obtained data.
[0003] Analysis of the data is sometimes performed by a medical professional or skin care professional, who can analyze data obtained about a person's skin or examine images of a person's skin to identify features such as pimples, skin conditions, ingrown hairs, moles, etc. However, capturing data and images and providing them to a professional for further analysis can be a time-consuming, complex, and expensive process. Summary of the Invention [Problem to be solved by the invention]
[0004] Therefore, what is desired is a system that can perform analysis of a person's skin without the need for a skin expert and without the use of artificial intelligence techniques that may lack the required accuracy. [Means for solving the problem]
[0005] To address at least some of the above-mentioned problems associated with existing analysis techniques, the inventors of the present invention recognized that people can perform some level of analysis themselves. Accordingly, embodiments of the present invention disclosed herein enable a user to provide annotations (e.g., provide an indication of the nature of the region of interest) regarding a region or regions of interest on their body. For example, a user may position themselves in front of a camera to capture an image of a portion of the skin on their face. The user may indicate the location of a region of interest (e.g., a blemish, mole, blemish, wrinkle, etc.), which may then be recorded in the image captured by the camera. However, in addition, by recording the identity of the region of interest, the user may also provide an indication of the nature of the region of interest (e.g., an annotation). In accordance with the present invention, by acquiring sensor data from a sensor, parameters of the region of interest are also recorded. In this manner, an image of the region of interest may be recorded along with an indication of the nature of the region of interest and details of the measurable parameters. The recorded data may be used and processed in various ways, as defined herein.
[0006] According to certain aspects, a computer-implemented method for assessing a region of interest in a subject is provided, the method comprising: receiving an image including a portion of the subject; receiving an indication of a location of a region of interest (ROI) on the portion of the subject; receiving sensor data from a sensor indicative of a measurable parameter of the ROI; receiving an indication of a property of the ROI from the subject via a user interface; generating an association between the received sensor data and the property of the ROI; and providing at least one of the image, the sensor data, the indication of the property of the ROI, and the generated association for further processing.
[0007] The measurable parameters include measurable skin parameters.
[0008] In some embodiments, the received image may include a first facial landmark feature (feature structure) and a second facial landmark feature structure, and the method further includes mapping the received image onto a three-dimensional wire mesh model based on a position of the first facial landmark feature structure relative to the second facial landmark feature structure.
[0009] The method may further comprise providing the sensor data and the indication of a property of an ROI as training data inputs for a predictive model, the predictive model being configured to determine a property of a given ROI based on sensor data indicative of a measurable parameter of the given ROI. In some embodiments, the method may further comprise training the predictive model using the provided sensor data and the provided indication of a property of the ROI.
[0010] The method may, in some embodiments, further comprise providing an indication of the nature of the ROI and the location of the ROI for presentation on the representation of the subject.
[0011] The method may further include adjusting an operating parameter of the personal care device based on the received sensor data.
[0012] In some embodiments, the method may further include comparing the received sensor data to a threshold value, and in response to determining that the received sensor data meets or exceeds the threshold value, generating at least one of: an alert to be provided to the subject; feedback to be provided to a healthcare professional; and a control signal to be provided to a personal care device.
[0013] In some embodiments, the method may include storing the received image, an indication of the location of the received ROI, the received sensor data, and an indication of the nature of the received ROI in a storage device.
[0014] The method, in some embodiments, may further include comparing the received sensor data with previously stored sensor data. In some embodiments, the method may include generating, based on the comparison, at least one of: an alert to be provided to the subject; feedback to be provided to a medical professional; a control signal to be provided to a personal care device; and a treatment profile for the subject.
[0015] The method may further include operating an image capture device to capture the image.
[0016] According to further particular aspects, there is provided a computer program product having a non-transitory computer readable medium having computer readable code embodied therein, the computer readable code being configured, when executed by a suitable computer or processor, to cause the computer or processor to perform the steps of the methods disclosed herein.
[0017] According to further particular aspects, there is provided an apparatus for assessing a region of interest in a subject, the apparatus comprising a processor configured to perform the steps of the methods disclosed herein.
[0018] According to further particular aspects, there is provided a system for assessing a region of interest of a subject, the system comprising: an image capture device for capturing an image including a portion of the subject; a region of interest (ROI) selector unit that enables a user to provide an indication of a location of an ROI on the portion of the subject; a sensor configured to acquire sensor data indicative of a measurable parameter of the ROI; a user interface configured to receive an indication of a property of the ROI from a user; and a processor; the processor is configured to: generate association information between the acquired sensor data and the indication of a property of the ROI; and provide at least one of the image, the sensor data, the indication of a property of the ROI, and the generated association information for further processing.
[0019] In some embodiments, the ROI selector unit may comprise at least one unit selected from the group including: a handheld pointing device; an optical pointing device; a finger-worn pointing device; a gesture interpretation unit for interpreting gestures made by a user; and a gaze interpretation unit for interpreting the user's gaze directed at a displayed representation of the ROI.
[0020] The sensor may, in some embodiments, comprise a sensor selected from the group including: an imaging device; a sensor for measuring the reflectance spectrum of a ROI; a moisture sensor; and a sebum measuring device.
[0021] These and other aspects will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief explanation of the drawings]
[0022] [Figure 1] FIG. 1 is a schematic diagram of an example system for assessing a region of interest in a subject. [Figure 2] FIG. 2 is a flowchart of an example method for assessing a region of interest in a subject. [Figure 3] FIG. 3 is a flowchart of another example method for assessing a region of interest in a subject. [Figure 4] FIG. 4 is a schematic diagram of another example of a system for assessing a region of interest in a subject. [Figure 5] FIG. 5 is a schematic diagram of an example of an apparatus for assessing a region of interest in a subject. [Figure 6] FIG. 6 is a schematic diagram of an example of a processor in communication with a computer-readable medium. DETAILED DESCRIPTION OF THE INVENTION
[0023] Exemplary embodiments will now be described, by way of example only, with reference to the drawings in which:
[0024] The embodiments disclosed herein provide a mechanism by which a user can provide input regarding their own physical characteristics, such as, for example, input regarding their skin characteristics. Identifying a person's skin characteristics is typically performed by a professional, such as a skin care specialist, dermatologist, or some other medical professional. However, many features that appear on a person's body are easily identifiable by the person, and the present invention allows such identification to be performed by the end user, thereby avoiding the need for a professional to spend the time and expense of examining images of the person at a later time and attempting to identify skin characteristics.
[0025] Examples disclosed herein refer to a region of interest (ROI) of a subject. As used herein, the term "region of interest" (ROI) is intended to refer to any area of a subject (e.g., a region of a subject's skin) that contains an area of interest, such as a feature to be monitored, examined, or further investigated. Examples of features or ROIs include, but are not limited to, pimples, blemishes, lesions, areas of acne, pigmented spots, areas of redness, areas of inflamed skin, cuts, dead skin, scars, moles, wrinkles, hair, and ingrown hairs.
[0026] It is envisioned that a user (i.e., a subject) may use the systems or devices disclosed herein in the course of performing a personal care activity while observing or examining their own skin (e.g., the skin of their face) in a mirror or interactive mirror, or in a live image (e.g., video image, etc.) captured by a camera and rendered on a display (e.g., a smartphone or tablet computer, etc.). While observing the mirror image or rendered image of their skin, the user positions themselves within the camera's field of view and indicates the location of an ROI on their skin so that the camera captures the location of the ROI relative to other features of the user (e.g., the user's facial features, such as the nose, eyes, and / or mouth). In addition, the user provides input via a user interface identifying the nature of the ROI, effectively providing a label or annotation that can be associated with the ROI captured by the camera. Measurements of parameters related to the ROI can also be performed using one or more sensors, and the measured parameters can also be associated with the labeled ROI. The data captured by the camera, supplied by the user, and / or measured using the sensors may then be used for various purposes, as described herein. For example, in one scenario, the data may be stored so that it can be compared with previously stored data and any changes in the ROI may be monitored.
[0027] Embodiments will now be described in detail with reference to the drawings. A first aspect of the present invention relates to a system. FIG. 1 is a schematic diagram of an example of a system 100. The system 100 may include a system for assessing an ROI of a subject 102 (e.g., a user of the system). The system 100 includes an image capture device 104 for capturing an image including a portion of the subject 102. The image capture device 104 may include a two-dimensional image capture device or a three-dimensional image capture device. For example, the image capture device 104 may be configured to capture an image of at least a portion of the face of the subject 102. Typically, an image captured using the image capture device 104 includes some skin of the subject 102. In the example shown in FIG. 1 , region 106 represents an area of the subject's skin. A dotted box 108 represents a portion of the subject 102 captured using the image capture device 104 in this example. It will be appreciated that different portions of the subject 104 may be imaged by positioning and / or aiming the image capture device 104 differently. In this example, portion 108 of subject 102 includes ROI 110 that subject 102 wishes to image (ie, capture an image of) and analyze.
[0028] Image capture device 104 may include any device capable of capturing an image or a series of images (e.g., video footage), and may be a standalone imaging device or may form part of another device or apparatus. For example, image capture device 104 may comprise a standalone camera or a camera in a smartphone, tablet computer, laptop computer, or interactive mirror. An interactive mirror (sometimes called a smart mirror) is a unit that, in addition to functioning as a mirror showing a user a mirrored image of themselves, can also display information to the user. Information such as text, images, and video can be displayed on the display portion of the interactive mirror, which may be located, for example, behind a mirrored (or partially mirrored) panel or a mirrored (or partially mirrored) surface. In this way, the display screen or a portion thereof can be seen through the mirrored portion, and thus the user can simultaneously see their own mirrored image and the information presented on the display screen.
[0029] System 100 also includes an ROI selector unit 112 for enabling subject 102 to provide an indication of the location of an ROI (e.g., ROI 110) on portion 108 of the subject. ROI selector unit 112 can take several different forms and generally provides a mechanism for enabling subject 102 to indicate the location or position of an ROI so that the location of the ROI will be visible in the captured image. In its simplest sense, according to one example, ROI selector unit 112 may comprise a pointing device (e.g., a handheld pointing device) that subject 102 uses to point to the ROI while image capture device 104 is capturing the image. This may be achieved, in some examples, by subject 102 pointing a finger toward the ROI. Indicating the location of the ROI may, in some embodiments, be achieved by the subject making a particular gesture with their hand and / or fingers. In such cases, ROI selector unit 112 may include a gesture interpretation unit for interpreting gestures made by the subject. For example, a user can make a one-finger pointing gesture to indicate the exact location of the ROI, or a two-finger pointing gesture to indicate the edges of the ROI (e.g., by marking a virtual bounding box around the ROI). In some examples, the subject 102 can wear a finger-worn device on their finger or a thumb-worn device on their thumb (e.g., a thimble-like device or a device that can be strapped or otherwise attached to a finger) that is used to aim or point at the ROI. The ROI selector unit 112, in some embodiments, can include an optical pointer, such as a laser pointer or a device that emits light at a particular wavelength band or wavelength, such that the light emitted by the device illuminates or is visible on the ROI. In other examples, eye tracking or gaze tracking can be used to select or identify the location of the ROI.For example, the ROI selector unit 112 may comprise a device such as a gaze interpretation unit configured to track the subject's eyes or gaze in such a way that the subject can indicate the location of the ROI on their skin by directing their gaze towards the ROI in their mirror image or on a rendered image or representation of themselves and the ROI on a display. Such a device may be referred to as a virtual pointer.
[0030] More generally, the ROI selector unit 112 may comprise at least one unit selected from the group including a handheld pointing device, an optical pointing device; a finger-worn pointing device, a gesture interpretation unit for interpreting gestures made by the subject, and a gaze interpretation unit for interpreting the gaze of the subject's eyes towards a displayed representation of the ROI.
[0031] As previously mentioned, the image capture device 104 can capture an individual image (e.g., a photograph) or a series of images or frames (e.g., video footage). In examples where the image capture device 104 is configured to capture a series of images, indicating the location of an ROI can be achieved by moving the ROI selector unit 112 over the ROI to indicate the size (e.g., width, length, or height) of the ROI or by indicating the boundaries of the ROI. For example, the subject 102 can use the ROI selector unit 112 to indicate the location and length of a wrinkle on their forehead by placing the ROI selector unit at one end of the wrinkle and moving the unit along the wrinkle to the other end. In this manner, it may be possible to measure or determine the length and / or depth of the wrinkle from the captured images.
[0032] In some examples, the image capture device 104 may be configured to capture an image including the portion of the subject 102 and an indication of the location of the ROI on the portion of the subject (e.g., as indicated by the subject using the ROI selector unit 112).
[0033] System 100 further comprises a sensor 114 configured to acquire data indicative of a measurable parameter of the ROI. The sensor 114 can be used to measure several different parameters of the ROI. For example, sensor 114 is used to acquire data indicative of a measurable skin parameter. In some embodiments, sensor 114 may comprise a non-contact sensor or a sensor forming part of a non-contact measurement unit that can acquire data about the ROI without contacting the ROI itself. In other embodiments, sensor 114 may comprise a contact sensor or a sensor forming part of a contact measurement unit that is configured to acquire data while in contact with the ROI.
[0034] The sensor 114 configured to acquire data without contacting the ROI may include an imaging device or imaging sensor, such as an RGB (i.e., red, green, and blue) camera, a multispectral imager, a hyperspectral imager, or a fluorescence imager. The imaging device / sensor may be configured to measure a parameter or characteristic of the ROI, such as reflectance, without necessarily acquiring an image of a portion of the subject. Thus, the sensor 114 may include a sensor for measuring the reflectance spectrum of the ROI. An RGB camera may be used to measure the color of the ROI, for example, to provide an indication of skin redness. In some examples, the image capture device 104 comprises or is used as an RGB camera, such that image data acquired using the image capture device can be analyzed to determine measurable parameters of the ROI. A multispectral imager can be used to acquire imaging data (e.g., reflectance data) at wavelengths outside the visible wavelength range, e.g., in the infrared and ultraviolet wavelength ranges, while a hyperspectral imager can be used to acquire imaging data (e.g., reflectance data) at wavelengths outside the infrared and ultraviolet wavelength ranges. A fluorescence imager can be used to acquire data indicative of the fluorescence of the ROI. Such data may be used, for example, to analyze or identify acne.
[0035] The sensor 114 configured to acquire data while in contact with the ROI may include, for example, a moisture sensor (e.g., a skin moisture sensor), or a sebum measuring device or seborrheometer / sensor for measuring the level of sebum within the ROI (e.g., skin), which may indicate the amount of oil within the skin.
[0036] In some examples, the sensor 114 may form part of the ROI selector unit 112. For example, a handheld unit may include one or more sensors 114 for acquiring data indicative of one or more measurable parameters of the ROI. In one example, the sensor 114 may be located at the end of a pen-like unit configured to be held by the subject 102 and contacted with the ROI to acquire the data. The pen-like unit may also include a laser source for projecting a laser onto the ROI, thereby also functioning as the ROI selector unit 112.
[0037] System 100 also includes a user interface 116 configured to receive an indication of the nature of the ROI from the subject. User interface 116 allows a user to provide annotations or contextual information about the ROI, and more specifically, about the data acquired using sensor 114. User interface 116 may include any interface, such as a user interface device that allows a subject to provide input indicating the nature of the ROI, for example, by labeling the ROI with a name or description of what the subject understands it to be. Thus, user interface 116 may include a keyboard for typing an indication of the nature of the ROI, a microphone that allows the subject to speak the indication, a button or series of buttons each associated with a different indication (e.g., each button associated with a different type of ROI) that allows the subject to select an indication by pressing one of the buttons, or a touchscreen display configured to display multiple options or ROI labels from which the subject can select using the touchscreen. In other embodiments, a particular ROI selector unit 112 and / or a particular sensor 114, or a particular accessory for that ROI selector unit and / or sensor, may be configured for use in association with a particular ROI, such that use of the particular ROI selector unit, sensor or accessory provides an indication of the nature of that ROI.
[0038] In some embodiments, user interface 116 may comprise the user interface of another device, such as a smartphone or tablet computer, while in other embodiments the user interface may form part of or be incorporated into components of system 100. For example, user interface 116 may be provided on image capture device 104, on ROI selector unit 112, and / or on sensor 114, thereby facilitating subject 102 to provide an indication of the nature of the ROI, provide an indication of the location of the ROI, and possibly simultaneously with, or shortly after, operating a sensor to acquire data from the ROI. Combining the features of system 100 into one component, or at least a few components, may allow the subject to more conveniently operate the various components.
[0039] The system 100 also includes a processor 118. The processor 118 may be in communication with the image capture device 104, the sensor 114, and the user interface 116. In some embodiments, the processor 118 may also be in communication with the ROI selector unit 112. The processor 118 is configured to generate an association between acquired sensor data and an indication of the nature of the ROI. For example, the processor 118 may store acquired data regarding the ROI along with an indication of the nature of the ROI. In this manner, any correlation between particular sensor data or particular measurable parameters and particular types of ROI can be recognized. For example, a high level of sebum may be determined to correspond to acne. Because the indication of the nature of the ROI is provided by the subject 102, the processor 118 (or other systems, such as machine learning models, as described below) can learn over time which parameters correspond to different types of ROI. In some embodiments, processor 118 may be configured to generate an association between the acquired sensor data, an indication of the nature of the ROI, and an appearance of the ROI obtained from an image captured using image capture device 104. The generated association may be stored in a storage device (e.g., a database), for example, along with the captured image or images, the indication of the location of the ROI, the acquired sensor data, and / or the indication of the nature of the received ROI.
[0040] The processor 118 is also configured to provide at least one of the image, the sensor data, the indication of the nature of the ROI, and the generated association information for further processing. The data may be used in a variety of ways, as described in more detail below. Once collected and acquired, the data may be used for a variety of purposes, including, for example, medical analysis, medical or personal care treatment, or optimization of device operating parameters.
[0041] In some embodiments, processor 118 may provide other functions. For example, processor 118 may be operatively coupled to and configured to operate one or more components of system 100. For example, processor 118 may control image capture device 104 to capture images, operate sensor 114 to measure measurable parameters of the ROI (i.e., record data), operate an ROI selector unit to record an indication of the location of the ROI, and / or operate a user interface to present a series of options to subject 102 regarding the nature of the ROI for selection by the subject and receive a selection from the user interface.
[0042] A second aspect of the present invention relates to a method. FIG. 2 is a flowchart of an example method 200. Method 200 may include a computer-implemented method and may be considered a method for assessing a region of interest in a subject. In some embodiments, one or more steps of method 200 may be performed using a processor, such as processor 118. Method 200 includes, at step 202, receiving an image including a portion 108 of subject 102. The image may be captured using image capture device 104 and then transferred (transmitted) to processor 118. The method includes, at step 204, receiving an indication of a location of an ROI on portion 108 of subject 102. The indication of the location of the ROI may be received from ROI selector unit 112, for example, from an instruction made by subject 102 indicating the ROI. In some embodiments, steps 202 and 204 of method 200 may be combined, such that step 202 includes receiving an image including a portion of the subject and an indication of a location of an ROI on the portion of the subject. For example, image capture device 104 may capture an image of a portion of a subject that includes an ROI and also includes an indication of its location (e.g., by a user pointing to the ROI). In embodiments in which steps 202 and 204 are not combined, steps 202 and 204 may be performed simultaneously, such that the indication of the location of the ROI is provided by subject 102 at the same time that the image of the portion of the subject is captured.
[0043] The method 200 further includes receiving sensor data indicative of a measurable parameter of the ROI from the sensor 114, at step 206. The sensor 114 may take the form of any of the sensors described herein, and the sensor data obtained using the sensor may be transferred directly or indirectly from the sensor to the processor 118.
[0044] Method 200 includes, at step 208, receiving an indication of the nature of the ROI from subject 102 via user interface 116. In other words, subject 102 provides a label, tag, or name for the ROI based on their understanding of what the ROI is. The indication can be provided to user interface 116 via any of the techniques described herein, which can provide the indication directly or indirectly to processor 118.
[0045] Method 200 includes generating association information between the received sensor data and the properties of the ROI at step 210. As previously mentioned, the association information may include stored records containing both the sensor data for a particular ROI and the properties (e.g., type) of the ROI. When method 200 is repeated for multiple ROIs on subject 102, or for multiple subjects, a database of association information can be built from which relationships between particular parameters (i.e., sensor data) and particular types of ROI can be determined.
[0046] Method 200 includes providing at least one of the image, sensor data, indication of the nature of the ROI, and generated association information for further processing at step 212. Further processing can be performed using the same processor (e.g., processor 118) used to perform method 200, or the data can be transferred to or acquired by a different processor or group of processors configured to further process the data. In the simplest sense, further processing of the data (step 212) can include storing at least one of the image, sensor data, indication of the nature of the ROI, and generated association information in a storage medium such as a memory.
[0047] 3 is a flowchart of another example method 300. Method 300 may include a computer-implemented method and may also be considered a method for assessing a region of interest in a subject. Method 300 may include one or more steps of method 200 described above, and one or more steps of method 300 may be performed using a processor, such as processor 118 described herein.
[0048] According to some embodiments, method 300 may further include operating an image capture device (e.g., image capture device 104) to capture an image at step 302. In this manner, a processor (e.g., processor 118) that performs the steps of methods 200, 300 may be operatively coupled to and control the operation of the image capture device.
[0049] Once an image is acquired by image capture device 104 (step 302) and received (e.g., by processor 118) in step 202, it may be possible to evaluate the ROI and generate association information between the data and the properties of the ROI without performing any additional processing. However, in some embodiments, additional processing of the image data captured by image capture device 104 may be performed to improve the quality of the output of the methods and systems disclosed herein. In some embodiments, the received image may include a first facial landmark feature and a second facial landmark feature. Thus, in such embodiments, the image captured using image capture device 104 comprises an image including at least a portion of a subject's head or face. A facial landmark feature refers to a feature of a subject's face that can be recognized by appropriate computer software and used for facial recognition techniques, and more generally, to determine the relative positions of facial features and the general orientation of the subject's head. Facial landmark features may include, for example, the corners and / or edges of the mouth, nose, eyes, eyebrows, or ears. Other features of the face or head can also be used as facial landmark features.
[0050] At step 304, the method 300 may include mapping the received image to a three-dimensional mesh model (e.g., a wire mesh model) based on the position of the first facial landmark feature structure relative to the second facial landmark feature structure. The three-dimensional wire mesh model may include, for example, a standard model of a human head. The first and second facial landmark feature structures may be used to map the image to the mesh model to convert the two-dimensional image into a three-dimensional model. In some embodiments, the image capture device 104 may be configured to capture a three-dimensional image of a portion of the subject 102; in such cases, mapping the image to the mesh model may be simplified and may not be required. It will be appreciated that, although two facial landmark feature structures may be required to determine the distance between features and, therefore, to predict the distance between other facial landmark feature structures, more accurate mapping of the image to the three-dimensional model may be achieved when multiple facial landmark feature structures are used. Accordingly, the subject 102 may be encouraged to move to a position that enables the image capture device 104 to capture as many facial landmark feature structures as possible. The result of this mapping step (i.e., step 304) is the generation of a reference face map, relative to which the location of the ROI may be defined. In some embodiments, a previously mapped 3D model may be used or updated using newly captured images.
[0051] As previously mentioned, at least one of the image, sensor data, indication of the nature of the ROI, and generated association information is provided for further processing (step 212), and various processing techniques that may be implemented are described below. One or more of the processing techniques described below may be performed individually or simultaneously, and various techniques are shown as alternatives in Figure 3 within the dotted box representing step 212.
[0052] At step 306, method 300 may include providing the sensor data and the indication of the nature of the ROI as training data input for a predictive model. The predictive model may be configured to determine the nature of a given ROI based on sensor data indicative of measurable parameters of the given ROI. Such a predictive model may comprise a machine learning model or algorithm, including, for example, an artificial neural network, a deep neural network, a regression tree, or a classifier. In some embodiments, the data provided at step 306 may be used as initial training data for training a predictive model, while in other embodiments, the data may be used to retrain a previously trained predictive model to improve the model and thus its output (e.g., its accuracy). Thus, step 308 of method 300 may include training a predictive model using the provided sensor data and the provided indication of the nature of the ROI.
[0053] Method 300 may include, at step 310, providing an indication of the location and nature of the ROI for presentation on the representation of subject 102. For example, if subject 102 is using an interactive mirror, the subject's mirror image may include the presentation, and the interactive mirror may present the ROI on the mirror image of the subject's face along with a label indicating what the ROI is (e.g., blemish, acne, wrinkle, etc.). In other embodiments, the subject's representation may include a photo-realistic image (e.g., a photograph) of subject 102, a generic image or drawing of a face, or a computer-generated representation such as an avatar. The representation of subject 102 may be presented or displayed to the subject on another device, such as a smartphone or tablet computer, or on another computing device, such as a wearable device (e.g., a smartwatch). Displaying the ROIs (e.g., the location and nature of the ROI) on the representation of subject 102 may allow the subject to know which ROIs have already been identified and therefore recorded (e.g., along with generated association information).
[0054] Method 300 may include, at step 312, comparing the received sensor data to a threshold value. Each different type of sensor may have a corresponding threshold value to which the data should be compared. In some embodiments, the threshold value may vary depending on the subject's details (e.g., the subject's age, gender, medical history, etc.). The threshold value may be stored in a storage device (e.g., memory) associated with system 100 and accessible by processor 118 so that the comparison can be made. In some embodiments, the threshold value may be selected or set based on acceptable health limits. For example, if sensor 114 detects that the redness of the ROI exceeds a defined threshold level of redness, this may indicate a medical condition requiring medical intervention. Method 300 may include, at step 314, generating at least one of: an alert to be provided to the subject; feedback to be provided to a medical professional; and a control signal to be provided to a personal care device in response to determining that the received sensor data meets or exceeds the threshold value. An alert to be provided to the subject 102 may include, for example, a notification or alert that a particular parameter (i.e., a parameter measured using the sensor) is at a level that requires monitoring of the ROI. In some embodiments, the alert to the subject 102 may include a notification that the parameter is at an appropriate level for administering a treatment or the next treatment in a treatment plan. Thus, the alert may include a reminder to the subject 102 to administer the next treatment in a treatment course if the parameter value is determined to be at a required level (i.e., meets a threshold). Feedback may be generated for provision to a medical professional when a measurable parameter is detected to have exceeded a level at which a cause for concern may exist. For example, if the sensor data is such that the ROI may be indicative of a particular medical condition (e.g., cancer), feedback (e.g., an alert) may be generated and provided to a medical professional.
[0055] In some examples, system 100 may be used in conjunction with a personal care device, such as a skin treatment device. The parameter of the ROI measured using the sensor may result from the application of treatment using the skin treatment device. The level of the parameter may determine that an adjustment should be made to the skin treatment device (e.g., reducing the power output of the device). In such cases, step 314 may include generating a control signal to be provided to the device to take the appropriate action.
[0056] Method 300 may include storing the received indication of the location of the ROI, the received sensor data, and the received indication of the nature of the ROI in a storage device at step 316. As previously mentioned, such storage device (e.g., memory) may form part of system 100 and / or may be accessible by a processor (e.g., processor 118) executing methods 200, 300.
[0057] Method 300 may include, at step 318, comparing the received sensor data with previously stored sensor data. The previously stored sensor data may include, for example, the data stored in storage in step 316. Method 300 may include, at step 320, generating at least one of an alert to be provided to the subject; feedback to be provided to a medical professional; a control signal to be provided to a personal care device; and a treatment profile for the subject based on the comparison. As previously described, an alert to the subject, feedback to a medical professional, and a control signal to a personal care device may be generated if it is determined that the new sensor data is sufficiently different from the previously stored sensor data that action should be taken. For example, the comparison may reveal that a particular parameter of the ROI has significantly deteriorated and therefore requires medical attention. Conversely, in an example where the subject 102 is applying a treatment to the ROI, the data comparison may reveal that a particular parameter of the ROI is improving (e.g., the level of redness is decreasing) and therefore a control signal should be provided to a personal care device (e.g., the treatment device being used for the treatment) to reduce the effectiveness of the treatment device (e.g., by reducing the power output of the device).
[0058] In one embodiment, the comparison 318 can be used to generate a treatment profile for the subject 102. Such a profile can be specific to the subject 102 and can include details about the subject's skin, and in particular, a measurable parameter(s) of the subject's skin. In one such example, the comparison can reveal that the subject's skin has become oilier, which can be recorded in a treatment profile for use by a skin treatment specialist or beauty therapist in developing a treatment program that includes topical medications (e.g., creams and ointments).
[0059] In some embodiments, method 300 may further include adjusting operating parameters of the personal care device (e.g., personal treatment device) based on the received sensor data at step 322. The adjustments performed at step 322 may be performed separately from the other processing of step 212 or as part of step 212. In this manner, the personal care device may adjust directly based on the received sensor data without comparing the received sensor data to previously stored data or thresholds.
[0060] FIG. 4 illustrates an example of the system 100 of FIG. 1 in use. In the illustrated example, the subject 102 views a mirrored image 102′ of themselves in an interactive mirror 400. The interactive mirror 400 includes a camera (i.e., image capture device) 104 and a processor (not shown), such as processor 118. In this example, the subject 102 holds a handheld device 402 that functions as both the ROI selector unit 112 and the sensor 114. The subject 102 can point at an ROI with the handheld device 402 to indicate the location of the ROI on a portion of their skin. In another embodiment, the subject 102 can operate a laser pointer integrated into the handheld device 402 to project a laser onto the ROI (e.g., at the center of the ROI). The handheld device 402 includes a button 404 that the subject 102 can press to provide an indication of the nature of the ROI. In this example, the handheld device 402 also includes a series of lights 406, each of which can represent a different type of ROI. By pressing button 404, subject 102 can cycle through the various types of ROI until the desired light corresponding to the type of ROI the subject is identifying is illuminated. For example, handheld device 402 may include a first light corresponding to acne, a second light corresponding to pigmented spots, and a third light corresponding to areas of inflamed skin. Once subject 102 indicates the location of an ROI and the nature of the ROI using button 404, these indications may be provided to a processor (not shown) so that association information may be generated.
[0061] 4, a representation 408 of the subject 102 is displayed on the interactive mirror 400, along with indications of the various ROIs (e.g., labeled "ACNE1," "ACNE2," and "ACNE3") that have been previously recorded by the subject. In this way, the subject 102 can see what ROIs he or she has identified.
[0062] A third aspect of the present invention relates to an apparatus. FIG. 5 is a schematic diagram of an example apparatus 500. The apparatus 500 may be referred to as an apparatus for assessing a region of interest in a subject 102. The apparatus 500 comprises a processor 502 configured to perform one or more steps of methods 200, 300 described herein. Thus, the processor 502 may be configured to receive an image including a portion of the subject; receive an indication of a location of a region of interest (ROI) on the portion of the subject; receive sensor data from a sensor indicative of a measurable parameter of the ROI; receive an indication of a property of the ROI from the subject via a user interface; generate an association between the received sensor data and the property of the ROI; and provide at least one of the image, the sensor data, the indication of the property of the ROI, and the generated association for further processing. In some embodiments, the processor 502 may include or be similar to the processor 118 described above. The apparatus 500 may comprise or form part of a computing device such as a desktop computer, a laptop computer, a tablet computer, a smartphone, a wearable device such as a smartwatch, an interactive mirror, or a server. In some embodiments, the device 500 may comprise one or more of the components 104, 112, 114, 116 of the system 100 described above.
[0063] A fourth aspect of the present invention relates to a computer program product. Figure 6 is a schematic diagram of an example of a processor 602 in communication with a computer readable medium 604. According to an embodiment, the computer program product comprises a non-transitory computer readable medium 604 having computer readable code embodied therein that, when executed by a suitable computer or processor 602, causes the computer or processor to perform the steps of the methods 200, 300 described herein. In this manner, the computer-readable medium 604 may have computer-readable code embodied therein that, when executed by a suitable computer or processor, is configured to cause the computer or processor to receive an image including a portion of a subject; receive an indication of the location of a region of interest (ROI) on the portion of the subject; receive sensor data from a sensor indicative of a measurable parameter of the ROI; receive an indication of a property of the ROI from the subject via a user interface; generate an association between the received sensor data and the property of the ROI; and provide at least one of the image, the sensor data, the indication of the property of the ROI, and the generated association for further processing.
[0064] Thus, embodiments of the present invention provide a mechanism by which a subject can provide annotations themselves to indicate the nature of an ROI. This takes advantage of the understanding that the subject knows what certain features of their skin are, allowing them to be labeled or tagged as such while an image of the skin is being captured for future analysis. At the same time, measurable parameters of the subject's body (e.g., skin) can be measured using sensors, and these measurements can be associated with the ROI, allowing the recorded data to be used for various subsequent processing techniques. Thus, the present invention helps reduce the amount of subsequent processing required for captured images by reducing the need for medical professionals to analyze the images and provide their own indication of the nature of the ROI. This not only reduces the amount of processing required, but also shortens the time between image capture and ROI labeling. This can be particularly advantageous when sensor data acquired about an ROI indicates that the ROI requires urgent medical attention.
[0065] The processor 118, 502, 602 may comprise one or more processors, processing units, multi-core processors, or modules configured or programmed to control the device 500 in the manner described herein. In particular configurations, the processor 118, 502, 602 may comprise multiple software and / or hardware modules each configured to or for performing individual or multiple steps of the methods described herein.
[0066] As used herein, the term "module" is intended to include a hardware component, such as a processor or processor component, configured to perform a particular function, or a software component, such as a set of instruction data that has a particular function when executed by a processor.
[0067] It will be understood that embodiments of the present invention also apply to computer programs, particularly those on or in a carrier, adapted to carry out the present invention. Such programs may be in the form of source code, object code, source and object intermediate code, such as partially compiled form, or any other form suitable for use in implementing methods according to embodiments of the present invention. It will also be understood that such programs may have many different architectural designs. For example, program code implementing the functionality of a method or system according to the present invention may be subdivided into one or more subroutines. Many different ways of distributing functionality among these subroutines will be apparent to those skilled in the art. The subroutines may be stored together in an executable file to form a self-contained program. Such an executable file may contain computer-executable instructions, such as processor instructions and / or interpreter instructions (e.g., Java interpreter instructions). Alternatively, one or more or all of the subroutines may be stored in at least one external library file and linked to the main program statically or dynamically (e.g., at run time). The main program contains at least one call to at least one of the subroutines. The subroutines may also contain function calls to each other. An embodiment of a computer program product includes computer-executable instructions corresponding to each processing step of at least one of the methods described herein. These instructions may be subdivided into subroutines and / or stored in one or more files, which may be statically or dynamically linked. Another embodiment of a computer program product includes computer-executable instructions corresponding to each means of at least one of the systems and / or products described herein. These instructions may be subdivided into subroutines and / or stored in one or more files, which may be statically or dynamically linked.
[0068] The carrier of a computer program may be any entity or device capable of carrying the program. For example, the carrier may include a data storage device such as a ROM (for example a CD-ROM or a semiconductor ROM) or a magnetic recording medium (for example a hard disk). Furthermore, the carrier may be a transmissible carrier such as an electric or optical signal, which may be conveyed via an electric or optical cable or by radio or by other means. When the program is embodied in such a signal, the carrier may be constituted by such a cable or other device or means. As another example, the carrier may be an integrated circuit in which the program is embedded, the integrated circuit being adapted for performing, or used for the performance of, the relevant method.
[0069] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the principles and techniques described herein, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the singular does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. A computer program can be stored or distributed on a suitable medium, such as an optical storage medium or a solid-state medium provided together with or as part of other hardware, as well as in other forms of distribution, such as via the Internet or other wired or wireless communication systems. Any reference signs in the claims should not be construed as limiting the scope.
Claims
1. 1. A computer-implemented method for assessing a region of interest in a subject, comprising: receiving an image including a portion of a subject; receiving an indication of a location of a region of interest (ROI) on a portion of the subject; receiving sensor data from a sensor indicative of a measurable skin parameter of the ROI; receiving an indication of a nature of the ROI from the subject via a user interface; generating association information between the received sensor data and a property of the ROI; providing at least one of the image, the sensor data, an indication of a nature of the ROI, and the generated association information for further processing; comparing the received sensor data to at least one of a threshold and previously stored sensor data; generating at least one of an alert to be provided to the subject, feedback to be provided to a medical professional, and a control signal to be provided to a personal care device, in response to determining that the received sensor data meets or exceeds the threshold and / or based on the comparison with the previously stored sensor data, respectively; 13. A computer-implemented method comprising:
2. the received image includes a first facial landmark feature structure and a second facial landmark feature structure; and mapping the received image onto a three-dimensional wire mesh model based on a position of the first facial landmark feature structure relative to the second facial landmark feature structure.
2. The computer-implemented method of claim 1.
3. providing the sensor data and an indication of a property of the ROI as training data inputs for a predictive model, the predictive model determining a property of a given ROI based on the sensor data indicative of a measurable skin parameter of the given ROI; training the predictive model using the provided sensor data and an indication of a nature of the provided ROI; 3. The computer-implemented method of claim 1 or claim 2, further comprising:
4. 4. The computer-implemented method of claim 1, further comprising the step of providing an indication of the nature of the ROI and a location of the ROI for presentation on a representation of the subject.
5. The computer-implemented method of claim 1 , further comprising adjusting an operating parameter of a personal care device based on the received sensor data.
6. 6. The computer-implemented method of claim 1 , further comprising storing the received image, an indication of a location of the received ROI, the received sensor data, and an indication of a nature of the received ROI in a storage device. generating a treatment profile for the subject based on a comparison of the received sensor data to the previously stored sensor data. The computer-implemented method of claim 1 , further comprising:
8. The computer-implemented method of claim 1 , further comprising the step of operating an image capture device to capture the image.
9. 9. A computer program having a non-transitory computer readable medium having computer readable code embodied therein which, when executed by a suitable computer or processor, causes the computer or processor to perform the computer implemented method of any one of claims 1 to 8.
10. An apparatus for assessing a region of interest in a subject, comprising a processor configured to perform the steps of the computer-implemented method of any one of claims 1 to 8.
11. 1. A system for assessing a region of interest in a subject, comprising: an image capture device for capturing an image including a portion of the subject; a region of interest (ROI) selector unit enabling the subject to provide an indication of a location of a region of interest (ROI) on a portion of the subject; a sensor for acquiring sensor data indicative of a measurable skin parameter of the ROI; a user interface for receiving an indication of a nature of the ROI from the subject; and Processor and wherein the processor: generating association information between the acquired sensor data and an indication of a property of the ROI; providing at least one of the image, the sensor data, the indication of a nature of the ROI, and the generated association information for further processing; comparing the received sensor data to at least one of a threshold and previously stored sensor data; and generating at least one of an alert to be provided to the subject, feedback to be provided to a medical professional, and a control signal to be provided to a personal care device, in response to determining that the received sensor data meets or exceeds the threshold and / or based on the comparison with the previously stored sensor data, respectively. system.
12. 12. The system of claim 11, wherein the ROI selector unit comprises at least one unit selected from the group including a hand-held pointing device, an optical pointing device, a finger-worn pointing device, a gesture interpretation unit for interpreting gestures made by the subject, and a gaze interpretation unit for interpreting the subject's gaze directed at a displayed representation of the ROI.
13. 13. The system of claim 11 or claim 12, wherein the sensor comprises a sensor selected from the group comprising an imaging device, a sensor for measuring the reflectance spectrum of the ROI, a moisture sensor, and a sebum measuring device.
