Method and system for body part measurement for skin treatment - Patents.com

Abstract

A method implemented by one or more computers for outputting dimensional measurements of a body part in an input image is provided, the method comprising: receiving an input image captured using an image capture unit, the input image including the body part and a reference object of predefined dimensions; predicting a segmentation mask for the input image, the segmentation mask distinguishing image pixels of the body part from image pixels of a background region in the input image; processing the reference object in the input image to predict pixel dimensions of the reference object; obtaining pixel coordinates of a plurality of key points of the body part using the segmentation mask; modifying the obtained pixel coordinates based on the predicted pixel dimensions of the reference object; and outputting dimensional measurements of the body part based on the modified pixel coordinates. A method and system for training a plurality of neural networks used to control an electronic device to output dimensional measurements of body parts in input images captured by an image capture unit of the electronic device is provided.

Description

[Technical field]

[0001] The present invention relates to a method for use in a skin treatment device, in particular a method, computer program and system executed by one or more computers using a neural network to output dimensional measurements of body parts in an input image. The present invention further relates to a method for training a neural network used to control an electronic device to output dimensional measurements of body parts in an input image, a computer program for training the neural network and the method and system. [Background technology]

[0002] Skin treatment devices typically utilize software applications (apps) that can be downloaded to electronic devices such as smartphones or tablets. These electronic devices are coupled to the skin treatment device by a connection means such as Wireless Fidelity (WiFi) or Bluetooth. The app can be used to control the operation of the skin treatment device, display messages to the user, etc. Summary of the Invention [Problem to be solved by the invention]

[0003] For operational control related to skin treatment, the app can prompt the user to input dimensions such as the length of the body part via a suitable user interface (e.g., touch screen) of the electronic device. Traditionally, the user manually measures the length of the body part to be treated by the skin treatment device, for example using a standard measuring tape. The input length is used by the app to calculate treatment parameters that are customized for the user. For example, in skin treatments such as Intense Pulsed Light (IPL) devices, the treatment parameters include the number of light flashes required to treat the body part, the intensity of the light, etc.

[0004] It is an object of the present invention to provide an improved solution which allows automating the measurement of body part dimensions and which is more convenient for the user.

[0005] Users are also prone to making errors in measurements, resulting in inaccurate calculation of the treatment parameters. Another object of the invention is to provide a solution that allows improving the accuracy of the body part being measured.

[0006] In the field of computer vision, human body keypoint detection refers to the detection of keypoints or fiducials of a human body in an image. However, this method requires complex image processing routines, which makes it slow for real-time applications. [Means for solving the problem]

[0007] It is yet another object of the present invention to provide an automated solution to the challenges associated with image processing methods such as keypoint detection. This is achieved by using a hybrid method based on neural networks and keypoint detection.

[0008] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks are deep neural networks that have one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of its respective set of parameters.

[0009] It is yet another object of the present invention to provide a technique for training a neural network that can be used to output measurements of body part dimensions in an input image on an electronic device.

[0010] Details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. For example, the present specification provides methods, systems, and apparatuses, including a computer program encoded on a computer storage medium, for performing any of the techniques disclosed herein. In some embodiments, the computer program includes instructions that, when executed by a computing device, cause the computer program to perform any of the techniques disclosed herein.

[0011] In one aspect, a method and system implemented by one or more computers for outputting measurements of dimensions of a body part in an input image is provided, the method comprising: receiving an input image captured using an image capture unit, the input image including a body part and a reference object of predefined dimensions; predicting a segmentation mask of the input image by a segmentation deep neural network (S-DNN), the segmentation mask distinguishing image pixels of the body part from image pixels of a background region in the input image; detecting the reference object in the input image by an object detection deep neural network (OD-DNN) and processing the detected reference object to predict pixel dimensions of the reference object; obtaining pixel coordinates of a plurality of key points of the body part using the segmentation mask; modifying the obtained pixel coordinates based on the predicted pixel dimensions of the reference object; measuring dimensions of the body part based on the modified pixel coordinates; and outputting the measured dimensions of the body part.

[0012] The method and system provide an improved measurement solution that can automate the measurement of the dimensions of a user's body parts, making devices incorporating the solution more user-friendly. As a result of the features used therein, the solution also improves the accuracy of the measurements of the body parts. The method further reduces the complexity of image processing routines, making them more suitable for real-time applications and therefore can be implemented on devices with relatively little processing power.

[0013] In one aspect, a method and system for training a plurality of neural networks used to control an electronic device to output measurements of dimensions of body parts in an input image is provided. The input image is captured by an image capture unit of the electronic device. The plurality of neural networks include an object detection deep neural network, an OD-DNN, and a segmentation deep neural network, an S-DNN. The method includes the steps of initializing neural network parameter values ​​of the S-DNN and the OD-DNN, acquiring a training dataset having a plurality of images, each image including a body part and a reference object of predefined dimensions, annotating the reference objects and the body parts of the training dataset to generate an annotated dataset, separating a portion of the annotated training dataset into a validation dataset, and inputting the annotated training dataset into the OD-DNN, where the OD-DNN detects the reference objects in the input image, processes the detected reference objects to generate a validation dataset. The method includes the steps of: inputting the annotated training dataset into an S-DNN, where the S-DNN learns to predict a segmentation mask of the input image, the segmentation mask distinguishing image pixels of the body parts from image pixels of background regions in the input image; validating the pixel dimensions of the reference objects learned by the OD-DNN and the segmentation mask learned by the S-DNN using a validation dataset; and updating neural network parameter values ​​of the S-DNN and the OD-DNN based on the validation.

[0014] The methods and systems provide training solutions for real-world applications that involve controlling an electronic device to output measurements of a body part. The outputs can be displayed to a user of the electronic device. In some implementations, the training solutions continuously interact with measurement inputs captured by one or more physical image capture units.

[0015] In one aspect, one or more computer program products are provided having instructions that, when executed by one or more computers, cause the one or more computers to perform the steps of the methods described above.

[0016] As noted above, other features, aspects, and advantages of the subject matter will become apparent from the specification, the drawings, and the claims. [Brief description of the drawings]

[0017] [Figure 1] 1 illustrates a method 100 for outputting measurements of dimensions of a body part in an input image, according to one embodiment. [Diagram 2] FIG. 1 illustrates a segmentation mask according to one embodiment. [Diagram 3] 1 illustrates a reference object according to one embodiment. [Figure 4] 1 illustrates aspects of image pixel measurement, according to one embodiment. [Figure 5A] 1 illustrates aspects of the keypoint method, according to one embodiment. [Figure 5B] FIG. 1 illustrates another aspect of the keypoint method, according to one embodiment. [Figure 6] FIG. 6 illustrates a system 600 for outputting measurements of body part dimensions, according to one embodiment. [Figure 7] FIG. 7 illustrates a method 700 for training a neural network, according to one embodiment. [Figure 8] FIG. 8 illustrates a training system 800 according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] FIG. 1 is a flow chart of an exemplary method 100 executed by one or more computers in a system for outputting measurements of a dimension of a body part in an input image. An example dimension is length, but the invention is not limited to measuring length. Other dimensions include, but are not limited to, width or height of a body part. For convenience, the method 100 is described as being executed by a system of one or more computers located at one or more locations. For example, an image processing system having a distributed architecture, suitably programmed according to the present specification, can execute the method 100. The method 100 can also be executed by a single computer, for example, by a processor of a user electronic device / computer. The order of steps of the method 100 is merely exemplary, and for example, one skilled in the art will recognize that it is possible to reorder at least some of the steps of the method. The term "user" refers to any user who uses the methods and / or systems and / or computer programs based on the methods described herein, and need not be the same person as the subject to whom the treatment is performed. The subject can be a human or an animal.

[0019] In step 102, the method 100 includes receiving an input image. The input image includes a body part intended to be treated, e.g., a leg, an arm, or any other body part. It can be a two-dimensional (2D) image (e.g., represented as a pixel array) and can be acquired by an image scanner of any modality, e.g., an image capture unit of a user electronic device (a smartphone camera).

[0020] The input image further includes a reference object of a given dimension. It is difficult to obtain the exact length of a region of interest (ROI) or a desired object in a captured image. This is because the measurement of the ROI in an image may deviate from its actual value based on factors such as the selected optical / digital zoom of the image capture unit, the resolution, the angle of image capture, etc. Since all objects in an image share similar characteristics, such deviations can be compensated for by linking the ROI to another object in the same image, i.e., another object with a known length.

[0021] Skin treatment devices often have at least one attachment with a standard length. In a particular example, the treatment opening of this attachment can be used as a reference object. Such a reference object is shown as 2. In an IPL device (e.g., Philips Lumea), this can be the treatment opening of the device attachment supplied with the IPL device. It has a known dimension of about 30 mm, specifically 32.7 mm. It is understood that other dimensions are possible for the treatment opening as well. In one example, the user is prompted by the app to capture an input image of the body part while keeping the device attachment within the field of view of the camera.

[0022] In step 104, the method 100 includes predicting at least one segmentation mask of the input image. To process the input image, a segmentation deep neural network (S-DNN) is used. In one example, a ResNet model is used. The S-DNN is configured to receive the input image and predict the segmentation mask by processing the input image according to the current (optimized) values ​​of the S-DNN parameters. Step 104 can be performed at a single location (e.g., in the user electronic device) by a single computer, or at different locations by another computer or processor. In the latter example, the S-DNN can be located in a distributed server, such as a cloud. Thus, the calculations are performed at the server location. The user electronic device can be configured to transmit the acquired input image to the server by wireless communication. In this example, the user electronic device and the server form multiple computers in a wireless communication system. Since the image processing is performed away from the user (client) electronic device, fewer resources are required by the device at the user location.

[0023] S-DNN is a semantic classifier that classifies pixels in an input image into multiple classes or types. In one example, the predicted segmentation mask includes classes ROI and background. Image pixels of the ROI in the segmentation mask are associated with a first class, and image pixels of the background region in the segmentation mask are associated with a second class. The second class associated with the background includes image pixels outside the boundaries of the ROI or body part, and can optionally be filtered out from the segmentation mask (shown dark in FIG. 3). In another example, multiple segmentation masks are generated. A first segmentation mask is associated with a first class, and a second segmentation mask is associated with a second class. In other words, the segmentation mask distinguishes image pixels of the ROI, body part from image pixels of the background region in the user image. An example of a segmentation mask indicating the first class / ROI class is shown in FIG. 3. The remaining steps of processing the image are performed on this predicted segmentation mask, which is a subset of the input image, making the processing faster than if the raw input image data was processed.

[0024] In some implementations, the segmentation mask has a score that represents the likelihood that the predicted mask is the most appropriate classification for the user. This score, among other parameters, can be used by the S-DNN for future prediction, i.e., to compute segmentation masks for other input images of the same or similar body parts at a later time.

[0025] As mentioned above, the input image includes a reference object of a given dimension / length / size in order to obtain an accurate measurement of the ROI in the input image. In step 106, the method 100 detects the reference object in the input image and processes the reference object in the input image to predict an image pixel dimension, e.g., the (total) pixel length of the reference object, in other words, the length (number of pixels) of a pixel row spanning the length of the reference object in the input image. The prediction acts as a correction factor and is applied to the pixel coordinates of the ROI (body part) determined via keypoint detection. This is described in more detail below.

[0026] An object detection deep neural network (OD-DNN) is used to detect and process the reference object in the input image. In one example, a You only look once (YOLO) object detection convolutional neural network (CNN) model is used. The OD-DNN is configured to receive the input image and predict the pixel length of the reference object by processing the input image according to the current (optimized) values ​​of the OD-DNN parameters. The OD-DNN can be further configured to detect the ROI by processing the input image according to the current (optimized) values ​​of the OD-DNN parameters. As in step 104, step 106 can be performed in a single location (e.g., in the user electronic device) by a single computer or in different locations by distributed computers. In the latter example, the OD-DNN can be located in a distributed server such as a cloud. Thus, the processing or calculation is performed at the server location. As in step 104, the user electronic device can be configured to transmit the captured input image to the server by wireless communication, and the electronic device and the server form a wireless communication system. Because image processing is performed away from the user (client) electronic device, fewer resources are required by the user-located device.

[0027] In some implementations, the OD-DNN is trained to output pixel coordinates of a bounding box that outlines a reference object. A bounding box indicates the spatial location of an object and is represented by a set of coordinates that bounds the object by a minimal enclosing box. An example of a bounding box is shown in FIG. 2. During training of the OD-DNN, a bounding box technique can be used to annotate or label the reference object in each image of the training image dataset, so that the OD-DNN learns to obtain pixel lengths from the predicted pixel coordinates of the bounding box in the input image with some accuracy.

[0028] Similar to step 104, the predicted pixel length of the reference object has a score, which represents the likelihood that the prediction is the best (linear) regression. This score can be used by the OD-DNN for future predictions, among other parameters.

[0029] In some implementations, the OD-DNN calculates a single pixel length, expressed as a measurement per pixel (MPP), based on the predefined dimensions of the reference object and the predicted pixel dimensions. In the field of computer vision, MPP is understood as quantifying the number of inches or centimeters of the environment shown in the image that are encoded or represented per pixel length or pixel width of the image. An example is shown in FIG. 4. The task performed by the OD-DNN is then to determine how much real-world spatial length is encoded per pixel length or pixel width of the image. The MPP is calculated as follows: MPP = object ground truth length / object pixel length

[0030] In the above formula, the predetermined dimension of the reference object is expressed by the ground truth length, and the predicted pixel dimension is expressed as the pixel length of the object. This metric acts as a correction factor and is applied to the pixel coordinates of the ROI (body part) determined through keypoint detection. Some other advantages of using MPP to measure a (reference) object in an image are that the metric does not depend on the distance between the image capture unit (camera) and the object, can be used for any object of known dimensions in the image, and does not depend on the pixel density of the image and camera calibration parameters such as skew factor and optical center.

[0031] In step 108, the method 100 obtains pixel coordinates of a number of key points of the body part from the segmentation mask obtained in step 104. This is shown in Figures 5A and 5B. The key point detection algorithm used in the present invention is specifically selected to improve resource usage of the method and / or the user electronic device, in other words its internal functions. When incorporated into a non-AI based model, it has the advantage that it does not require pre-training to perform the task. In contrast, AI models typically require a large amount of training data to learn the underlying patterns or features in the data. It is therefore suitable to be implemented on the user electronic device. As mentioned above, the algorithm processes a subset of the image data, making the implementation on the user device faster. In some embodiments, step 108 is performed by a remote / distributed computer in the system.

[0032] In one example, key points of the body part "leg" are selected as the ankle and knee joints. An example of how these key points are detected is as follows: In FIG. 5A, a segmentation mask shows the image details of the leg in a particular orientation. The mask has a height of h pixels and a width of w pixels. A set of coordinate axes x (horizontal) and y (vertical) is defined with respect to the segmentation mask, and an arbitrary origin is defined as (0,0). As shown in FIG. 5(a) and FIG. 5(b), the origin is located at the top left. The leg is preferably aligned at a non-zero angle α with respect to the horizontal axis of the mask. The recommended leg orientation allows to determine the points of the desired key points using basic calculations in a simple manner. The ankle and knee joints are identified as follows: Given the leg orientation, the leg contour pixel on the mask associated with the y-minimum is assigned as the knee joint K. This is as shown in the right half of FIG. 5A. In other words, the pixel with the minimum y coordinate is treated as the knee joint coordinate. To determine the ankle joint A, the method determines a set of four points T, H, Q, and J on the lower leg, and the ankle joint coordinate A is determined by the intersection between line segments TJ and HQ. A second minimal y coordinate point in the mask is determined as the toe pixel T. A maximal y coordinate point is determined as the heel pixel H. A maximal point in the mask near the upper intersection of the foot and leg is selected as point Q, and point J is empirically defined as x(H)+100 units, where x(H) is the x coordinate of the heel H.

[0033] An exemplary implementation for calculating the minimum / maximum is shown in FIG. 5B. A vertical slider with a pixel width of 1 can be scanned in a given direction across the segmentation mask. In FIG. 5B, to identify the knee joint K, the vertical slider scans the mask in a given direction D, for example, from left to right. A set of coordinate axes is defined as described above. The vertical slider identifies all ROI pixels on the mask contour (boundary) along the scanning direction and checks the height of the pixels along said contour. The pixel with the lowest height for the assigned y=0 is selected as the knee joint pixel y coordinate. The corresponding x coordinate is also identified. The implementation can be repeated for the detection of each key point. It is clear to a person skilled in the art that any identification means, such as a cursor, can be used instead of the vertical slider.

[0034] In some embodiments, the method calculates the distance between the obtained keypoints.

[0035] In step 110, the method 100 modifies the obtained pixel coordinates of the keypoints or the distance between the keypoints based on the predicted pixel dimensions of the reference object. In some implementations, the method corrects the obtained coordinates using the pixel size derived from the measurement per pixel calculation according to the formula in paragraph

[38] , so that the actual leg length = MPP * distance between the keypoints (pixel length of the ROI).

[0036] Such corrections are based on reference to objects in the input image with known dimensions and take into account variations in ROI size due to factors such as the selected optical / digital zoom of the image capture unit, image resolution, and angle of image capture.

[0037] In step 112, the method 100 outputs a measurement of the length of the body part based on the corrected pixel coordinates. The measurement can be displayed to the user by the user electronic device on its display screen. The measurement can be further transmitted by the user electronic device to the skin treatment device for calculation of treatment parameters. In some embodiments, the measurement can be transmitted directly to the skin treatment device, i.e., without further display to the user.

[0038] The method can use at least one of the input image, the segmentation mask, the predicted pixel dimensions, and the output measurements of the body part dimensions for future prediction of the segmentation mask, the pixel dimensions, and the output measurements of the body part dimensions.

[0039] 6 illustrates an exemplary image processing system 600 in accordance with an exemplary embodiment of the present invention, which is one example of a system in which the systems, components, and techniques described herein may be implemented as computer programs on one or more computers at one or more locations.

[0040] The image processing system 600 is configured to receive and process an input image 601 depicting or including the body part to be processed. The body part is considered as a ROI 602 in the input image. The input image 601 is acquired using an image capture unit 603 which may form part of the system 600 or may be external to the system 600. As mentioned above, the image capture unit 603 may be part of a user electronic device and the body part may be part of a user or another subject.

[0041] The system 600 processes an input image 601 to predict a segmentation mask of the input image. In doing so, it provides the input image as an input to an S-DNN 604 (as described above). The S-DNN is configured to receive the input image 601 and process it according to current values ​​of the S-DNN parameters to generate or predict a segmentation mask 605 (based on a hypothesis that approximates a function between the input to the DNN and the expected output). The segmentation mask characterizes the segmentation of the input image into at least one class, which class includes image pixels of the ROI that are body parts.

[0042] System 600 may store the generated segmentation mask in a data store (e.g., a logical data storage area or a physical data storage device, not shown) wirelessly coupled to system 600. A user of system 600 (e.g., a user operating a device at home) may subsequently retrieve the segmentation mask from the data store. In some cases, system 600 may present the generated segmentation mask directly on a display device (e.g., a computer screen) that is visible to a user of system 600, for example using a display of a user electronic device.

[0043] The S-DNN is trained with different initialized parameter values, different training images, or both. An exemplary process for training the S-DNN is described with reference to FIG. 7. In some implementations, the S-DNN is pre-trained to predict the segmentation mask of any image. The pre-trained model is further trained using customized training data images. The advantage of using a pre-trained model is that the computation converges faster, i.e., optimized neural network parameters are obtained faster. In one example, the S-DNN is a semantic classifier. Some model architectures that can perform semantic segmentation include Fully Convolutional Networks (FCN), U-Net, Mask RCNN, etc. In one example, the S-DNN can be implemented using a DeepLabv3 PyTorch model with an FCN-ResNet101 backbone.

[0044] To generate the output measurements 610 of the body part, the system 600 further provides the segmentation mask 605 to a keypoint detector 606. The algorithm or model of the keypoint detector 606 can be embedded in a user device, which can be included inside the system 600 or can be external, at a server location along with S-DNN and / or other neural networks (the latter shown in FIG. 6). The implementation of the keypoint algorithm is as described above. The keypoint detection algorithm does not rely on a resource-intensive deep neural network architecture to identify image pixels corresponding to keypoints within the ROI, and therefore can significantly improve the latency of prediction. The output measurements of the body part include the dimensions of the body part that are determined by locating the keypoints among the ROI / body part image pixels within the segmentation mask. Optionally, the system 600 outputs the measurements to the user device.

[0045] In some implementations, the system 600 further processes the input image 601 to correct the acquired dimensions. In this case, the input image 601 further includes a reference object 607. The system 600 provides the input image 601 as an input to the OD-DNN 608 (OD-DNN details as described above). The OD-DNN processes the reference object 607 to predict pixel dimensions of the reference object 609. In some implementations, the system 600 further calculates a pixel size based on the predicted pixel dimensions of the reference object and the known dimensions of the reference object 607. The system 600 corrects the acquired dimensions of the body part 610 based on the predicted pixel dimensions of the reference object or the derived pixel size / metric MPP. The keypoint detector 606 can apply this correction to the pixel coordinates of the obtained keypoints or to the dimensions of the body part calculated based on these pixel coordinates in its output 610.

[0046] The OD-DNN is trained with different initialized parameter values, different training images, or both. An exemplary process for training the OD-DNN is described with reference to FIG. 7. In some implementations, the OD-DNN is pre-trained to predict pixel dimensions of any image. The pre-trained OD-DNN is further trained using customized training data images. In one example, the OD-DNN can be implemented using the Yolo AI model. Depending on the degree of accuracy desired in predicting the dimensions of the body parts, a heavyweight or lightweight (a term well known in the art) YOLO model can be used. Other object detection neural network architectures include Fast R-CNN, Single Shot Detector (SSD), and the like.

[0047] Both S-DNN and OD-DNN can be trained using separate supervised learning procedures. Neural networks can also be trained by the same supervised training procedure using the same training dataset. The procedure can include using manually labeled training images. An example of the latter approach is shown in Figure 7.

[0048] FIG. 7 is a flow chart of an exemplary method 700 for training S-DNNs and OD-DNNs according to an embodiment of the present invention. Method 700 is described as being performed by one or more computer systems located at one or more locations. Although it is possible to implement method 700 using system 600, it is advantageous to train neural networks using a separate procedure before implementing them in the method of FIG. 1. The trained OD-DNNs and S-DNNs can be used in method 100 to predict segmentation masks and pixel dimensions of body parts in an input image. As described above, the input image can be captured by an image capture unit of an electronic device. The electronic device can be one of one or more computers in the system. Thus, the training procedure of FIG. 7 can be used to control the electronic device to obtain and / or output measurements of dimensions of body parts in an input image.

[0049] In step 702, the method includes initializing neural network parameter values ​​of the S-DNN and the OD-DNN. In some implementations, the method can initialize the S-DNN and the OD-DNN neural network parameter values ​​based on trained parameter values ​​of another neural network having the same architecture as the S-DNN and the OD-DNN, respectively, which are trained to perform similar predictions. For example, as described above, the S-DNN and the OD-DNN can be built on a pre-trained model, i.e., the OD-DNN and the S-DNN are pre-trained to detect objects in any image and semantically segment any image into at least one segment mask of a particular image class, respectively. In this case, the initialization can be based on the trained parameter values ​​of the pre-trained S-DNN and the OD-DNN.

[0050] In some implementations, the scores of the predictions made by system 600 in method 100 are taken into account during initialization of parameter values ​​in step 702, so that the training of the neural network can be adjusted each time method 100 is executed. As a result, over time, system 600 learns to obtain and / or output dimensional measurements of body parts in input images to a user of the electronic device with improved accuracy.

[0051] In step 704, the method 700 obtains training examples, i.e., a training dataset that includes a number of images, each of which includes a body part and a reference object of a given dimension. Each image in the training dataset is further annotated by a human expert in an automated or semi-automated manner. Annotation refers to the process of assigning types to objects in images, either manually by a person or using a computer, for the purpose of training a neural network.

[0052] In some implementations, the method 700 periodically obtains the training data set from one or more computers in the system. For example, each time the method 100 is executed by the system 600, the input image can be stored in the system for later training purposes. In this manner, the S-DNN and OD-DNN can be trained iteratively.

[0053] In step 706, the method includes annotating reference objects and body parts in the acquired training dataset to generate an annotated training dataset.

[0054] In one example, for each image in the training dataset, a particular class or image pixels of a particular class in the training dataset are annotated to generate an expected segmentation mask output. The annotated training dataset is input to the S-DNN. Each example in the training dataset input to the S-DNN has (i) a training image (input) and (ii) an annotated training segmentation mask indicating image pixels of the body part class in the training image (expected output). The S-DNN learns to map each input to each output using a possible hypothesis function.

[0055] In the training dataset for the OD-DNN, the method includes annotating a reference object in each image using annotation techniques, such as by using a bounding box. As shown in FIG. 2, the processing aperture of the device attachment can be used as a reference object, and thus the aperture can be annotated with a bounding box ABCD in each image of the training dataset. Each example in the training dataset for the OD-DNN has (i) a training image (input) and (ii) an annotated reference object in the training image (expected output).

[0056] It is beneficial to modify the training dataset to increase or augment the size of the training set and thus the accuracy of the model prediction. In some implementations, the method includes using image augmentation techniques such as image transformation and image inversion to generate different versions of each image in the training dataset. Image augmentation also addresses the overfitting problem of S-DNN and OD-DNN. Image augmentation can obtain heterogeneous data that helps the model learn different patterns, thus avoiding overfitting.

[0057] In step 708, the method includes splitting or dividing a portion of the annotated training dataset into a validation dataset. A portion of the annotated training dataset is selected as the validation dataset. Thus, it is ensured that the training set and the validation set are obtained using the same distribution of data. This subset is used to evaluate the performance of the S-DNN and the OD-DNN in their predictions.

[0058] At step 710, the method includes inputting the annotated training dataset into the OD-DNN and the S-DNN, which learn to detect reference objects in the input images, process the detected reference objects to predict pixel dimensions of the reference objects, and learn to predict a segmentation mask for each image, which distinguishes image pixels of the ROI from a background in the image.

[0059] Step 712 involves using the validation dataset to validate the pixel dimensions of the segmentation masks and reference objects learned by the trained S-DNN and OD-DNN, respectively.

[0060] In one example, the trained YOLO OD-DNN achieves a map score of 0.70 on the validation dataset. For each image in the validation set, the YOLO OD-DNN detects the reference object and predicts the pixel coordinates A, B, C, D of the bounding box that bounds the reference object. The trained DeepLab S-DNN achieves an mIOU score of 0.75 on the validation dataset. For each image in the validation set, the DeepLab S-DNN predicts a segmentation mask indicating the image pixels of the ROI / body part.

[0061] Step 714 includes updating neural network parameter values ​​of the S-DNN and the OD-DNN based on the validation. The updated parameter values ​​can be further used to test the OD-DNN and the S-DNN using a test dataset having at least one image showing the body part and the reference object.

[0062] In some implementations, the pixel dimensions of the reference object learned by the OD-DNN are the size of a single pixel, preferably a pixel-by-pixel measurement, calculated based on predefined dimensions and predicted pixel dimensions.

[0063] Figure 8 shows an example of a system 800 for training a neural network 803 according to the method of Figure 7. The neural network training system 800 is an example of a system implemented, for example as a computer program, on one or more computers at one or more locations. The system 800 can be implemented on the system 600 (in other words they are one and the same system) or as a separate system 800 wirelessly coupled to the system 600. There may be additional systems 1000, 1002, ... coupled to the system 800. The systems 1000, 1002 may be other training systems or image processing systems that can be configured to execute the method 100 or database management systems whose data can be used by the system 800 to train the neural network 803.

[0064] The system 800 includes a training engine 801 for training the neural networks S-DNN and OD-DNN, which has access to a memory 802 shared by one or more computers in the system. In some implementations, the training engine 801 is a distributed processor.

[0065] The training engine 801 can be configured to initialize neural network parameters of the S-DNN and OD-DNN. The training engine 801 retrieves the parameters from the memory 801. The memory 801 can store trained parameter values ​​of other neural networks that have the same architecture as the S-DNN and OD-DNN, respectively, and are trained to perform similar predictions. If the S-DNN and OD-DNN are built on pre-trained models, the memory 801 can store trained parameter values ​​of the pre-trained S-DNN and OD-DNN, respectively.

[0066] The training engine 801 may further be configured to obtain a training data set stored in the memory 802, such training data set comprising an annotated training set. In some implementations, the training set stored in the memory 802 is periodically updated as described above. The continuously updated training set may be retrieved by the training engine 801. The training engine 801 may further obtain pixel dimensions and segmentation masks of reference objects of each image in the training set predicted by the OD-DNN and S-DNN, respectively, verify the predictions (check whether the training criteria are met, in some implementations iteratively), and update the neural network parameters of the S-DNN and OD-DNN accordingly to arrive at optimized neural network parameters of the S-DNN and OD-DNN. The trained neural networks S-DNN and OD-DNN may be used to control an electronic device to obtain and / or output measurements of dimensions of body parts in an input image. The optimized neural network parameters are used in the next initialization by the training engine 801.

[0067] The term "configured" is used herein in connection with systems and computer program components. A system having one or more computers configured to perform a particular operation or action means a system having installed thereon software, firmware, hardware, or a combination thereof that, during operation, causes the system to perform said operation or action. One or more computer programs configured to perform a particular operation or action means one or more programs that include instructions that, when executed by a data processing device, cause the device to perform an operation or action.

[0068] The embodiments and functional operations of the subject matter described herein can be implemented in digital electronic circuitry, intangible computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. The embodiments of the subject matter described herein can be realized as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or for controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random access memory device or a serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0069] The terms "computer", "processor", "data processing apparatus", "electronic apparatus", etc. refer to data processing hardware and encompass all types of apparatus or apparatuses, devices, and machines for processing data. An apparatus may also be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an execution environment for a computer program, e.g., code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0070] A computer program may be called or written as a program, software, software application, app, module, software module, script, or code, and may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily have to, correspond to a file in a file system. A program may be stored in part of a file that holds other programs or data, for example one or more scripts stored in a markup language document, a single file dedicated to the program in question, or in several cooperating files, for example a part of a file that stores one or more modules, subprograms, or parts of code. A computer program may be deployed to be executed on one computer, or on several computers located at one site or distributed across several sites and interconnected by a data communications network.

[0071] The term "engine" is used broadly herein to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed and operated on the same computer or multiple computers.

[0072] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as a field programmable gate array or ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0073] A computer suitable for executing instructions of a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. In general, the central processing unit receives instructions and data from a read-only memory or a random access memory, or both. The essential components of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by or incorporated in special-purpose logic circuitry. In general, the computer is also operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical, or optical disks. However, a computer need not have such devices. Furthermore, the computer can be embedded in another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0074] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0075] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer or electronic device having a display device, such as a CRT (cathode ray tube) or LCD display (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide user interaction, for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input from the user can be received in any form, including acoustic, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. The computer can also interact with the user by sending a text message or other form of message to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0076] A data processing device or computer for implementing machine learning models may also include dedicated hardware accelerator units for handling the common and computationally intensive parts of machine learning training or production, i.e. inference, workloads.

[0077] The machine learning model can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0078] An embodiment of the subject matter described herein can be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0079] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., HTML pages, to a user device, e.g., for the purpose of displaying the data to a user interacting with the device acting as a client and receiving user input from the user. Data generated at the user device, e.g., a result of a user interaction, can be received at the server from the device.

[0080] Although the present specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or claims, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

[0081] Similarly, although operations are illustrated in the figures and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order illustrated, or in sequential order, or that all of the illustrated operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be incorporated into a single software product or packaged into multiple software products.

[0082] Specific embodiments of the present subject matter have been described. Other embodiments are possible as long as they are within the scope of the appended claims. The enumerated embodiments can be further combined. The processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method implemented by one or more computers for outputting dimensional measurements of a body part in an input image, comprising: receiving an input image captured using an image capture unit, the input image having a body part and a reference object of predetermined dimensions; predicting a segmentation mask of the input image by a segmentation deep neural network (S-DNN), the segmentation mask distinguishing image pixels of the body part from image pixels of background regions in the input image; detecting a reference object in the input image by an object detection deep neural network (OD-DNN) and processing the detected reference object to predict pixel dimensions of the reference object; using the segmentation mask to obtain pixel coordinates of a number of key points of the body part; modifying the obtained pixel coordinates based on predicted pixel dimensions of the reference object; measuring dimensions of the body part based on the corrected pixel coordinates and outputting the measured dimensions of the body part; The method according to claim 1,

2. 2. The method of claim 1, wherein the S-DNN is a semantic S-DNN, wherein the image pixels of the body parts in the segmentation mask are associated with a first class and the image pixels of the background regions in the segmentation mask are associated with a second class.

3. 3. The method of claim 1 or claim 2, wherein the OD-DNN is configured to calculate a per-pixel measurement corresponding to a dimension of the environment shown in the input image represented by a single pixel based on predefined dimensions of the reference object and predicted pixel dimensions.

4. The method of claim 1 or 2, further comprising using at least one of the input image, the segmentation mask, the predicted pixel dimensions of the reference object, and the output measurements of the dimensions of the body part for future prediction of the segmentation mask and / or the pixel dimensions of the reference object.

5. obtaining pixel coordinates of a plurality of key points of the body part using the segmentation mask, defining a plurality of coordinate axes on the segmentation mask; identifying image pixels within a particular region of the body part on the segmentation mask that have maximum and / or minimum coordinates along at least one selected coordinate axis of a plurality of coordinate axes; obtaining pixel coordinates of a plurality of key points of the body part based on the identified pixels of the identified region; The method according to claim 1 or 2, comprising:

6. A computer program comprising instructions which, when executed by one or more computers, cause said one or more computers to carry out the steps of the method according to claim 1 or 2.

7. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the method of claim 1 or 2.

8. 1. A method for training a plurality of neural networks used to control an electronic device to output dimensional measurements of body parts in an input image, the input image being captured by an image capture unit of the electronic device, the plurality of neural networks comprising an object detection deep neural network (OD-DNN) and a segmentation deep neural network (S-DNN), the method comprising: initializing neural network parameter values ​​of the S-DNN and the OD-DNN; obtaining a training dataset comprising a plurality of images, each image comprising said body part and a reference object of predefined dimensions; annotating the reference objects and the body parts in images of the training dataset to generate an annotated training dataset; splitting a portion of the annotated training dataset into a validation dataset; inputting the annotated training dataset into the OD-DNN, wherein the OD-DNN detects the reference objects in the input image and processes the detected reference objects to predict pixel dimensions of the reference objects; inputting the annotated training dataset into the S-DNN, such that the S-DNN learns to predict a segmentation mask of the input image, the segmentation mask distinguishing image pixels of the body part from image pixels of background regions in the input image; using the validation data set to validate pixel dimensions of the reference objects learned by the OD-DNN and the segmentation mask learned by the S-DNN; updating neural network parameter values ​​of the S-DNN and the OD-DNN based on the verification; The method according to claim 1,

9. 9. The method of claim 8, wherein the OD-DNN and the S-DNN are pre-trained to detect objects in any image and to semantically segment the image to generate at least one segment mask for a particular image class.

10. annotating the reference object and the body part, annotating the reference object in each image of a training dataset using a bounding box; annotating image pixels of the body part in each image of the training dataset; 10. The method of claim 8 or claim 9, wherein the body part is associated with a first class.

11. The method of claim 8 or 9, wherein the OD-DNN is further trained to calculate a per-pixel measure corresponding to a dimension of the environment shown in the input image represented by a single pixel based on the predefined dimensions of the reference object and the predicted pixel dimensions.

12. The method of claim 8 or 9, further comprising periodically acquiring the training data set comprising the plurality of images for iteratively training the S-DNN and the OD-DNN.

13. 10. The method of claim 8 or 9, further comprising generating multiple versions of each image in the training dataset to augment the training dataset.

14. A computer program comprising instructions which, when executed by one or more computers, cause said one or more computers to carry out the steps of the method according to claim 8 or 9.

15. one or more computers; one or more storage devices storing instructions which, when executed by said one or more computers, cause said one or more computers to carry out the method of claim 8 or 9; A system having

Images