Information processing device, information processing method, and program

Abstract

The present disclosure pertains to an information processing device, an information processing method, and a program by which the size of a target can be more accurately measured. Provided is an information processing device comprising a processing unit that: subjects images including at least a depth image acquired by a sensor, and at least a portion of the information obtained from the images, to processing that uses a learned model learned via machine learning; and measures the size of a target included in the images. The present disclosure can be applied, for example, to a mobile terminal having a sensor.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that enable more accurate measurement of the size of an object. [Background technology]

[0002] As a method for measuring a user's foot size, a method using a dedicated foot measuring device and a method for calculating by analyzing a photographed image of the foot have been proposed. Patent Document 1 discloses a technology for calculating foot size based on the actual length ratio calculated using a foot image photographed with a mobile terminal and the number of pixels between vertical and horizontal sliders. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6295400 Summary of the Invention [Problem to be solved by the invention]

[0004] When measuring the size of an object such as a user's foot, it is necessary to measure the size accurately.

[0005] The present disclosure has been made in light of these circumstances, and makes it possible to measure the size of an object more accurately. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present disclosure includes: and RGB images including at least the depth image The image and at least a part of the information obtained from the image are processed using a trained model trained by machine learning, and User's body parta processing unit for measuring the size of the trained model 、 The image is input, User's body part A first trained model that outputs feature points of User's body part The features of User's body part and the size of User's body part The posture of the User's body part and a second trained model that outputs the corrected size obtained by correcting the size of the first trained model.

[0007] An information processing method and a program according to one embodiment of the present disclosure are an information processing method and a program corresponding to the information processing device according to one embodiment of the present disclosure.

[0008] In an information processing device, an information processing method, and a program according to an embodiment of the present disclosure, a depth image acquired by a sensor is and RGB images including at least the depth image An image and at least a portion of information obtained from the image are processed using a trained model trained by machine learning, and the image is User's body part The size of the trained model is measured. 、 The image is input, User's body part A first trained model that outputs feature points of User's body part The features of User's body part and the size of User's body part The posture of the User's body part and a second trained model that outputs the corrected size obtained by correcting the size of the first trained model.

[0009] Note that the information processing device according to one aspect of the present disclosure may be an independent device or an internal block constituting a single device. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram illustrating a configuration example of an information processing device to which the present disclosure is applied. [Figure 2] 2 is a block diagram showing a first example of the configuration of a processing unit in FIG. 1. FIG. [Figure 3] 10 is a flowchart illustrating a first example of the flow of a foot length measurement process. [Figure 4] FIG. 4 is a diagram schematically illustrating a data flow in the leg length measurement process of FIG. 3. [Figure 5] FIG. 2 is a block diagram showing a second example of the configuration of the processing unit in FIG. 1. [Figure 6] 10 is a flowchart illustrating a second example of the flow of the foot length measurement process. [Figure 7] FIG. 7 is a diagram schematically illustrating a data flow in the leg length measurement process of FIG. 6. [Figure 8] FIG. 2 is a block diagram showing a third example of the configuration of the processing unit in FIG. 1. [Figure 9] 10 is a flowchart illustrating a third example of the flow of the foot length measurement process. [Figure 10] FIG. 10 is a diagram schematically illustrating a data flow in the leg length measurement process of FIG. 9. [Figure 11] FIG. 1 is a diagram illustrating an example of a development workflow and platform for providing an application executed on an information processing device to which the present disclosure is applied. [Figure 12] 10 is a flowchart illustrating the flow of a shoe try-on and purchase process. [Figure 13] FIG. 10 is a diagram showing a first example of a display of a foot length measurement application. [Figure 14] FIG. 10 is a diagram showing a second example of a display of the foot length measurement application. [Figure 15] FIG. 10 is a diagram showing a third example of a display of the foot length measurement application. [Figure 16] FIG. 10 is a diagram showing a fourth example of a display of the foot length measurement application. [Figure 17] FIG. 1 is a diagram illustrating an example of the configuration of a system including a device that performs AI processing. [Figure 18] FIG. 1 is a block diagram illustrating an example of the configuration of an electronic device. [Figure 19] FIG. 2 is a block diagram illustrating a configuration example of an edge server or a cloud server. [Figure 20] FIG. 2 is a block diagram showing a configuration example of an optical sensor. [Figure 21] FIG. 2 is a block diagram showing an example of the configuration of a processing unit. [Figure 22] FIG. 1 is a diagram illustrating the flow of data between multiple devices. DETAILED DESCRIPTION OF THE INVENTION

[0011] <1. First embodiment>

[0012] (Device configuration example) FIG. 1 is a block diagram showing an example configuration of an information processing device to which the present disclosure is applied.

[0013] The information processing device 1 has a function of measuring the size of a target using captured image data. The information processing device 1 is configured as a mobile terminal such as a smartphone, tablet terminal, or mobile phone. As the size of the target, the information processing device 1 can measure the foot size of the user who owns the mobile terminal.

[0014] 1, the information processing device 1 includes a depth sensor 11, a depth processing unit 12, an RGB sensor 13, an RGB processing unit 14, a processing unit 15, a display unit 16, and an operation unit 17.

[0015] The depth sensor 11 is a distance measurement sensor such as a ToF (Time of Flight) sensor. The ToF sensor may be either a dToF (direct Time of Flight) sensor or an iToF (indirect Time of Flight) sensor. The depth sensor 11 measures the distance to an object and supplies the resulting distance measurement signal to the depth processing unit 12. The depth sensor 11 may also be a structure light sensor, a LiDAR (Light Detection and Ranging) sensor, a stereo camera, or the like.

[0016] The depth processing unit 12 is a signal processing circuit such as a DSP (Digital Signal Processor). The depth processing unit 12 performs signal processing such as depth development processing and depth pre-processing (e.g., resizing processing) on ​​the ranging signal supplied from the depth sensor 11, and supplies the resulting depth image data to the processing unit 15. A depth image is an image that represents an object using depth information. For example, a depth map is used as the depth image. The depth processing unit 12 may be included in the depth sensor 11.

[0017] The RGB sensor 13 is an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. The RGB sensor 13 captures an image of the target and supplies the resulting image signal to the RGB processing unit 14. Note that the image of the target may be captured not only by an RGB camera using the RGB sensor 13, but also by a monochrome camera, an infrared camera, or the like.

[0018] The RGB processing unit 14 is a signal processing circuit such as a DSP. The RGB processing unit 14 performs signal processing such as RGB development processing and RGB pre-processing (e.g., resizing processing) on ​​the image signal supplied from the RGB sensor 13, and supplies the resulting RGB image data to the processing unit 15. An RGB image is an image that represents an object image using color information (surface information). For example, a color camera image is used as the RGB image. The RGB processing unit 14 may be included in the RGB sensor 13.

[0019] The processing unit 15 is configured by a processor such as a CPU (Central Processing Unit), etc. The processing unit 15 receives the depth image data from the depth processing unit 12 and the RGB image data from the RGB processing unit .

[0020] The processing unit 15 performs a length measurement process to measure the size of the target based on the depth image data and RGB image data. When performing a foot length measurement process to measure the size of the user's foot as the target size, the depth image and the RGB image include the user's foot (a part of the foot beyond the ankle) as the subject. Note that the length measurement process uses at least one of the depth image and the RGB image, but does not necessarily use both images.

[0021] In the length measurement process, the depth image data, RGB image data, and at least a portion of the information obtained from these image data are processed using a trained model trained by machine learning, and the size of the object is measured. The size of the object measured in the length measurement process is supplied to the display unit 16.

[0022] The display unit 16 is configured with a panel such as a liquid crystal panel or an OLED (Organic Light Emitting Diode) panel, a signal processing circuit, etc. The display unit 16 displays information such as the size of the object supplied from the processing unit 15.

[0023] The operation unit 17 is configured with physical buttons, a touch panel, etc. The operation unit 17 supplies an operation signal corresponding to an operation by a user to the processing unit 15. The processing unit 15 performs various processes based on the operation signal from the operation unit 17.

[0024] 1 is an example, and components may be deleted or other components may be added. For example, if only a depth image is used in the length measurement process in the processing unit 15, there is no need to provide the RGB sensor 13 and the RGB processing unit 14. The information processing unit 1 may also be provided with a communication unit for exchanging data with a server on the Internet, a storage unit for recording various data and programs, an input unit such as a microphone, an output unit such as a speaker, and the like.

[0025] (Example of processing unit configuration) FIG. 2 is a block diagram showing a first example of the configuration of the processing unit 15 in FIG.

[0026] 2, the processing unit 15A has a trained model 111, a 3D coordinate calculation unit 112, a foot size and posture calculation unit 113, and a trained model 114. In the processing unit 15A, the foot size of the user is measured as the size of the object.

[0027] The trained model 111 is a model that has been trained using a deep neural network (DNN) during training. By using the trained model 111 during inference, 2D feature points related to the feet can be predicted from a depth image or an RGB image.

[0028] Hereinafter, the trained model 111 trained using a deep neural network will also be referred to as DNN1 to distinguish it from other trained models. Training of DNN1 will be described later with reference to FIG. 11 .

[0029] Depth images or RGB images obtained by photographing the user's feet are supplied to the processing unit 15A as measurement data, and are input to the trained model 111. The processing unit 15A receives the depth images or RGB images as input and performs inference using the trained model 111, thereby outputting 2D feature points related to the feet.

[0030] For example, the 2D feature points include at least three feature points: the tip of the finger, the base of the thumb, and the heel. The 2D feature points are represented by 2D coordinates. Increasing the number of 2D feature points can improve measurement accuracy.

[0031] The 2D feature points output from the trained model 111 are supplied to the 3D coordinate calculation unit 112. In addition, a depth image as measurement data is supplied to the 3D coordinate calculation unit 112.

[0032] Here, when a depth image is input to the trained model 111, the same depth image is supplied to the 3D coordinate calculation unit 112. On the other hand, when an RGB image is input to the trained model 111, a depth image captured at approximately the same timing as the RGB image is supplied to the 3D coordinate calculation unit 112.

[0033] The 3D coordinate calculation unit 112 calculates 3D coordinates corresponding to the 2D feature points using the depth image and inherent parameters. Specifically, a point cloud, which is a collection of 3D coordinates (x, y, z), is generated from the depth image using the camera parameters at the time of shooting. By using this point cloud, it is possible to obtain the coordinates (x, y, z) of the 3D feature points corresponding to the coordinates (X, Y) of the 2D feature points related to the feet.

[0034] The 3D feature points calculated by the 3D coordinate calculation unit 112 are supplied to the foot size and posture calculation unit 113 and the trained model 114.

[0035] The foot size and posture calculation unit 113 calculates the foot size and posture of the user using information such as 3D feature points.

[0036] Foot size includes foot length, which is the distance from the heel to the toes, foot width, which is the distance from the base of the big toe to the base of the little toe, and foot height, which is the height from the ground to the instep. Below, we will explain how to calculate foot length as foot size.

[0037] In addition, when calculating your foot size, the toe joints are bent (toes are bent) etc. In this case, calculating the foot size as is will not result in an accurate value. Therefore, in order to take these conditions into account, the foot posture is calculated. The foot posture is expressed by a vector that represents a position in space in the camera coordinate system.

[0038] The foot size and foot posture calculated by the foot size and posture calculation unit 113 are supplied to the trained model 114.

[0039] The trained model 114 is a model that has been trained using a deep neural network during training. By using the trained model 114 during inference, it is possible to predict a corrected foot size from 3D feature points, foot size, and foot posture.

[0040] Hereinafter, the trained model 114 trained using a deep neural network will also be referred to as DNN2 to distinguish it from other trained models. Training of DNN2 will be described later with reference to FIG. 11 .

[0041] The trained model 114 is supplied with the 3D feature points from the 3D coordinate calculation unit 112 and the foot size and foot posture from the foot size posture calculation unit 113. The processing unit 15A receives the 3D feature points, foot size, and foot posture as input, performs inference using the trained model 114, and outputs a corrected foot size.

[0042] For example, if the joints of the toes are bent, calculating the length from the toes to the heel will not result in an accurate foot size. However, in the trained model 114, this information is input as the foot posture, so the input foot size can be corrected to the foot size when the joints of the toes are not bent, and the corrected foot size can be output.

[0043] User information and other measurement results may be input to the trained model 114. For example, the user information may include information about the user being measured, such as gender and age. Other measurement results may include measurement results of foot sizes other than foot length (for example, measurement results of toe length). By adding user information and the like to the input, the accuracy of the prediction results by the trained model 114 can be further improved.

[0044] In the trained model 114, if the difference between the input and output values, i.e., the difference between the foot size value before correction and the foot size value after correction, exceeds a predetermined threshold, the corrected foot size value does not need to be used as the prediction result. That is, since data such as depth images and 3D feature points contain errors and a large difference between the input and output values ​​is likely to be an incorrect value, prediction results are aggregated in the time direction to remove outliers.

[0045] For example, in the information processing device 1, depth images and RGB images are acquired at predetermined time intervals, and foot length measurement processing can be performed according to the timing at which these images are acquired.However, since the corrected foot size can be obtained from the prediction results excluding outliers, the accuracy of foot size measurement can be improved.

[0046] The corrected foot size output from the trained model 114 is supplied to the display unit 16. The processing unit 15A may perform a predetermined process on the corrected foot size before supplying it to the display unit 16. The display unit 16 displays information according to the corrected foot size supplied from the processing unit 15A.

[0047] In the processing unit 15A configured as described above, the depth image or RGB image is processed using the trained model 111 as DNN1, and the 3D feature points, foot size, and foot posture obtained by processing the depth image or RGB image are processed using the trained model 114 as DNN2, thereby obtaining a corrected foot size.

[0048] In this way, when measuring the size of an object, inference is performed using trained models, DNN1 and DNN2, so as the prediction accuracy of the trained models improves, more accurate size measurements can be made.In addition, when performing inference using DNN2, the posture of the object is taken into consideration, so accurate size measurements can be made even if the object is not in a suitable state for size measurement.

[0049] Furthermore, the accuracy of prediction results can be improved by adding user information or other measurement results as inputs to DNN2. Also, if there is a large difference between the input and output values ​​of DNN2, it can be excluded from the prediction results as an outlier, further improving the measurement accuracy.

[0050] (Foot length measurement process flow) Next, the flow of the foot length measurement process executed by the processing unit 15A in Fig. 2 will be described with reference to the flowchart in Fig. 3. Fig. 4 schematically shows the data flow in the foot length measurement process shown in Fig. 3, and will be referred to as appropriate in the description.

[0051] The process of the flowchart in FIG. 3 starts when a user points the information processing device 1, such as a mobile terminal, at his or her foot and takes a photograph.

[0052] In step S11, the processing unit 15A acquires a depth image from the depth processing unit 12 or an RGB image from the RGB processing unit 14. For example, a depth map is acquired as the depth image, or a color camera image is acquired as the RGB image (S11 in FIG. 4).

[0053] In step S12, the processing unit 15A outputs 2D feature points by performing inference using the trained model 111 with the acquired depth image or RGB image as input. For example, by performing inference using the trained model 111 trained as DNN1 with the depth map or color camera image as input, the coordinates (100, 25) of the fingertip, the coordinates (85, 58) of the base of the thumb, and the coordinates (65, 157) of the heel are output as 2D feature points (S12 in FIG. 4).

[0054] In step S13, the 3D coordinate calculation unit 112 calculates 3D feature points corresponding to the 2D feature points. For example, a point cloud, which is a collection of 3D coordinates (x, y, z), can be generated from a depth map using camera parameters (information on the viewing angle of the depth sensor 11, etc.) at the time of shooting. Using this point cloud, the coordinates of the fingertip (15,170,600), the coordinates of the base of the thumb (-2,100,500), and the coordinates of the heel (-45,85,600) are calculated as 3D feature points corresponding to the 2D feature points (S13 in FIG. 4).

[0055] In step S14, the foot size and posture calculation unit 113 calculates the foot size and posture. For example, as the foot size, the foot length is calculated by performing a calculation using the 3D coordinates of the toes and heel (S14 in FIG. 4). As the foot posture, the foot length is calculated by performing a calculation using information such as 3D feature points, and the posture is calculated by determining whether the toe joints are bent. etc. For example, the foot posture is expressed as a 3D vector in the camera coordinate system.

[0056] In step S15, the processing unit 15A outputs a corrected foot size by performing inference using the trained model 114 with the 3D feature points, foot size, and foot posture as input. For example, when the trained model 114 trained as DNN2 is used and the 3D feature points (3D coordinates), the foot size (distance) which is the foot length, and the foot posture (3D vector) which indicates that the toe joints are bent are input, the processing unit 15A corrects the input foot length to the foot length when the toe joints are not bent, and outputs the corrected foot length (Foot Size) (S15 in FIG. 4).

[0057] In this way, the joints of the toes are bent etc.Therefore, by inputting the foot posture into the trained model 114, the foot size measured in those states will not be an accurate value. In addition, by inputting user information such as gender and age (Option User Information in FIG. 4) and other measurement results such as toe length into the trained model 114, the accuracy rate of the prediction result can be improved. Note that the user can register their own user information in advance by operating an application started on the information processing device 1.

[0058] Furthermore, in the foot length measurement process, the corrected foot size can be calculated each time a depth image or RGB image is acquired, so each time the corrected foot size is obtained, it is compared with the foot size before correction, and if the difference is large, it is removed as an outlier. This allows the foot size to be finally calculated with the outliers removed, thereby improving the accuracy of foot size measurement.

[0059] In the processing of step S12, an example of obtaining 2D feature points from a depth image or an RGB image is shown. However, since a depth image contains less information than an RGB image, the amount of calculation required during processing can be reduced by using a depth image.

[0060] As described above, in the foot length measurement process, processing is performed using the trained model 111 trained as DNN1 and the trained model 114 trained as DNN2, and a corrected foot size is obtained from an image such as a depth image obtained by photographing the user's foot. Therefore, as the prediction accuracy of the trained models (DNN1, DNN2) improves, the foot size can be measured more accurately. Furthermore, in the foot length measurement process, the corrected foot size can be obtained simply by inputting an image such as a depth image, and therefore, as the processing capacity of the processing unit 15A improves, the foot size can be measured more quickly.

[0061] <2. Second Embodiment>

[0062] In the above explanation, the configuration and processing flow when 2D feature points are output by using the trained model 111 trained as DNN1 in the processing unit 15 in Fig. 1 have been shown, but 3D feature points may also be output. Next, the configuration and processing flow when the output of the trained model in the previous stage in the processing unit 15 in Fig. 1 is 3D feature points will be explained.

[0063] (Example of processing unit configuration) FIG. 5 is a block diagram showing a second example of the configuration of the processing unit 15 in FIG.

[0064] 5, the processing unit 15B has a trained model 211, a foot size and posture calculation unit 113, and a trained model 114. In the processing unit 15B, the foot size of the user is measured as the size of the object.

[0065] In processing unit 15B, parts corresponding to processing unit 15A (FIG. 2) are assigned the same reference numerals. That is, in processing unit 15B, compared to processing unit 15A, a trained model 211 is provided instead of trained model 111 and 3D coordinate calculation unit 112.

[0066] The trained model 211 is a model that has been trained using a deep neural network during training. By using the trained model 211 during inference, 3D feature points related to the feet can be predicted from the depth image.

[0067] Hereinafter, the trained model 211 trained using a deep neural network will also be referred to as DNN3 to distinguish it from other trained models. Training of DNN3 will be described later with reference to FIG. 11.

[0068] Depth images obtained by photographing the user's feet are supplied to the processing unit 15B as measurement data, and are input to the trained model 211. The processing unit 15B uses the depth images as input and performs inference using the trained model 211, thereby outputting 3D feature points related to the feet. For example, the 3D feature points include at least three feature points: the fingertips, the base of the big toe, and the heel. The 3D feature points are represented by 3D coordinates.

[0069] The 3D feature points output from the trained model 211 are supplied to the foot size and posture calculation unit 113 and the trained model 114. Explanations of the foot size and posture calculation unit 113 and the trained model 114 would be redundant and will be omitted where appropriate.

[0070] The foot size and posture calculation unit 113 calculates the foot size and posture based on information such as the 3D feature points, etc. The processing unit 15B receives the 3D feature points, the foot size, and the foot posture as input, performs inference using the trained model 114, and outputs a corrected foot size.

[0071] In the processing unit 15B configured as above, the depth image is processed using the trained model 211 as DNN3, and the 3D feature points, foot size, and foot posture obtained by processing the depth image are processed using the trained model 114 as DNN2, thereby obtaining a corrected foot size. That is, in comparison with the processing unit 15A (FIG. 2), the processing unit 15B directly obtains 3D feature points, not 2D feature points, using a trained model at a previous stage that receives an image as input.

[0072] In this way, when measuring the size of an object, inference is performed using trained models such as DNN3 and DNN2, so as the prediction accuracy of the trained models improves, more accurate size measurements can be made. Furthermore, when performing inference using DNN2, the posture of the object is taken into consideration, so accurate size measurements can be made even if the object is not in a suitable state for size measurement.

[0073] (Foot length measurement process flow) Next, the flow of the foot length measurement process executed by the processing unit 15B in Fig. 5 will be described with reference to the flowchart in Fig. 6. Fig. 7 schematically shows the data flow in the foot length measurement process shown in Fig. 6, and will be referred to as appropriate in the description.

[0074] The process of the flowchart in FIG. 6 starts when a user points the information processing device 1, such as a mobile terminal, at his or her foot and takes a photograph.

[0075] In step S21, the processing unit 15B acquires a depth image from the depth processing unit 12. For example, a depth map is acquired as the depth image (S21 in FIG. 7).

[0076] In step S22, the processing unit 15B outputs 3D feature points by performing inference using the acquired depth image as input using the trained model 211. For example, by performing inference using the trained model 211 trained as DNN3 and using the depth map as input, the coordinates of the fingertip (15,170,600), the coordinates of the base of the thumb (-2,100,500), and the coordinates of the heel (-45,85,600) are output as 3D feature points (S22 in FIG. 7).

[0077] In steps S23 and S24, similar to steps S14 and S15 in FIG. 3 described above, the foot size and foot posture are calculated by the foot size posture calculation unit 113, and inference is performed using the trained model 114 trained as DNN2 with the 3D feature points, foot size, and foot posture as input, and the corrected foot size is output (S23 and S24 in FIG. 7).

[0078] As described above, in the foot length measurement process, the trained model 211 trained as DNN3 and the trained model 114 trained as DNN2 are used to obtain a corrected foot size from a depth image obtained by photographing the user's foot. Therefore, as the prediction accuracy of the trained models (DNN3, DNN2) improves, the foot size can be measured more accurately.

[0079] <3. Third Embodiment>

[0080] In the above explanation, the configuration and processing flow when two trained models are used in the processing unit 15 of Fig. 1 are shown, but one trained model may be used. Next, the configuration and processing flow when one trained model is used in the processing unit 15 of Fig. 1 to output a corrected foot size will be explained.

[0081] (Example of processing unit configuration) FIG. 8 is a block diagram showing a third example of the configuration of the processing unit 15 in FIG.

[0082] 8, processing unit 15C has a trained model 311. In processing unit 15C, the user's foot size is measured as the target size.

[0083] The trained model 311 is a model that has been trained by a deep neural network during training. By using the trained model 311 during inference, it is possible to predict a corrected foot size from a depth image.

[0084] Hereinafter, the trained model 311 trained using a deep neural network will also be referred to as DNN4 to distinguish it from other trained models. Training of DNN4 will be described later with reference to FIG. 11.

[0085] Depth images obtained by photographing the user's feet are supplied to the processing unit 15C as measurement data, and are input to the trained model 311. The processing unit 15C uses the depth images as input and performs inference using the trained model 311, thereby outputting a corrected foot size.

[0086] For example, when the joints of the user's toes are bent, the foot size predicted from the depth image will not be an accurate value. Therefore, the trained model 311 learns features such as foot posture during training, so that it outputs a foot size corrected to a measurement in an ideal state (when the joints of the toes are not bent).

[0087] In the configuration of the processing unit 15C shown in FIG. 8, a depth image is input to the trained model 311, but an RGB image may also be input.

[0088] In the processing unit 15C configured as described above, a corrected foot size is obtained by processing the depth image using the trained model 311 as DNN4. In this way, when measuring the size of the target, inference is performed using the trained model as DNN4, and therefore, as the prediction accuracy of the trained model improves, the size can be measured more accurately.

[0089] (Foot length measurement process flow) Next, the flow of the foot length measurement process executed by the processing unit 15C in Fig. 8 will be described with reference to the flowchart in Fig. 9. Fig. 10 schematically shows the flow of data in the foot length measurement process shown in Fig. 9, and will be referred to as appropriate in the description.

[0090] The process of the flowchart in FIG. 9 starts when a user points the information processing device 1, such as a mobile terminal, at his or her foot and takes a photograph.

[0091] In step S31, the processing unit 15C acquires a depth image from the depth processing unit 12. For example, a depth map is acquired as the depth image (S31 in FIG. 10).

[0092] In step S32, the processing unit 15C outputs a corrected foot size by performing inference using the acquired depth image as an input using the trained model 311. For example, when the user's toe joints are bent when taking a photo with the mobile terminal, a foot size corrected to a measurement in an ideal state is output (S32 in FIG. 10).

[0093] As described above, in the foot length measurement process, the trained model 311 trained as DNN4 is used to obtain a corrected foot size from a depth image obtained by photographing the user's foot. Therefore, as the prediction accuracy of the trained model (DNN4) improves, the foot size can be measured more accurately.

[0094] The trained model 311 is trained to output corrected foot size when a depth image is input, but during training, rather than just inputting training data into DNN4, correct answer data may also be provided midway through DNN4 to allow training to take place.

[0095] <4. Development Workflow Platform>

[0096] FIG. 11 is a diagram showing an example of a development workflow and platform for providing an application executed on an information processing device to which the present disclosure is applied.

[0097] In FIG. 11, an application developed using an information processing device 2 such as a PC (Personal Computer) is provided to and installed in an information processing device 1 such as a mobile terminal.

[0098] Algorithm development and application development are carried out in the information processing device 2. In the algorithm development, a program for foot measurement processing (foot measure code) and a trained model to be called when the foot measurement processing is executed are developed.

[0099] In the algorithm development, a trained model is generated by machine learning using training data. The information processing device 2 can acquire a large amount of training data by storing depth images captured by the image capture device 3 running a photography application in the database 4. The training data can include RGB images.

[0100] Annotation work for the learning data is performed in the information processing device 2. For example, a developer uses a GUI tool (GUI Labeling Tool) to label feature points (for example, correct feature points such as the toes or heels) at specific parts of the feet included in the depth images as the learning data, thereby generating training data.

[0101] In addition, data augmentation can be performed, for example, by enlarging or flipping an existing image, which increases the variety of learning data used in machine learning. This allows for the addition of information that cannot be covered by capturing images using the image capture device 3 alone.

[0102] In the information processing device 2, a trained model is generated by performing machine learning by deep learning using training data. For example, any of the above-mentioned DNN1 to DNN4 can be generated as the trained model.

[0103] More specifically, when a depth image or RGB image is input, DNN1 is expected to output 2D feature points such as fingertips, but in the early stages of learning, it outputs feature points of parts other than fingertips, etc. Here, by labeling the correct 2D feature points and repeating learning, the correct 2D feature points such as fingertips will be output, and the learning of DNN1 will converge.

[0104] Similarly, when DNN3 receives a depth image as input, it is expected to output 3D feature points, so by labeling the correct 3D feature points and repeating the learning process, it will be able to output the correct 3D feature points.

[0105] Furthermore, when DNN2 receives 3D feature points, foot size, and foot posture as input, it is expected to output a corrected foot size, so by repeatedly learning about human foot features, it will be able to output the correct foot size.When DNN4 receives a depth image as input, it is expected to output a corrected foot size, so by repeatedly learning about human foot features, it will be able to output the correct foot size.

[0106] In addition, when user information or other measurement results are input, DNN2 and DNN4 perform learning taking that information into consideration. Furthermore, when training a trained model, in addition to inputting training data into the DNN, it is also possible to provide correct answer data midway through the DNN to perform learning.

[0107] By performing machine learning by preparing more training data and labeled data, or by increasing the variety of training data through data augmentation, the prediction accuracy of the trained model (DNN1, DNN2, etc.) can be improved. Note that it is desirable to use a high-performance PC as the information processing device 2.

[0108] The trained model (DNN1, DNN2, etc.) generated in this way is called as appropriate when the foot length measurement process is executed, and outputs a prediction result for the input.

[0109] In the application development, the foot length measurement processing program and the trained model developed in the algorithm development are used to develop an application (hereinafter also referred to as a foot length measurement application) that utilizes the foot size obtained by the foot length measurement processing. The foot length measurement application developed in this way is provided to and installed in the information processing device 1 via a server on the Internet or the like.

[0110] In the information processing device 1, when a foot measurement application is started and an image of the user's foot (a part of the ankle) is taken, a foot measurement process is executed and the foot size is displayed. When the foot measurement process is executed, the foot measurement application appropriately calls a trained model such as DNN1 or DNN2 to obtain a prediction result for the input as an output, and then performs processing using the output.

[0111] 11 shows a case where algorithm development and application development are performed on the same information processing device 2 such as a single PC, but algorithm development and application development may be performed on different information processing devices.Also, algorithm development and application development may each be performed on multiple information processing devices.

[0112] Also, in Figure 11, when developing an algorithm, learning is performed using training data to generate a trained model, but learning can also be performed without training data.

[0113] <5. Use Cases>

[0114] The foot length measurement application can provide various services. For example, it can use AR (Augmented Reality) technology to allow users to try on shoes that fit their foot size (so-called AR try-on), and if the user likes the shoes, they can purchase them via EC (Electronic Commerce).

[0115] (Shoe try-on purchase process) The flow of the shoe try-on and purchase process executed by the information processing device 1 will be described with reference to the flowchart of FIG.

[0116] In the information processing device 1, a foot length measurement application is started when the shoe try-on and purchase process is executed.

[0117] In step S111, the processing unit 15 determines whether or not the desired shoes have been selected by the user, based on the operation signal from the operation unit 17. If it is determined in step S111 that the desired shoes have been selected by the user, the process proceeds to step S112.

[0118] In step S112, the processing unit 15 starts the foot length measurement process. When starting the foot length measurement process, the user points the information processing device 1 toward his or her own foot (a part beyond the ankle), and the depth sensor 11 and the RGB sensor 13 photograph (measure) the user's foot.

[0119] In this foot length measurement process, the process shown in any one of the three embodiments described above is performed. That is, the image including the depth image and at least a part of the information obtained from the image are processed using a trained model that is DNN1 and DNN2, DNN3 and DNN2, or DNN4.

[0120] In step S113, the processing unit 15 superimposes the AR image of the selected shoe on the user's foot included in the captured RGB image, and displays it on the display unit 16.

[0121] In step S114, the processing unit 15 displays on the display unit 16 the progress status linked to variations in foot posture, the imaging time, the time required for the foot length measurement process, and the like.

[0122] 13, on the display unit 16 of the information processing device 1, an AR image 521 of the shoes selected by the user is superimposed on the part of the user's feet included in a captured screen 511 corresponding to the captured RGB image. A known technique can be used for the superimposed display of the AR image. Since the foot length measurement process can recognize the posture of the foot, an image marker (AR marker) is not required.

[0123] Furthermore, a progress status 531 is displayed on the photographing screen 511. While the user is trying on shoes in AR, a foot length measurement process is performed. That is, the foot length measurement process requires a certain amount of time because it involves calculating the foot size and removing outliers from values ​​obtained sequentially in the time direction, but the progress is presented taking that time into consideration.

[0124] The progress may be presented as a percentage in a doughnut-shaped graph, as in progress status 531 in Fig. 13, or as a percentage in a bar-shaped graph, as in progress status 532 in Fig. 14. The progress display is not limited to doughnut-shaped or bar-shaped graphs, and other display formats may be used. Furthermore, the progress presentation method is not limited to display, and other presentation methods such as sound output or vibration may also be used.

[0125] If the position of the user's feet is too close or too far from the information processing device 1, the processing unit 15 can display a message to that effect on the display unit 16 based on information obtained in the foot length measurement process and information from other sensors. For example, as shown in Fig. 15, if the position of the user's feet is too close, a message 541 is displayed. By presenting the message, the user can be guided to move the information processing device 1 or their own feet to a distance appropriate for foot length measurement.

[0126] 12, in step S115, the processing unit 15 determines whether the foot length measurement process has ended. If it is determined in step S115 that the foot length measurement process has not ended, the process returns to step S113, steps S113 and S114 are repeated, and the progress is displayed together with the AR image.

[0127] On the other hand, if it is determined in step S115 that the foot length measurement process has ended, the process proceeds to step S116. In step S116, the processing unit 15 displays the foot size obtained by the foot length measurement process on the display unit 16. This foot size is a corrected foot size, and has been corrected to a value measured in an ideal state, for example, when the user's toe joints are bent during image capture by the information processing device 1.

[0128] 16, on the display unit 16 of the information processing device 1, an AR image 521 of shoes is superimposed on a photographing screen 511, and a foot size 551 is also displayed. This allows the user to recognize their own foot size.

[0129] Returning to FIG. 12, in step S117, the processing unit 15 determines, based on an operation signal from the operation unit 17, whether or not the user has selected to purchase the shoes being tried on in AR.

[0130] If it is determined in step S117 that the purchase of shoes has been selected, the process proceeds to step S118. In step S118, the processing unit 15 performs a product purchase process.

[0131] For example, as shown in FIG. 16 , a button 552 for purchasing the shoes that the user has tried on in AR is displayed on the display unit 16 of the information processing device 1. If the user wishes to purchase the shoes that the user has tried on in AR, the user taps the button 552, which displays a purchase screen. The user performs the necessary operations on the purchase screen to perform processing such as payment, and can purchase the shoes that the user has tried on in AR. On the purchase screen, the user can purchase shoes according to the corrected foot size obtained by the foot length measurement processing, but the user may also be able to input or change their own foot size after checking the foot size 551.

[0132] The foot length measurement application may provide a function to try on or purchase shoes similar to the selected shoes in AR, not limited to the shoes selected by the user. The foot length measurement application may also calculate and display a fit rate for each shoe based on information obtained by the foot length measurement process and information from other sensors. The user can check this information to decide whether to purchase the shoes.

[0133] Furthermore, the information processing device 1 can access a server on the Internet, transmit the corrected foot size obtained by the foot length measurement process and user information (gender, age, etc.), and request confirmation of shoes according to the user's characteristics. The foot length measurement application can recommend shoes according to the user's characteristics based on the response from the server.

[0134] The series of processes ends when the process of step S118 ends. Also, if it is determined in step S117 that the purchase of shoes has not been selected, the process of step S118 is skipped and the series of processes ends.

[0135] The above explains the flow of the shoe try-on and purchase process. In the shoe try-on and purchase process, the user is shown a more accurate foot size and can try on and purchase the desired shoes using AR. This increases the percentage of users who end up purchasing the desired shoes.

[0136] <6. Variations>

[0137] (Other examples of target sizes) In the above description, foot size is exemplified as the size of the target object, but the information processing device 1 may measure other parts of the user's body and superimpose AR images of clothing, accessories, etc. according to the measured parts. Furthermore, the information processing device 1 may display the size of the measured parts.

[0138] For example, by processing a depth image obtained by photographing a user and at least a portion of the information obtained from the image using a trained model, the user's shoulder width, chest width, etc. are measured, and an AR image of clothing is superimposed and displayed on the user's upper body part included in the simultaneously captured RGB image. Alternatively, by measuring the circumference of the user's finger, an AR image of a ring is superimposed and displayed on the user's finger part included in the captured RGB image. Furthermore, the measured shoulder width, chest width, finger circumference, etc. may be displayed together with the AR image of the clothing or ring.

[0139] (rule-based application) In the above description, in the information processing device 1, the processing unit 15 performs processing using a trained model trained by machine learning, but part of the processing may be performed on a rule basis.

[0140] (Examples of other configurations) FIG. 17 shows an example of the configuration of a system including a device that performs AI processing.

[0141] The electronic device 20001 is a mobile terminal such as a smartphone, a tablet terminal, or a mobile phone. The electronic device 20001 corresponds to, for example, the information processing device 1 in FIG. 1 and has an optical sensor 20011 corresponding to the depth sensor 11 (FIG. 1). The optical sensor is a sensor (image sensor) that converts light into an electrical signal. The electronic device 20001 can connect to a network 20040 such as the Internet via a core network 20030 by connecting to a base station 20020 installed at a predetermined location via wireless communication corresponding to a predetermined communication method.

[0142] An edge server 20002 for implementing Mobile Edge Computing (MEC) is provided at a position closer to the mobile terminal, such as between the base station 20020 and the core network 20030. A cloud server 20003 is connected to the network 20040. The edge server 20002 and the cloud server 20003 can perform various processes according to the application. The edge server 20002 may be provided within the core network 20030.

[0143] AI processing is performed by the electronic device 20001, the edge server 20002, the cloud server 20003, or the optical sensor 20011. The AI ​​processing is processing the technology according to the present disclosure using AI such as machine learning. The AI ​​processing includes a learning process and an inference process. The learning process is a process of generating a trained model. The learning process also includes a re-learning process, which will be described later. The inference process is a process of performing inference using the trained model. The trained model can include at least any of the above-mentioned DNN1 to DNN4.

[0144] In the electronic device 20001, the edge server 20002, the cloud server 20003, or the optical sensor 20011, AI processing is realized by a processor such as a CPU (Central Processing Unit) executing a program or by using dedicated hardware such as a processor specialized for a specific application. For example, a GPU (Graphics Processing Unit) can be used as the processor specialized for a specific application.

[0145] 18 shows an example configuration of electronic device 20001. Electronic device 20001 has a CPU 20101 that controls the operation of each unit and performs various processes, a GPU 20102 that is specialized for image processing and parallel processing, a main memory 20103 such as a DRAM (Dynamic Random Access Memory), and an auxiliary memory 20104 such as a flash memory.

[0146] The auxiliary memory 20104 stores data such as programs for AI processing and various parameters. The CPU 20101 loads the programs and parameters stored in the auxiliary memory 20104 into the main memory 20103 and executes the programs. Alternatively, the CPU 20101 and the GPU 20102 load the programs and parameters stored in the auxiliary memory 20104 into the main memory 20103 and execute the programs. This allows the GPU 20102 to be used as a general-purpose computing on graphics processing units (GPGPU).

[0147] The CPU 20101 and GPU 20102 may be configured as a System on a Chip (SoC). When the CPU 20101 executes a program for AI processing, the GPU 20102 does not need to be provided.

[0148] The electronic device 20001 also has an optical sensor 20011 to which the technology disclosed herein is applied, an operation unit 20105 such as a physical button or touch panel, a sensor 20106 including at least one sensor, a display 20107 for displaying information such as images and text, a speaker 20108 for outputting sound, a communication I / F 20109 such as a communication module compatible with a predetermined communication method, and a bus 20110 connecting them.

[0149] The sensor 20106 has at least one of various sensors such as an optical sensor (image sensor), a sound sensor (microphone), a vibration sensor, an acceleration sensor, an angular velocity sensor, a pressure sensor, an odor sensor, and a biosensor. In AI processing, data acquired from at least one of the sensors 20106 can be used along with data (image data) acquired from the optical sensor 20011. That is, the optical sensor 20011 corresponds to the depth sensor 11 (FIG. 1), and the sensor 20106 corresponds to the RGB sensor 13 (FIG. 1).

[0150] Data acquired from two or more optical sensors using sensor fusion technology or data obtained by processing the data in an integrated manner may be used in AI processing. The two or more optical sensors may be a combination of optical sensor 20011 and an optical sensor in sensor 20106, or optical sensor 20011 may include multiple optical sensors. For example, optical sensors include RGB visible light sensors, distance sensors such as ToF (Time of Flight), polarization sensors, event-based sensors, sensors that acquire IR images, and sensors capable of acquiring multiple wavelengths.

[0151] In electronic device 20001, AI processing can be performed by a processor such as CPU 20101 or GPU 20102. When the processor of electronic device 20001 performs inference processing, the processing can be started immediately after image data is acquired by optical sensor 20011, allowing for high-speed processing. Therefore, when inference processing is used in electronic device 20001 for applications that require information transmission with short latency, the user can operate the device without experiencing discomfort due to delays. Furthermore, when the processor of electronic device 20001 performs AI processing, there is no need to use communication lines or computer equipment for the server, compared to using a server such as cloud server 20003, and processing can be achieved at low cost.

[0152] 19 shows an example configuration of an edge server 20002. The edge server 20002 has a CPU 20201 that controls the operation of each component and performs various processes, and a GPU 20202 that is specialized for image processing and parallel processing. The edge server 20002 also has a main memory 20203 such as a DRAM, an auxiliary memory 20204 such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and a communication I / F 20205 such as a NIC (Network Interface Card), all of which are connected to a bus 20206.

[0153] The auxiliary memory 20204 stores data such as programs for AI processing and various parameters. The CPU 20201 loads the programs and parameters stored in the auxiliary memory 20204 into the main memory 20203 and executes the programs. Alternatively, the CPU 20201 and the GPU 20202 can use the GPU 20202 as a GPGPU by loading the programs and parameters stored in the auxiliary memory 20204 into the main memory 20203 and executing the programs. Note that when the CPU 20201 executes programs for AI processing, the GPU 20202 does not need to be provided.

[0154] In the edge server 20002, AI processing can be performed by a processor such as a CPU 20201 or a GPU 20202. When the processor of the edge server 20002 performs AI processing, the edge server 20002 is located closer to the electronic device 20001 than the cloud server 20003, thereby achieving low processing latency. Furthermore, the edge server 20002 has higher processing capabilities, such as calculation speed, than the electronic device 20001 and the optical sensor 20011, and can therefore be configured for general use. Therefore, when the processor of the edge server 20002 performs AI processing, the AI ​​processing can be performed as long as data can be received, regardless of differences in the specifications and performance of the electronic device 20001 and the optical sensor 20011. When AI processing is performed in the edge server 20002, the processing load on the electronic device 20001 and the optical sensor 20011 can be reduced.

[0155] The configuration of the cloud server 20003 is similar to that of the edge server 20002, and therefore a description thereof will be omitted.

[0156] In the cloud server 20003, AI processing can be performed by a processor such as a CPU 20201 or a GPU 20202. The cloud server 20003 has higher processing capabilities, such as higher calculation speed, than the electronic device 20001 and the optical sensor 20011, and can therefore be configured for general use. Therefore, when the processor of the cloud server 20003 performs AI processing, the AI ​​processing can be performed regardless of differences in the specifications and performance of the electronic device 20001 and the optical sensor 20011. Furthermore, if it is difficult for the processor of the electronic device 20001 or the optical sensor 20011 to perform high-load AI processing, the processor of the cloud server 20003 can perform the high-load AI processing and feed back the processing results to the processor of the electronic device 20001 or the optical sensor 20011.

[0157] 20 shows an example configuration of optical sensor 20011. Optical sensor 20011 can be configured as a one-chip semiconductor device having a stacked structure in which a plurality of substrates are stacked, for example. Optical sensor 20011 is configured by stacking two substrates, substrate 20301 and substrate 20302. Note that the configuration of optical sensor 20011 is not limited to a stacked structure, and for example, a substrate including an imaging unit may include a processor that performs AI processing, such as a CPU or a DSP (Digital Signal Processor).

[0158] An imaging unit 20321 configured with a plurality of pixels arranged two-dimensionally is mounted on the upper substrate 20301. An imaging processing unit 20322 that performs processing related to image capture by the imaging unit 20321, an output I / F 20323 that outputs the captured image and signal processing results to the outside, and an imaging control unit 20324 that controls image capture by the imaging unit 20321 are mounted on the lower substrate 20302. The imaging unit 20321, imaging processing unit 20322, output I / F 20323, and imaging control unit 20324 configure an imaging block 20311.

[0159] The lower substrate 20302 is equipped with a CPU 20331 that controls each component and performs various processes, a DSP 20332 that performs signal processing using captured images and external information, a memory 20333 such as an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory), and a communication I / F 20334 that exchanges necessary information with the outside. The CPU 20331, DSP 20332, memory 20333, and communication I / F 20334 form a signal processing block 20312. AI processing can be performed by at least one of the processors, the CPU 20331 and the DSP 20332.

[0160] In this way, the signal processing block 20312 for AI processing can be mounted on the lower layer substrate 20302 in a stacked structure in which multiple substrates are stacked. As a result, image data acquired by the imaging block 20311 for imaging mounted on the upper layer substrate 20301 is processed by the signal processing block 20312 for AI processing mounted on the lower layer substrate 20302, allowing a series of processes to be performed within a single-chip semiconductor device.

[0161] In the optical sensor 20011, AI processing can be performed by a processor such as the CPU 20331. When the processor of the optical sensor 20011 performs AI processing such as inference processing, a series of processes is performed within a single-chip semiconductor device, preventing information from leaking outside the sensor and enhancing confidentiality of the information. Furthermore, since there is no need to transmit data such as image data to other devices, the processor of the optical sensor 20011 can perform AI processing such as inference processing using image data at high speed. For example, when inference processing is used in applications requiring real-time performance, real-time performance can be sufficiently ensured. Here, ensuring real-time performance means that information can be transmitted with short latency. Furthermore, when the processor of the optical sensor 20011 performs AI processing, various metadata can be passed from the processor of the electronic device 20001, thereby reducing processing and achieving lower power consumption.

[0162] 21 shows an example of the configuration of the processing unit 20401. The processing unit 20401 corresponds to the processing unit 10 in FIG. 1. The processor of the electronic device 20001, the edge server 20002, the cloud server 20003, or the optical sensor 20011 executes various processes according to a program, thereby functioning as the processing unit 20401. Note that multiple processors in the same or different devices may function as the processing unit 20401.

[0163] The processing unit 20401 has an AI processing unit 20411. The AI ​​processing unit 20411 performs AI processing. The AI ​​processing unit 20411 has a learning unit 20421 and an inference unit 20422.

[0164] The learning unit 20421 performs a learning process to generate a trained model. In the learning process, trained models such as DNN1 to DNN4 are generated. The learning unit 20421 may also perform a re-learning process to update the generated trained model. In the following explanation, a distinction is made between generating and updating a trained model, but since updating a trained model can also be said to generate a trained model, the generation of a trained model is considered to include the meaning of updating a trained model.

[0165] Furthermore, the generated trained model is recorded in a storage medium such as a main memory or auxiliary memory of the electronic device 20001, the edge server 20002, the cloud server 20003, the optical sensor 20011, or the like, so that it can be newly used in the inference processing performed by the inference unit 20422. This makes it possible to generate the electronic device 20001, the edge server 20002, the cloud server 20003, the optical sensor 20011, or the like that performs inference processing based on the trained model. Furthermore, the generated trained model may be recorded in a storage medium or electronic device independent of the electronic device 20001, the edge server 20002, the cloud server 20003, the optical sensor 20011, or the like, and provided for use in other devices. Note that generating the electronic device 20001, the edge server 20002, the cloud server 20003, the optical sensor 20011, or the like includes not only recording a new trained model in the storage medium at the time of manufacturing, but also updating an already-recorded generated trained model.

[0166] The inference unit 20422 performs inference processing using a trained model. In the inference processing, processing using trained models such as DNN1 to DNN4 is performed.

[0167] Machine learning techniques that can be used include neural networks and deep learning. A neural network is a model that mimics the neural circuits of the human brain and consists of three types of layers: an input layer, an intermediate layer (hidden layer), and an output layer. Deep learning is a model that uses a multi-layered neural network, which repeats characteristic learning in each layer and is able to learn complex patterns hidden in large amounts of data.

[0168] Supervised learning can be used to set up machine learning problems. For example, supervised learning learns features based on given labeled training data. This makes it possible to derive labels for unknown data. The training data can be image data actually acquired by an optical sensor, previously acquired image data that is aggregated and managed, or a dataset generated by a simulator.

[0169] Note that supervised learning is not the only method that can be used; unsupervised learning, semi-supervised learning, reinforcement learning, etc. can also be used. Unsupervised learning analyzes large amounts of unlabeled training data to extract features, and performs clustering etc. based on the extracted features. This makes it possible to analyze and predict trends based on huge amounts of unknown data. Semi-supervised learning is a combination of supervised and unsupervised learning, in which features are learned using supervised learning, and then a huge amount of training data is provided using unsupervised learning, and learning is repeated while the features are automatically calculated. Reinforcement learning deals with the problem of an agent in an environment observing its current state and determining the action to take.

[0170] In this way, the processor of the electronic device 20001, the edge server 20002, the cloud server 20003, or the optical sensor 20011 functions as the AI ​​processing unit 20411, and AI processing is performed in one or more of these devices.

[0171] The AI ​​processing unit 20411 may have at least one of the learning unit 20421 and the inference unit 20422. That is, the processor of each device may not only execute both the learning process and the inference process, but may also execute one of the learning process and the inference process. For example, if the processor of the electronic device 20001 performs both the inference process and the learning process, it may have the learning unit 20421 and the inference unit 20422, but if it only performs the inference process, it may have only the inference unit 20422.

[0172] The processor of each device may execute all of the processes related to the learning process or the inference process, or a portion of the processes may be executed by the processor of each device, and the remaining processes may be executed by the processor of another device. Furthermore, each device may have a common processor for executing each function of the AI ​​processes such as the learning process or the inference process, or may have a separate processor for each function.

[0173] It should be noted that AI processing may be performed by devices other than those described above. For example, AI processing may be performed by other electronic devices to which electronic device 20001 can be connected via wireless communication or the like. Specifically, if electronic device 20001 is a smartphone, the other electronic devices that perform AI processing may be other smartphones, tablet devices, mobile phones, PCs (Personal Computers), game consoles, television sets, wearable devices, digital still cameras, digital video cameras, and the like.

[0174] Furthermore, AI processing such as inference processing can be applied to configurations using sensors mounted on mobile objects such as automobiles or sensors used in remote medical devices, but such environments require short latency. In such environments, latency can be reduced by performing AI processing on a processor in a local device (e.g., electronic device 20001 as an in-vehicle device or medical device) rather than on a processor in cloud server 20003 via network 20040. Furthermore, even in cases where there is no environment for connection to network 20040 such as the Internet, or for devices used in environments where high-speed connection is not possible, AI processing can be performed in a more appropriate environment by performing AI processing on a processor in a local device such as electronic device 20001 or optical sensor 20011.

[0175] The above-described configuration is merely an example, and other configurations may be adopted. For example, the electronic device 20001 is not limited to a mobile terminal such as a smartphone, but may also be an electronic device such as a PC, a game console, a television set, a wearable terminal, a digital still camera, or a digital video camera, an in-vehicle device, or a medical device. Furthermore, the electronic device 20001 may be connected to the network 20040 via wireless communication or wired communication corresponding to a predetermined communication method such as a wireless LAN (Local Area Network) or a wired LAN. AI processing is not limited to processors such as a CPU or GPU of each device, and may also utilize a quantum computer or a neuromorphic computer.

[0176] Incidentally, data such as trained models, image data, and corrected data may be used within a single device, or may be exchanged between multiple devices and used within those devices. Figure 22 shows the flow of data between multiple devices.

[0177] Electronic devices 20001-1 to 20001-N (N is an integer equal to or greater than 1) are owned by, for example, individual users, and can each be connected to a network 20040 such as the Internet via a base station (not shown) or the like. At the time of manufacture, a learning device 20501 is connected to electronic device 20001-1, and a trained model provided by learning device 20501 can be recorded in auxiliary memory 20104. Learning device 20501 generates a trained model using a dataset generated by simulator 20502 as training data, and provides the trained model to electronic device 20001-1. Note that the training data is not limited to the dataset provided by simulator 20502, and image data actually acquired by an optical sensor, acquired image data that is aggregated and managed, or the like may also be used.

[0178] Although not shown in the figures, the electronic devices 20001-2 to 20001-N can also have trained models recorded at the time of manufacture, similar to the electronic device 20001-1. Hereinafter, when there is no need to distinguish between the electronic devices 20001-1 to 20001-N, they will be referred to as electronic device 20001.

[0179] In addition to the electronic device 20001, a learning model generation server 20503, a learning model provision server 20504, a data provision server 20505, and an application server 20506 are connected to the network 20040, and data can be exchanged between them. Each server can be provided as a cloud server.

[0180] The learning model generation server 20503 has a configuration similar to that of the cloud server 20003, and can perform learning processing using a processor such as a CPU. The learning model generation server 20503 generates a trained model using training data. The illustrated configuration illustrates a case where the electronic device 20001 records the trained model at the time of manufacture, but the trained model may also be provided from the learning model generation server 20503. The learning model generation server 20503 transmits the generated trained model to the electronic device 20001 via the network 20040. The electronic device 20001 receives the trained model transmitted from the learning model generation server 20503 and records it in the auxiliary memory 20104. As a result, the electronic device 20001 including the trained model is generated.

[0181] That is, in the electronic device 20001, if a trained model has not been recorded at the time of manufacture, a new trained model from the trained model generation server 20503 is newly recorded, thereby generating an electronic device 20001 having a new trained model recorded therein. Also, in the electronic device 20001, if a trained model has already been recorded therein at the time of manufacture, the recorded trained model is updated with the trained model from the trained model generation server 20503, thereby generating an electronic device 20001 having an updated trained model recorded therein. The electronic device 20001 can perform inference processing using a trained model that is updated as appropriate.

[0182] The trained model may not only be provided directly from the training model generation server 20503 to the electronic device 20001, but may also be provided via the network 20040 by a training model providing server 20504 that aggregates and manages various trained models. The training model providing server 20504 may provide the trained model to other devices, not just the electronic device 20001, thereby generating other devices equipped with the trained model. The trained model may also be recorded and provided on a removable memory card such as a flash memory. The electronic device 20001 can read and record the trained model from a memory card inserted in a slot. This allows the electronic device 20001 to acquire the trained model even when used in harsh environments, when it does not have a communication function, or when it has a communication function but the amount of information that can be transmitted is small.

[0183] The electronic device 20001 can provide data such as image data, corrected data, and metadata to other devices via the network 20040. For example, the electronic device 20001 transmits data such as image data and corrected data to the learning model generation server 20503 via the network 20040. This allows the learning model generation server 20503 to use data such as image data and corrected data collected from one or more electronic devices 20001 as learning data to generate a learned model. Using more learning data can improve the accuracy of the learning process.

[0184] Data such as image data and corrected data may not necessarily be provided directly from the electronic device 20001 to the learning model generation server 20503, but may also be provided by a data providing server 20505 that aggregates and manages various types of data. The data providing server 20505 may collect data from other devices, not just the electronic device 20001, and may provide data to other devices, not just the learning model generation server 20503.

[0185] The learning model generation server 20503 may update an already generated trained model by performing a re-learning process in which data such as image data and corrected data provided from the electronic device 20001 or the data providing server 20505 is added to the training data. The updated trained model can be provided to the electronic device 20001. When the learning model generation server 20503 performs a training process or a re-learning process, the process can be performed regardless of differences in the specifications or performance of the electronic device 20001.

[0186] Furthermore, when a user performs a correction operation on corrected data or metadata in the electronic device 20001 (for example, when the user inputs correct information), feedback data related to the correction process may be used in the re-learning process. For example, by transmitting feedback data from the electronic device 20001 to the learning model generation server 20503, the learning model generation server 20503 can perform a re-learning process using the feedback data from the electronic device 20001 and update the trained model. Note that when a correction operation is performed by the user, the electronic device 20001 may use an application provided by the application server 20506.

[0187] The re-learning process may be performed by the electronic device 20001. When the electronic device 20001 performs the re-learning process using image data or feedback data to update the trained model, the trained model can be improved within the device. This generates the electronic device 20001 equipped with the updated trained model. Furthermore, the electronic device 20001 may transmit the updated trained model obtained by the re-learning process to the trained model providing server 20504 so that it is provided to other electronic devices 20001. This allows the updated trained model to be shared among multiple electronic devices 20001.

[0188] Alternatively, the electronic device 20001 may transmit difference information of the re-trained trained model (difference information between the trained model before the update and the trained model after the update) as update information to the training model generation server 20503. The training model generation server 20503 can generate an improved trained model based on the update information from the electronic device 20001 and provide it to other electronic devices 20001. By exchanging such difference information, privacy can be protected and communication costs can be reduced compared to when all information is exchanged. Note that, similar to the electronic device 20001, the optical sensor 20011 mounted on the electronic device 20001 may perform the re-training process.

[0189] The application server 20506 is a server capable of providing various applications via the network 20040. The applications provide predetermined functions using data such as trained models, corrected data, and metadata. The electronic device 20001 can realize the predetermined functions by executing an application downloaded from the application server 20506 via the network 20040. Alternatively, the application server 20506 can realize the predetermined functions by acquiring data from the electronic device 20001 via, for example, an API (Application Programming Interface) and executing the application on the application server 20506.

[0190] In this way, in a system including devices to which the present disclosure is applied, data such as trained models, image data, and corrected data is exchanged and distributed between the devices, making it possible to provide various services using this data. For example, it is possible to provide a service that provides trained models via the trained model providing server 20504, and a service that provides data such as image data and corrected data via the data providing server 20505. It is also possible to provide a service that provides applications via the application server 20506.

[0191] Alternatively, image data acquired from the optical sensor 20011 of the electronic device 20001 may be input to a trained model provided by the learning model providing server 20504, and corrected data obtained as the output may be provided. Also, a device such as an electronic device that implements the trained model provided by the learning model providing server 20504 may be generated and provided. Furthermore, by recording data such as the trained model, corrected data, and metadata on a readable storage medium, a storage medium on which the data is recorded, or a device such as an electronic device equipped with the storage medium, may be generated and provided. The storage medium may be a non-volatile memory such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, or a volatile memory such as an SRAM or DRAM.

[0192] It should be noted that the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be present. In this specification, "2D" refers to two dimensions, and "3D" refers to three dimensions.

[0193] The present disclosure can also be configured as follows.

[0194] (1) The system includes a processing unit that processes an image including at least a depth image acquired by a sensor and at least a portion of information obtained from the image using a trained model trained by machine learning, and measures the size of an object included in the image. Information processing device. (2) The trained model is a deep neural network trained using at least one of the image and the information as input and the size of the object as output. The information processing device according to (1) above. (3) The trained model is a first trained model that uses the image as input and outputs feature points of the object; a second trained model that receives the feature points of the object, the size of the object, and the posture of the object as input, and outputs a corrected size obtained by correcting the size of the object; Contains The information processing device according to (2) above. (4) The second trained model receives user information about the object along with the feature points, size, and posture of the object as input, and outputs the corrected size. The information processing device according to (3) above. (5) The processing unit Calculating the size and posture of the object based on the feature points of the object output from the first trained model; The calculated size of the object and the posture of the object are input into the second trained model. The information processing device according to (3) or (4). (6) The first trained model outputs 2D feature points or 3D feature points as the feature points; The second trained model inputs 3D feature points as the feature points. The information processing device according to any one of (3) to (5). (7) When the feature points are 2D feature points, the processing unit calculates 3D feature points from the 2D feature points. The information processing device according to (6) above. (8) a display unit that displays the corrected size The information processing device according to any one of (3) to (7). (9) The display unit displays an AR image by superimposing it on a part of the image corresponding to the target included in the captured image of the user. The information processing device according to (8). (10) the object is a user's foot; The target size is the user's foot size The information processing device according to any one of (1) to (9). (11) The image further comprises an RGB image. The information processing device according to any one of (1) to (10). (12) The sensor is configured as a mobile terminal having the processing unit, the processing unit, and a display unit that displays the processing results by the processing unit. The information processing device according to any one of (1) to (11). (13) The information processing device An image including at least a depth image acquired by a sensor and at least a portion of information obtained from the image are processed using a trained model trained by machine learning, and the size of an object included in the image is measured. Information processing methods. (14) Computer, The system includes a processing unit that processes an image including at least a depth image acquired by a sensor and at least a portion of information obtained from the image using a trained model trained by machine learning, and measures the size of an object included in the image. A program that functions as an information processing device. [Explanation of symbols]

[0195] 1 Information processing device, 2 Information processing device, 3 Imaging device, 4 Database, 11 Depth sensor, 12 Depth processing unit, 13 RGB sensor, 14 RGB processing unit, 15, 15A, 15B, 15C Processing unit, 16 Display unit, 17 Operation unit, 111 Trained model, 112 3D coordinate calculation unit, 113 Foot size posture calculation unit, 114 Trained model, 211 Trained model, 311 Trained model

Claims

1. a processing unit that processes an image including at least the depth image among the depth image and RGB image acquired by the sensor, and at least a part of information obtained from the image, using a trained model trained by machine learning, and measures the size of a part of the user's body included in the image; The trained model includes a first trained model that receives the image as an input and outputs feature points of the user's body parts, and a second trained model that receives the feature points of the user's body parts, the size of the user's body parts, and the posture of the user's body parts as input and outputs corrected sizes of the user's body parts. Information processing device.

2. The second trained model receives user information about the user, including feature points of the user's body parts, sizes of the user's body parts, and postures of the user's body parts, and outputs the corrected sizes. The information processing device according to claim 1 .

3. The processing unit Calculating a size of the user's body part and a posture of the user's body part based on the feature points of the user's body part output from the first trained model; The calculated size of the user's body part and the calculated posture of the user's body part are input into the second trained model. The information processing device according to claim 1 .

4. The first trained model outputs 2D feature points or 3D feature points as the feature points, The second trained model inputs 3D feature points as the feature points. The information processing device according to claim 1 .

5. When the feature points are 2D feature points, the processing unit calculates 3D feature points from the 2D feature points. The information processing device according to claim 4 .

6. a display unit that displays the corrected size The information processing device according to claim 1 .

7. The display unit displays an AR image by superimposing it on a body part of the user included in a captured image of the user. The information processing device according to claim 6 .

8. the part of the user's body is the user's foot, The size of the user's body part is the size of the user's foot. The information processing device according to claim 1 .

9. The sensor, the processing unit, and the display unit are configured as a mobile terminal. The information processing device according to claim 6 .

10. The information processing device The method includes a step of processing an image including at least the depth image among the depth image and RGB image acquired by the sensor, and at least a part of information obtained from the image, using a trained model trained by machine learning, and measuring the size of a part of the user's body included in the image; The trained model includes a first trained model that receives the image as an input and outputs feature points of the user's body parts, and a second trained model that receives the feature points of the user's body parts, the size of the user's body parts, and the posture of the user's body parts as input and outputs corrected sizes of the user's body parts. Information processing methods.

11. Computer, a processing unit that processes an image including at least the depth image among the depth image and RGB image acquired by the sensor, and at least a part of information obtained from the image, using a trained model trained by machine learning, and measures the size of a part of the user's body included in the image; The trained model includes a first trained model that receives the image as an input and outputs feature points of the user's body parts, and a second trained model that receives the feature points of the user's body parts, the size of the user's body parts, and the posture of the user's body parts as input and outputs corrected sizes of the user's body parts. A program that functions as an information processing device.

Images