Weight estimation device and training data creation method

Abstract

To provide a weight estimation device that can appropriately estimate the weight of an object even if the object is not limited to a solid body and even if the distance to the object is not fixed.SOLUTION: A weight estimation device 10 comprises: a machine learning model 22 that has been machine-learned so that it can estimate the weight of an object 12 on the basis of a color image and a depth data image created by an RGB camera 14a and a stereo camera 14b, respectively, for the object 12; and a weight estimation unit 18 that estimates the weight of the object 12 by using the machine learning model 22.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a weight estimation device that estimates the weight of an object in an image, and a method for creating training data used in machine learning for weight estimation. [Background technology]

[0002] Conventionally, there are known devices that can capture an image of an object using an imaging device and estimate its weight or mass. The weight of the object is estimated using a determination device that uses image processing technology or a machine learning model.

[0003] For example, the mass estimation device in Patent Document 1 includes a conveying unit having a conveying surface for conveying an object to be estimated, an imaging unit fixed above the conveying unit, and a calculation unit connected to the imaging unit. When a certain amount of food scattered on the conveying surface is conveyed and stopped, the calculation unit estimates the partial mass of the food on the conveying surface. In particular, the calculation unit estimates the mass of food contained within a predetermined area defined by a position identification marker within the imaging field of view that can be captured by the imaging unit.

[0004] The mass estimation algorithm uses a machine learning-based system that automatically constructs an estimation formula from past data. A convolutional neural network is used for the machine learning. In addition, a position identification marker can be used to identify which area of ​​the conveyance surface the mass was measured in, so by accumulating the partially measured masses, it becomes easy to extract the required amount of the object to be estimated (Patent Document 1 / paragraphs 0023, 0024, 0037, Figures 1 and 2). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-139775 Summary of the Invention [Problem to be solved by the invention]

[0006] However, because Patent Document 1 is a method for estimating weight and other parameters by recognizing the overall shape of an object from each viewpoint while it is moving, it has the problem of being difficult to accurately estimate physical quantities unless the object is a solid object with a solid shape. For example, leafy vegetables such as lettuce are fluffy and unstable in shape, and may have gaps inside. Therefore, even if the two-dimensional shape and size as seen from the imaging unit are the same, the weight may vary significantly.

[0007] Furthermore, the mass estimation device of Patent Document 1 images an object being transported on a transport surface when it reaches a predetermined position, so the distance between the imaging unit and the object at the time of image capture is fixed. Therefore, when estimating the mass of an object for which the distance between the imaging unit and the object is not fixed, the estimation accuracy is low.

[0008] The present invention has been made in consideration of the above circumstances, and aims to provide a weight estimation device and a teacher data creation method that can appropriately estimate the weight of an object, even if the object is not limited to a solid object and the distance to the object is not fixed. [Means for solving the problem]

[0009] The weight estimation device of the present invention comprises: an RGB camera capable of capturing a color image of an object; a stereo camera capable of capturing a depth data image including depth information to the object; a machine learning model that has been trained to estimate the weight of the object based on the color image and the depth information of the object; a weight estimation unit that estimates a weight of the object using the machine learning model when the captured color image and the depth data image are input; It is equipped with:

[0010] The teaching data creation method of the present invention includes: A teacher data creation method for creating machine learning data from a color image of an object captured by an RGB camera and a depth data image including depth information to the object captured by a stereo camera, a first object cutting step of cutting out a first image portion surrounding an outer periphery of the object from the color image; a first image pasting step of pasting the first image portion cut out by the first object cutting step, without processing, into a frame of a predetermined size; a second object extraction step of extracting a second image portion from the depth data image, the second image portion having the same imaging range as the first image portion; a second image pasting step of pasting the second image portion cut out by the second object cutting step, without processing, into a new frame of the predetermined size; It is equipped with: [Effects of the Invention]

[0011] According to the weight estimation device of the present invention, the machine learning model estimates the weight of an object using a depth data image in addition to a color image, which allows for appropriate estimation of the weight of the object even if the distance to the object is not fixed or even if each part of the object is at a different distance from the imaging unit.

[0012] According to the training data creation method of the present invention, in creating machine learning data, first and second image portions that surround the periphery of the object and have the same shooting range are cut out from the color image and depth data image, and pasted into respective frames of the same predetermined size without processing, i.e., without normalizing the size. This makes it possible to create an appropriate machine learning model for weight estimation even for objects whose distances from the RGB camera vary depending on the part. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram of a weight estimation device equipped with a machine learning model to estimate the weight of an object. [Figure 2]This is a diagram showing the layout of a working device that creates color images and depth data images of training data when a machine learning model performs machine learning. [Figure 3] 1A and 1B are diagrams showing various color images used as training data during machine learning of a machine learning model. [Figure 4] FIG. 10 shows paired color and depth data images at multiple coordinate positions of an object. [Figure 5] FIG. 10 shows color images of objects with different weights. [Figure 6] 10A and 10B are explanatory diagrams illustrating a depth data interpolation process for a first depth data image. [Figure 7] 10A and 10B are explanatory diagrams illustrating a depth data interpolation process for a second depth data image. [Figure 8] 10A and 10B are explanatory diagrams illustrating depth data interpolation processing for the third depth data image. [Figure 9] FIG. 10 is a diagram illustrating the process of extracting an image area of ​​an object from a color image to create final training data. [Figure 10] 10A and 10B are diagrams showing the steps of pasting cut-out regions cut out from a color image and a depth data image onto the corresponding canvases. [Figure 11] FIG. 10 is an explanatory diagram of the expansion of training data by increasing or decreasing the depth. [Figure 12] This is a graph that examines the relationship between Epoch and MSE and MAE when a machine learning model is trained. [Figure 13] This is a graph showing the relationship between Epoch and MSE and MAE when tuning a predetermined Epoch in Figure 12 was performed separately from a predetermined number of times and machine learning was performed from the first time. [Figure 14] This is a graph that examines the relationship between Epoch and MSE and MAE when a machine learning model is trained using only RGB out of the pairwise RGB and depth. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described. It goes without saying that the present invention is not limited to the embodiments. Note that the same reference numerals will be used for components common to multiple embodiments.

[0015] (Embodiment) 1 is a schematic diagram of a weight estimation device 10 that estimates the weight of an object 12. The weight estimation device 10 includes a compound camera 14, a weight estimation unit 18, a weight display 20, and a machine learning model 22.

[0016] In this example, the object 12 is a lettuce, a leafy vegetable. The entire color of this lettuce is almost entirely green (G). The fact that the object 12 is green is utilized in creating a mask 60a, which will be described later. The lettuce as the object 12 has gaps inside, is fluffy overall, and has an unstable shape and size compared to solid objects. Estimating the weight of such an object 12 is more difficult than with a solid object. The weight estimation device 10 constructs a machine learning model 22 that can appropriately estimate the weight of an object 12 such as lettuce.

[0017] The compound camera 14 is equipped with an RGB camera 14a and a stereo camera 14b. The compound camera 14 may be, for example, a commercially available RealSense (registered trademark) depth camera from Intel (registered trademark).

[0018] The RGB camera 14a and the stereo camera 14b generate a color image (specifically, composed of three images of red R, green G, and blue B) and a depth data image, respectively. The weight estimation unit 18 detects the coordinate position of each part of the object 12 in the field of view coordinate space 16 of the compound camera 14 based on the color image and the depth data image.

[0019] In the figure, for ease of illustration, the X-axis, Y-axis, and Z-axis in the field of view coordinate space 16 are drawn as Cartesian coordinates. However, in more detail, the coordinate position detected by the compound camera 14 for each part of the subject in the field of view coordinate space 16 is the coordinate position assigned to the angle portion occupied by the photographed part of each pixel in the horizontal direction, the front-back direction (depth direction), and the vertical direction by dividing the viewing angle in the horizontal direction, the front-back direction (depth direction), and the vertical direction as seen from the compound camera 14 by an angle corresponding to one pixel of the RGB camera 14a or the stereo camera 14b.

[0020] The weight estimation unit 18 detects the X coordinate and Z coordinate of each part of the object 12 as a subject from the color image of the object 12, and detects the Y coordinate (depth) of each part of the object 12 as a subject from the depth data image of the object 12. In detail, the color image has data of three colors, R (red), G (green), and B (blue), for each coordinate position corresponding to the pixel of the RGB camera 14a.

[0021] In this example, the pixels of the RGB camera 14a and the pixels of the stereo camera 14b correspond one-to-one. However, the correspondence between the pixels of the RGB camera 14a and the pixels of the stereo camera 14b may be a one-to-multiple or multi-to-one relationship. When the correspondence between the pixels of the RGB camera 14a and the pixels of the stereo camera 14b is one-to-multiple, the depth (distance from the compound camera 14) of an area in the color image corresponding to each pixel of the RGB camera 14a is determined to be one of the depths of multiple corresponding pixels in the stereo camera 14b (e.g., minimum depth, average depth, maximum depth, or any depth). When the correspondence between the pixels of the RGB camera 14a and the pixels of the stereo camera 14b is multi-to-one, the depth of an area in the color image corresponding to multiple pixels of the RGB camera 14a is commonly determined to be the depth of one pixel in the stereo camera 14b corresponding to the multiple pixels.

[0022] The machine learning model 22 is constructed by machine learning based on color images and depth data images prepared in advance as training data, and is incorporated into the weight estimation device 10 as a trained AI model.

[0023] The weight estimation unit 18 uses a machine learning model 22 to estimate the weight of the object 12 based on a color image and a depth data image of the object 12 whose weight is to be estimated.

[0024] The weight display device 20 has an object display section 20a and a weight display section 20b. The object display section 20a displays an image of the object 12 cut out from the color image generated by the RGB camera 14a. The weight display section 20b displays the estimated weight of the object 12 displayed on the object display section 20a.

[0025] (Teacher data) 2 is a layout diagram of a working device that creates color images and depth data images of training data (an example of data for machine learning) when machine learning is performed by the machine learning model 22. In order to create color images and depth data images of training data to be used for machine learning in the machine learning model 22, it is necessary to prepare a sufficient number of color images and depth data images of the object 48 whose parameter values ​​of relative position, size, shape, and weight in the field of view coordinate space 16 are appropriately distributed within expected ranges for the object 12 whose weight is to be estimated by the weight estimation device 10.

[0026] Detecting the relative position of the object 12 in the field of view coordinate space 16 is not essential because the object 12 is not extracted from the color image and the depth data image based on its relative position, but is extracted based on a determination of which image portion of the object 12 is located in the color image and the depth data image. However, the "appearance variations" obtained by capturing images of the object 12 at different relative positions from the compound camera 14 are useful as training data.

[0027] The object 12 in Figure 1 is an object whose weight is actually estimated by the weight estimation device 10. In contrast, the object 48 in Figure 2 is an object for creating training data when machine learning is performed by the machine learning model 22. The object 48 is of the same type as the object 12 (e.g., lettuce), and has parameters (e.g., shape, size, weight, and coordinate position) that are widely distributed within the range of values ​​expected for the object 12 whose weight is to be estimated, and is prepared for machine learning.

[0028] The X-axis, Y-axis, and Z-axis in Fig. 2 correspond to the X-axis, Y-axis, and Z-axis of the field of view coordinate space 16 in Fig. 1. In Fig. 1, the compound camera 14 is fixed in position, and the object 12 has various relative positions with respect to the compound camera 14. In contrast, in Fig. 2, the position of the object 48 is fixed, and the compound camera 14 is able to change position with respect to the object 48 in a fixed position.

[0029] The camera stand 30 comprises a base 32 placed on the floor 28 so as to be movable in the X-axis and Y-axis directions, a strut 34 erected on the base 32, a vertical pipe 38 whose protrusion from the strut 34 is adjustable in the Z-axis direction, a horizontal pipe 40 fixed to the upper end of the vertical pipe 38, and the horizontal pipe 40 supported by a vertical head 36 so as to be movable in the Y-axis direction. 24 is attached to the tip of the horizontal pipe 40 with the lens facing downward.

[0030] A rectangular nonwoven fabric 44 is placed on the floor 28 below the compound camera 14. A piece of construction paper 46 has the same shape and size as the nonwoven fabric 44 and is laid on top of the nonwoven fabric 44. An object 48 is placed at the center of the top surface of the construction paper 46.

[0031] The light stand 50 has a lamp 52 facing downward at its upper end. The light stand 50 is placed on the floor 28 outside the construction paper 46 so that the lamp 52 is directly above the center of the construction paper 46. The lamp 52 irradiates light toward the construction paper 46 so that the entire construction paper 46 is illuminated with a substantially uniform brightness.

[0032] To create color images and depth data images of the training data, the operator places the object 48 at the center of the construction paper 46, moves the base 32 of the camera stand 30 in the X-axis and Y-axis directions, and displaces the vertical head 36 in the Z-axis direction. The orientation of the compound camera 14 is maintained directly downward. In this way, the relative position between the compound camera 14 and the object 48 changes in various ways. The operator operates the compound camera 14 at multiple relative positions between the compound camera 14 and the object 48, and creates paired color images and depth data images of the same object 48 at different relative positions as training data.

[0033] After creating training data for one object 48, the worker changes the object 48 to another object 48 with different parameters (e.g., shape, dimensions, weight) or their values, and creates training data in the same way. The training data created in this way is used for machine learning in the machine learning model 22.

[0034] (Color image / position for training data) Fig. 3 is a diagram showing various color images used as training data during machine learning of the machine learning model 22. Note that although each color image shown in Fig. 3 is a single image, the color image is actually composed of three images: red R, green G, and blue B. For the sake of simplicity, the drawing shows the three images of red R, green G, and blue B as one image.

[0035] For the object 48 to be used as training data for generating a learning model for estimating the weight of the object 12, it is preferable to collect a sufficient number of captured images (color images and depth data images) as training data, with weight as the essential parameter and a wide range of values ​​for each parameter, including multiple other parameters (e.g., shape and size).

[0036] Then, for each object 48, a color image and a depth data image of the training data are created at various coordinate positions within the visual coordinate space 58. In the drawings, the color images are shown in grayscale.

[0037] In FIG. 3, the field of view coordinate space 58 is defined for the compound camera 14 in FIG. 2, but since the compound camera 14 is common to the compound camera 14 of the weight estimation device 10 in FIG. 1, it is defined to be the same as the field of view coordinate space 16 in FIG. 1.

[0038] The X-axis, Y-axis, and Z-axis of the visual field coordinate space 58 in Figure 3 correspond to the X-axis, Y-axis, and Z-axis in Figure 2. The visual field coordinate space 58 is equally divided into three parts in the X-axis, Y-axis, and Z-axis directions. As a result, the visual field coordinate space 58 is divided into a total of 27 cell spaces, 3 x 3 x 3. To distinguish between the cell spaces, subscripts a, b, and c are assigned to each cell space in the X-axis, Y-axis, and Z-axis directions, starting from the cell space closest to the origin O. The coordinates of each cell space correspond to the coordinates of the center of the cell space.

[0039] When a portion corresponding to each pixel of the compound camera 14 is defined for the surface of the object 48 facing the compound camera 14, an X coordinate, a Y coordinate, and a Z coordinate are defined for each portion. The center coordinates of each cell space in FIG. 3 are used simply to indicate the position of the object 48 in the photograph of the color image and the depth data image in the drawing, and in actual machine learning and weight estimation, the coordinates of each pixel as a portion on the surface of the object 12 or the object 48 are determined.

[0040] In the visual coordinate space 58, the coordinates of the cell space closest to the origin O are represented by (Xa, Ya, Za). The coordinates of the cell space farthest from the origin O are represented by (Xc, Yc, Zc). The coordinates of the cell space at the center of the visual coordinate space 58 are represented by (Xb, Yb, Zb).

[0041] 3, the compound camera 14 is placed on the (Xb, Ya, Zb) side of the field of view coordinate space 58 in the Y-axis direction, and the optical axis of the lens of the compound camera 14 is parallel to the Y-axis and points toward (Xb, Ya, Zb). Ya, Yb, and Yc are the distances on the optical axis from the compound camera 14, and are Y coordinates that are, for example, 42 cm, 47 cm, and 57 cm.

[0042] 3 is the same object 48, the size of the object 48 in the color image becomes larger when the object 48 is placed in the cell space closer to the compound camera 14. Also, the position of the object 48 in each color image moves to the right, left, up or down with respect to the center of the cell space (Xb, Yb, Zb) of the field of view coordinate space 58 as seen from the compound camera 14, depending on whether the object 48 is to the right, left, up or down.

[0043] (Pairwise color and depth data images) 4 shows paired color images and depth data images at multiple coordinate positions of an object 48. Paired color images and depth data images refer to color images and depth data images captured simultaneously by the RGB camera 14a and the stereo camera 14b at the same coordinate position of the same object 48. In FIG. 4, color images and depth data images at the same vertical level are in a paired relationship.

[0044] The color image is actually in color, but is shown in grayscale in Figure 4. The depth data image is generated as a grayscale image without color in the illustration. The depth (distance) from the compound camera 14 to the object 48 is represented by the degree of gray of each part in the illustrated depth data image.

[0045] In each depth data image in Figure 4, the depth-unacquirable area is black. The depth-unacquirable area is likely to occur in the rear leaves that are in the shadow of the front leaves in an object 48 such as leafy vegetables. Depth data images with depth-unacquirable areas are not suitable as training data, so it is preferable to interpolate them using appropriate depth data.

[0046] (Color image / weight for training data) 5 is a diagram showing color images of objects 48 of different weights. While maintaining the object 48 in a predetermined cell space in the visual field coordinate space 58, the worker tears off the leaves of the object 48 (e.g., lettuce) little by little to gradually reduce the weight, and create a total of four objects 48 of different weights (depth data images are omitted from FIG. 5).

[0047] In this example, the initial weight of the object 48 is 10.5 g, and then changes to 7.5 g, 5.5 g, and 3.9 g. The object 48 changes not only in weight but also in size and shape. By arranging these four objects 48 with different weights in a total of nine cell spaces of the field of view coordinate space 58, a total of 36 (= 9 × 4) paired color images and depth data images can be added as training data. The nine fields of the field of view coordinate space 58 mean that the compound camera 14's shooting positions relative to the object 48 are three in the Y-axis direction (depth direction)—42 cm, 47 cm, and 52 cm—and three in the X-axis direction (horizontal direction)—left, center, and right—leading to 3 × 3 = 9 combinations of relative positions.

[0048] (depth data interpolation) 6-8 are explanatory diagrams of the depth data interpolation process for different depth data images. As mentioned above, areas in a depth data image where depth data could not be acquired are displayed in black. A depth data image that remains black reduces the value of the training data. Depth data interpolation addresses this issue.

[0049] A feature of the depth data interpolation employed in the embodiment is the use of a mask derived from a color image. Specifically, for a paired color image and depth data image, a mask of the object 48 is created based on the color image, and the depth data image is divided into an object area (object area) within the mask and an object area (background area) outside the mask based on the created mask. Then, missing values ​​(depths of depth-unacquirable areas) in the object area and object area are interpolated based only on the depths (depth values) of adjacent depth acquisition areas only in the object area and object area, respectively.

[0050] In Figures 6 to 8, areas of interest (e.g., areas where interpolation processing is to be performed or has been performed) are circled. Also, in Figures 6 and subsequent figures, "RGB" means a color image, and "depth" means a depth data image.

[0051] In Figures 6 to 8, the images are arranged in three rows vertically. Of the three rows, the image in the first row from the top is a color image, and the images in the second and third rows are depth data images that are paired with the color image in the first row. The depth data image in the second row is the depth data image output directly from the image captured by the stereo camera 14b, i.e., before depth interpolation processing. The two images in the third row are depth data images after depth interpolation has been performed on the depth data image in the second row. The depth data images on the left and right are the result data obtained by performing the interpolation processing of distant view extension interpolation and object interior interpolation, respectively.

[0052] In Fig. 6, a depth acquisition unavailable area occurs on the left side of the object 48 in the color image. In Fig. 7, a depth acquisition unavailable area occurs in the rear leaf portion of the object 48 that is hidden by the front leaf.

[0053] In FIG. 8, the object 48 is placed inside a transparent glass cup and photographed from the open side of the glass cup. Light is reflected off the glass cup, and the area where incident light and reflected light coexist is the depth acquisition impossible area. Note that in FIG. 8, the object 48 is placed inside the glass cup because the object 48 is small and would fall over, so the glass cup was used to photograph the object 48 in an upright position. Even if the object 48 is photographed inside the glass cup, the depth acquisition impossible area can be interpolated.

[0054] In the distant view extended interpolation (the left image of the two images in the bottom row of Figures 6 to 8), the black area of ​​the depth acquisition impossible area (the image before interpolation in the second row from the top) is changed to a shade equal to the shade of the depth area of ​​the depth acquisition area adjacent to the black area. In Figures 6 and 7, the dashed circle is a mark added to draw attention to the depth acquisition impossible area.

[0055] In Fig. 8, an object 48 is placed in a cup. In each image in Fig. 8, two objects 48 are included: a large one on the left and a small one on the right. This is because the object 48 is actually located on the right, but is small and difficult to see, so it has been enlarged to make it easier to see as the object 48 on the left.

[0056] In background extension interpolation, each black area is surrounded by multiple depth acquisition areas with different depths. In background interpolation, each depth acquisition unavailable area is changed by interpolation to the depth of one of the multiple depth acquisition areas with different depths that surround it. For example, each depth acquisition unavailable area may be interpolated to the maximum depth, minimum depth, or average depth of the multiple depth acquisition areas with different depths that surround it. Furthermore, the depth of each position within each depth acquisition unavailable area may be interpolated to the depth of the adjacent depth acquisition area closest to that position.

[0057] In the object interior interpolation (the right image of the two images in the bottom row of Figures 6 to 8), a mask is used. The mask uses the outline of the object 48 in the color image as a boundary line, and the area inside the boundary line is the interpolation processing area. In this embodiment, lettuce is selected as an example of the object 48. Since lettuce is green, when creating a mask, for example, pixels where the proportion of G in the R:G:B ratio on a pixel-by-pixel basis is equal to or greater than a certain value are determined to be pixels where the object 48 exists, and the area of ​​those pixels is used as the mask. Alternatively, a mask can be created using a model (such as semantic segmentation) that detects object areas in an input image and outputs a label for the object detected for each pixel.

[0058] In a specific example of object interior interpolation, (a) areas with a value of zero (areas where depth cannot be acquired) are filled in by dilation processing using a square filter of a specified size on the entire depth data image. Also, (b) pixels that originally contained values ​​are overwritten with the values ​​before the dilation processing so that they do not change due to the dilation processing. Then, by repeating (a) and (b) above, all areas where depth cannot be acquired are interpolated. For areas where depth cannot be acquired, it is possible to choose between "the value changes due to the influence of the dilation processing with each repetition" or "once a non-zero value is entered, the value is fixed in the same way as areas where depth can be acquired." Which method is more appropriate depends on the situation and task.

[0059] (How to create training data) 9 is a diagram illustrating the process of extracting an image region of the object 48 from a color image to create final training data. A mask 60a is used to determine an extraction region 62a in the color image (the top image in FIG. 9). The extraction region 62a is defined as a rectangular region obtained by adding a margin of 20 pixels to the outside of each side of the smallest rectangle that contains the mask 60a.

[0060] The cropped area 62a in STEP 1, which is the second from the top in Figure 9, is an enlarged version of the cropped area 62a in the first from the top in Figure 9. In STEP 2, the cropped area 62a is cropped from the color image and pasted at the center of the canvas 64a. The canvas 64 has a pixel count of 480 x 480. The closer the object 48 in the color image is to the coordinate position of the object 48 photographed, the larger the size of the color image and the larger the size of the cropped area 62a. The 480 x 480 size of the canvas 64 is set as a size that ensures that the cropped area 62a of the largest possible size can be pasted inside.

[0061] FIG. 10 shows STEPs 2 and 4, in which cropped regions 62a and 62b, respectively, cropped from the color image and depth data image, are pasted onto corresponding canvases 64a and 64b. The color image contains images for three colors, R, G, and B, so the amount of information expressed in terms of length, width, and number of images is (480, 480, 3). The amount of information in the depth data image is expressed as (480, 480, 1). Therefore, one set of (480, 480, 4) constitutes one training data set.

[0062] STEPs 1, 2, and 4 correspond to the first object cutting step, the first image pasting step, and the second image pasting step of the present invention, respectively. The second object cutting step of the present invention is not shown, but is simply a case in which the cutout area 62a in STEP 1 of Fig. 8 is replaced with a cutout area for the depth data image. Furthermore, the mask used for the depth data image is a mask 60a created by extracting the contour as an edge through differential processing of a color image of a predetermined color (G in this embodiment).

[0063] (Expanding training data) FIG. 11 is an explanatory diagram of training data expansion (data augmentation) by increasing or decreasing depth (increasing or decreasing distance). Data augmentation is a method of padding data by using data obtained by applying transformations to the original training data as training data. Generally, data augmentation methods include (a) a method of performing training using a data set with an increased amount of data by adding data obtained by applying transformations to the original training data in advance to the training data, and (b) a method of using a generator that applies random transformations to the training data.

[0064] In (b), the original number of training data items is not changed, but instead a random transformation is applied to the original data for each training epoch. This allows different data generated from the original data for each epoch to be provided to the model. Common examples of generator usage include horizontal flipping, where there is a 50% chance of whether or not to apply the process, and random shifting, where the entire image is translated in the vertical and horizontal directions by a range of -50 to +50 pixels. These are examples of binary selections or numerical ranges that are set as parameters. In this case, different random values ​​are applied to each piece of data within the same epoch.

[0065] In this embodiment, as a special example of data augmentation using the generator (b), the depth is increased or decreased by a setting value (random distance) randomly generated from the range of -100 to +100 mm of the original learning data. Then, for the existing color image (original color image) and depth data image (original depth data image) in a paired relationship (the image without distance correction in the center in the horizontal direction in FIG. 11), the distances of all pixels of the original depth data image are uniformly changed by the random distance, and new color image and new depth data image data (the data augmented images at the left and right ends in the horizontal direction in FIG. 11) are generated as augmented data by scaling the original color image and original depth data image based on the changed distance.

[0066] 11, "RGB" and "depth" refer to a color image and a depth data image, respectively. In addition, in the three columns of RGB and depth pairs in the left-right direction of FIG. 11, the center column (original) is the original image (an image generated from actual photography by the compound camera 14). The images in the column (-100 mm) to the left of the center column and the column (+100 mm) to the right of the center column are newly created extended images based on the assumed size and shape of the object 48 when the compound camera 14 is moved 100 mm closer to and farther from the original image.

[0067] In expanding the training data by increasing or decreasing the depth, the expanded dimensions (the dimensions of the object 48 in the image that is the expansion of the training data) are changed to a value obtained by multiplying the ratio Lb / La of the expanded distance Lb (the distance between the compound camera 14 and the object 48 in the expanded image) to the original distance La (the distance between the compound camera 14 and the object 48 in the original image) by the original dimensions of the object 48 (the dimensions of the object 48 in the original image). Therefore, the center coordinates of the original image and the expanded image are the same, and the shapes are similar.

[0068] The depth of the object 48 resulting from the depth increase or decrease is written into each depth data image, which is therefore distinct from the well-known expansion of training data by simply enlarging or reducing the object 48 without changing the depth.

[0069] In this example, the training data is expanded by increasing or decreasing the depth only. However, the training data expansion by increasing or decreasing the depth can also be performed in combination with one or more of the well-known training data expansion techniques of image rotation, horizontal translation, vertical translation, horizontal flipping, vertical flipping, and brightness adjustment.

[0070] (Points to note when expanding data) When applying well-known data augmentation techniques, it is important to note that the target of the transformation must be appropriately selected depending on the characteristics of the method. For example, data augmentation that focuses on the luminance of color images, such as brightness adjustment, only applies the transformation to the color image, and it is inappropriate to include the depth data image in the transformation target. On the other hand, data augmentation that changes the concept of position, such as left-right flipping, random rotation, and translation, requires similar processing to be performed on both the color image and the depth data image. Although not used in this study, when using data augmentation that converts part of the data into random noise, such as random erasing, applying random values ​​independently to the color image and the depth data image makes it possible to estimate noise areas from each other's information, which may improve inference accuracy.

[0071] (verification) Figure 12 is a graph examining the relationship between epochs (number of epochs) and MSE (mean squared error, set as the loss function) and MAE (mean absolute error, equivalent to the error in grams of weight) when machine learning model 22 was trained. As mentioned above, the data used for training this time was obtained from a single plant by changing the photography method, removing leaves, etc. When training the model, we made sure that data from the same plant was not mixed between the training data and validation data in order to properly confirm generalization performance.

[0072] In this machine learning, as shown below, training was performed in two steps (steps R1 and R2), with 1000 epochs as one step. RMSprop was used as the optimizer in both steps R1 and R2, but the learning rate was set to 0.001 in step R1 and 0.1 times that of step R1, or 0.0001, in step R2. Training was performed for 1000 epochs in step R1, and the weights at the point in time when the loss value (MSE) for the validation data was minimized were saved. In step R2, the weight parameters saved in step R1 were set as the initial values ​​for the model, the optimizer settings were initialized, and retraining was performed for 1000 epochs. The model equipped with the weight parameters at the point in time when the loss value for the validation data was minimized in step R2 was used as the final model for evaluation.

[0073] 12 to 14, the solid and dashed lines show the characteristics for the training data and the validation data, respectively. As the epoch progresses, both the MSE and MAE of the validation data exponentially decrease, indicating that the weight estimation device 10 can properly train the machine learning model 22. In the machine learning of FIG. 12, both pairwise RGB (color image) and depth (depth data image) are used as training data.

[0074] Figure 13 is a graph that examines the relationship between Epoch and MSE and MAE when machine learning is performed separately from Epoch=1 using the tuning of Epoch=735 in Figure 12 as the initial value. Note that the scale intervals on the vertical axis in Figure 13 are wider than those in Figure 12. It can be seen that the learning effect is higher in Figure 13 than in Figure 12. Note that Epoch=735 was selected because the model with 735 epochs had the smallest loss (1.77) on the validation data in step R1.

[0075] Fig. 14 is a graph showing the relationship between Epoch and MSE and MAE when machine learning model 22 is trained using only RGB among the pairwise RGB and depth. Specifically, the same verification test as in Fig. 12 was conducted.

[0076] To add a bit more about Figure 14, training in step R2 (from epoch 1001 onwards) is actually performed using the weight parameters at epoch 822, which had the smallest loss on the validation data out of the 1000 epochs of step R1. Therefore, although Figure 14 makes it appear as if epochs 1000 -> 1001 are performed consecutively, in reality, training restarts with the weight parameters from epoch 822 from epoch 1001 onwards. For validation using only RGB images, steps R1 and R2 were performed consecutively, resulting in continuous recording, which causes the graphs to be merged.

[0077] Comparing Figure 14 with Figure 12, for RGB+depth in Figure 12, the loss was 0.79 from 735 epochs (step R1) to +677 epochs (step R2). Also, for RGB only in Figure 14, the loss was 1.47 from 822 epochs (step R1) to +107 epochs (step R2). From the above, comparing the loss on the validation data, RGB+depth is much lower, which shows that the depth information contributes to improving generalization performance.

[0078] (Variation) The weight estimation device 10 estimates the weight of only one specific type of object 12 (e.g., lettuce). However, the weight estimation device of the present invention can estimate the weight of multiple types of objects as long as they are clearly distinguishable by color (e.g., green lettuce, orange mandarin oranges, purple eggplants, yellow bananas, and red apples).

[0079] The weight estimation device 10 estimates the weight of only one specific type of object 12 (e.g., lettuce). However, the weight estimation device 10 can also be used to estimate the weight of multiple types of object 12 (e.g., lettuce, broccoli, and Chinese cabbage) by providing multiple machine learning models 22 trained by machine learning for each type and allowing a person to manually switch between the machine learning models 22 corresponding to each type of object 12. Of course, a single machine learning model can also be used to estimate the weight of multiple types of objects by training the single machine learning model on multiple types of objects.

[0080] In the weight estimation device 10, background extension interpolation and object interior interpolation (FIGS. 6-8) are performed as interpolation of depth data images used as training data during machine learning of the machine learning model 22. In the weight estimation device of the present invention, background extension interpolation and object interior interpolation can also be performed on the depth data image of the object whose weight is to be estimated (e.g., object 12 of the weight estimation device 10) during actual weight estimation.

[0081] The weight estimation device may perform interpolation only during actual weight estimation, not during learning. Alternatively, the weight estimation device may perform interpolation on the depth data image both as training data during color image learning and during actual weight estimation. [Explanation of symbols]

[0082] 10···Weight estimation device, 12,48···Object, 14a···RGB camera, 14b···Stereo camera, 16···View coordinate space, 18···Weight estimation unit, 22···Machine learning model, 60a···Mask, 62a···Cutout region, 64a···Canvas.

Claims

1. an RGB camera capable of capturing a color image of a target object that is a leafy vegetable; a stereo camera capable of capturing a depth data image including depth information to the object; a machine learning model that has been trained to estimate the weight of the object based on the color image and the depth information of the object; When the captured color image and the depth data image are input, a weight estimation unit that estimates the weight of the object using a model; Equipped with The machine learning of the machine learning model of this weight estimation device divides the depth data image into an area inside the object and an area outside the object based on a mask created from a color image that is paired with the depth data image, and uses as machine learning data a depth data image interpolated for the area inside the object where depth acquisition is not possible based on the depth of the depth acquisition area adjacent to the area inside the object where depth acquisition is not possible.

2. The machine learning model is a random distance setting unit that sets the randomly generated increase or decrease distance as a random distance; an extended data generating unit that treats an existing paired color image and depth data image as an original color image and an original depth data image, respectively, and uniformly changes the depth of all pixels of the original depth data image by an amount corresponding to the random distance, and generates data corresponding to a new color image and a new depth data image as extended data by scaling the original color image and the original depth data image to a size corresponding to the changed depth; The weight estimation device according to claim 1 , comprising:

3. an RGB camera capable of capturing a color image of a target object that is a leafy vegetable; a stereo camera capable of capturing a depth data image including depth information to the object; a machine learning model that has been trained to estimate the weight of the object based on the color image and the depth information of the object; When the captured color image and the depth data image are input, a weight estimation unit that estimates the weight of the object using a model; Equipped with The machine learning model is a random distance setting unit that sets the randomly generated increase or decrease distance as a random distance; an extended data generating unit that treats an existing paired color image and depth data image as an original color image and an original depth data image, respectively, and uniformly changes the depth of all pixels of the original depth data image by an amount corresponding to the random distance, and generates data corresponding to a new color image and a new depth data image as extended data by scaling the original color image and the original depth data image to a size corresponding to the changed depth; A weight estimation device comprising:

4. 4. A method for creating teacher data for creating machine learning data when machine learning is performed on the machine learning model of the weight estimation device according to claim 1, comprising: a first object cutting step of cutting out a first image portion surrounding an outer periphery of the object from the color image; a first image pasting step of pasting the first image portion cut out by the first object cutting step, without processing, into a frame of a predetermined size; a second object extraction step of extracting a second image portion from the depth data image, the second image portion having the same imaging range as the first image portion; a second image pasting step of pasting the second image portion cut out by the second object cutting step, without processing, into a new frame of the predetermined size; A teacher data creation method comprising:

5. The training data creation method of claim 4, wherein the machine learning data includes data corresponding to new color images and new depth data images obtained by uniformly increasing or decreasing the depth of all pixels of a paired existing color image and depth data image by an amount corresponding to a randomly generated random distance, and scaling the original color image and original depth data image to a size corresponding to the changed depth, as extended data.

Images