Data acquisition device, data acquisition method, and data acquisition station

Abstract

The data acquisition device is provided with a control unit which is configured so as to be able to control the light-emitting panel, and is configured so as to be able to acquire at least one captured image obtained by capturing an image of the light-emitting surface of the light-emitting panel. The control unit generates mask data of the object on the basis of a captured image, among the at least one captured image, obtained by capturing the light-emitting panel and the object located in front of the light-emitting panel in a state in which the light-emitting panel is caused to emit light.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority from Japanese Patent Application No. 2022-88690 (filed on May 31, 2022), the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to a data acquisition device, a data acquisition method, and a data acquisition station. Background Art

[0004] Conventionally, there is known a system that generates learning data for learning in semantic segmentation or the like (see, for example, Patent Document 1).

[0005] Prior Art Literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2020-102041 Summary of the invention

[0008] A data acquisition device according to one embodiment of the present disclosure includes a control unit configured to control a light emitting panel and to acquire a photographic image obtained by photographing the light emitting panel and an object located in front of the light emitting panel. The control unit causes the light emitting panel to emit light and generates mask data of the object based on the photographic image.

[0009] A data acquisition method according to an embodiment of the present disclosure includes: causing a light emitting panel to emit light; and generating mask data of the object based on a photographic image obtained by photographing the object located in front of the light emitting panel and the light emitting panel.

[0010] A data acquisition station according to one embodiment of the present disclosure includes: a light emitting panel that emits light in a predetermined color; and a light transmissive member located between an object disposed in front of the light emitting panel and the light emitting panel. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a block diagram showing a configuration example of a data acquisition system according to one embodiment.

[0012] Figure 2 It is a top view showing a configuration example of a data acquisition system.

[0013] Figure 3 yes Figure 2 AA section view.

[0014] Figure 4AThis is a diagram showing an example of the brightness of each pixel in a photographic image of an object.

[0015] Figure 4B Is based on Figure 4A Figure 3 is a diagram showing an example of a mask image generated from a photographic image.

[0016] Figure 5 This is a plan view showing an example of an object located on the light emitting panel.

[0017] Fig. 6A This is a diagram showing an example of a photographic image of a light emitting panel in a light emitting state.

[0018] Figure 6B This is a diagram showing an example of a photographic image of an object located above a light emitting panel in a light emitting state.

[0019] Figure 6C Is based on Fig. 6A Photographic images and Figure 6B A diagram showing an example of a mask image generated by the difference of photographic images.

[0020] Fig. 7A This is a diagram showing an example of a photographic image of an object located on a light emitting panel in a turned-off state.

[0021] Figure 7B It means Figure 6C FIG. 1 is a diagram showing an example of the same mask image.

[0022] Figure 7C It means in Fig. 7A Photographic image applications Figure 7B A diagram showing an example of an extracted image in which an image of an object is extracted using a mask image.

[0023] Figure 8 It means that Figure 7C An example of training data generated by overlapping an extracted image with a background image.

[0024] Fig. 9 This is a flowchart showing an example of the procedure of the data acquisition method.

[0025] Fig.10 It is a plan view showing an example of an object having side surfaces located on the light emitting panel.

[0026] Fig.11A This is a plan view showing an example in which the color of light emitted by the light emitting panel is the same as the color of the side surface of the object.

[0027] Fig. 11B This is a plan view showing an example in which the color of light emitted by the light emitting panel is different from the color of the side surface of the object.

[0028] Fig. 12A This is a diagram showing an example of a mask image generated when the color of light emitted by the light emitting panel is the same as the color of the side surface of the object.

[0029] Fig. 12B This is a diagram showing an example of a mask image generated when the color of light emitted by the light emitting panel is different from the color of the side surface of the object.

[0030] Fig. 12C It means that by calculating Fig. 12A Each pixel and Fig. 12B FIG. 1 is a diagram showing an example of a mask image generated by taking the logical sum of each pixel.

[0031] Fig.13 This is a flowchart showing an example of a procedure of a data acquisition method including a step of causing a light-emitting panel to emit light in at least two colors.

[0032] Fig.14 This is a schematic diagram showing a configuration example of a robot control system. DETAILED DESCRIPTION

[0033] (Configuration Example of Data Acquisition System 1)

[0034] A data acquisition system 1 involved in one embodiment of the present disclosure acquires training data for generating a learned model, which outputs a recognition result of the recognition object included in the input information. The learned model can be constructed to include a CNN (Convolution Neural Network) having multiple layers. For the information input to the learned model, convolution based on a given weighting coefficient is performed in each layer of the CNN. During the learning of the learned model, the weighting coefficient is updated. The learned model can be constructed to include a fully combined layer. The learned model can also be composed of VGG16 or ResNet50. The learned model can also be constructed as a transformer. The learned model is not limited to these examples, and can also be constructed as various other models.

[0035] like Figure 1 , Figure 2 as well as Figure 3As shown, a data acquisition system 1 according to an embodiment of the present disclosure includes a data acquisition device 10, a light emitting panel 20, and a photographing device 30. The light emitting panel 20 has a light emitting surface, and is configured to be able to place an object 50 for acquiring training data on the light emitting surface. The photographing device 30 is configured to photograph the object 50 placed on the light emitting panel 20 and the light emitting panel 20. The photographing device 30 may also photograph the light emitting panel 20 when no object 50 is placed on the light emitting panel 20. The data acquisition device 10 controls the light emitting state of the light emitting panel 20. The data acquisition device 10 acquires an image obtained by photographing the object 50 from the photographing device 30. An image of the light emitting panel 20 and the object 50, or an image of the light emitting panel 20 is also referred to as a photographed image. The data acquisition device 10 is configured to be able to acquire a photographed image. The data acquisition device 10 can, for example, generate data that can identify the object 50 based on the photographed image. The data acquisition device 10 can, for example, generate training data of the object 50 based on the photographed image and acquire the training data.

[0036] <Data acquisition device 10>

[0037] The data acquisition device 10 includes a control unit 12 , a storage unit 14 , and an interface 16 .

[0038] The control unit 12 is configured to be able to control the light-emitting panel 20, and is configured to be able to obtain at least one photographic image of the light-emitting surface of the light-emitting panel 20. In order to provide control and processing capabilities for performing various functions, the control unit 12 can be configured to include at least one processor. The processor can execute programs that implement various functions of the control unit 12. The processor can be implemented as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor can be implemented as an integrated circuit and a discrete circuit that can be connected in a communicative manner. The processor can be implemented based on various other known technologies.

[0039] The storage unit 14 may include an electromagnetic storage medium such as a magnetic disk, or may include a memory such as a semiconductor memory or a magnetic memory. The storage unit 14 stores various information. The storage unit 14 stores programs executed by the control unit 12, etc. The storage unit 14 may also be configured as a non-temporary readable medium. The storage unit 14 may function as a working memory of the control unit 12. At least a part of the storage unit 14 may also be configured separately from the control unit 12.

[0040] The interface 16 is configured to mutually input and output information or data with the light emitting panel 20 and the photographing device 30. The interface 16 may be configured to include a communication device configured to communicate by wire or wirelessly. The communication device may be configured to communicate in a communication manner based on various communication standards. The interface 16 may be configured using known communication technologies.

[0041] The interface 16 may be configured to include a display device. The display device may include various displays such as a liquid crystal display. The interface 16 may be configured to include a sound output device such as a speaker. The interface 16 is not limited to these and may also be configured to include various other output devices.

[0042] The interface 16 may be configured to include an input device for receiving input from a user. The input device may include, for example, a keyboard or physical keys, or may include a touch panel or a touch sensor or a pointing device such as a mouse. The input device is not limited to these examples, and may also be configured to include various other devices.

[0043] <Light-emitting panel 20>

[0044] The light-emitting panel 20 has a light-emitting surface. The light-emitting panel 20 can be configured as a diffusion plate that disperses the light emitted from the light source and emits it in a planar shape. The light-emitting panel 20 can be configured as a panel that emits light by itself. The light-emitting panel 20 can be configured to emit one of given colors. For example, the light-emitting panel 20 can be configured to emit a single color such as white as the light-emitting color. The light-emitting panel 20 is not limited to white, and can also be configured to emit light in various colors. The light-emitting panel 20 can be configured to emit light in a given color. The light-emitting panel 20 can be configured to emit light in at least two colors as the light-emitting color. For example, the light-emitting panel 20 can be configured to control the spectrum of the light-emitting color by combining the brightness values ​​of each color of RGB (Red Green Blue).

[0045] The light emitting panel 20 may also have a plurality of pixels. The light emitting panel 20 may be configured to control the state of each pixel to be a lit state or an unlit state. The light emitting panel 20 may be configured to control the color of light emitted by each pixel. The light emitting panel 20 may be configured to control the light emission color or light emission pattern of the light emitting panel 20 as a whole by the state of each pixel or the combination of light emission colors.

[0046] <Photographic Device 30 and Illuminating Device 40>

[0047] The photographing device 30 may be configured to include various imaging elements or cameras. The photographing device 30 is configured to be able to photograph the light-emitting surface of the light-emitting panel 20 or the object 50 placed on the light-emitting surface. In other words, the photographing device 30 is configured to be able to photograph the object 50 located in front of the light-emitting panel 20 as viewed from the photographing device 30 together with the light-emitting panel 20. The photographing device 30 may also be configured to photograph the light-emitting surface of the light-emitting panel 20 from various directions. The photographing device 30 may be configured so that the normal direction of the light-emitting surface of the light-emitting panel 20 coincides with the optical axis of the photographing device 30.

[0048] The data acquisition system 1 may further include a dark room for storing the light emitting panel 20 and the photographing device 30. When the light emitting panel 20 and the photographing device 30 are stored in the dark room, the side of the object 50 facing the photographing device 30 is not irradiated by ambient light. When the side of the object 50 facing the photographing device 30 is not irradiated by ambient light, the photographing device 30 photographs the object 50 using the light emitted from the light emitting panel 20 as a background, thereby acquiring an image of the outline of the object 50 as a photographed image.

[0049] Although not essential, the data acquisition system 1 further includes a lighting device 40. Figure 3 As shown, the lighting device 40 is configured to emit an illumination light 42 for illuminating the object 50. The lighting device 40 may be configured to emit the illumination light 42 as light of various colors. When the data acquisition system 1 includes the lighting device 40, the photographing device 30 may photograph the object 50 in a state where the object 50 is illuminated by the illumination light 42 and the ambient light. When the data acquisition system 1 includes the lighting device 40 and a dark room, the photographing device 30 may photograph the object 50 in a state where the object 50 is illuminated by the illumination light 42.

[0050] When the data acquisition system 1 does not include the lighting device 40 , the imaging device 30 can image the object 50 in a state where the object 50 is illuminated by ambient light.

[0051] (Operation Example of Data Acquisition System 1)

[0052] In the data acquisition system 1, the data acquisition device 10 acquires training data used for learning to generate a learned model for recognizing the object 50 from an image obtained by photographing the object 50. The image of the photographed object 50 includes the background of the object 50. The control unit 12 of the data acquisition device 10 may, for example, Figure 4AAs shown, training data is obtained from a photographic image 60 having 25 pixels arranged in 5×5. The numerical value recorded in the cell corresponding to each pixel of the photographic image 60 corresponds to the brightness of each pixel when the color of each pixel is represented by grayscale. The numerical value represents the brightness in 256 stages from 0 to 255. The larger the numerical value, the closer the pixel is to white. When the numerical value is 0, the color of the pixel corresponding to the cell is black. When the numerical value is 255, the color of the pixel corresponding to the cell is white.

[0053] exist Figure 4A In the above example, the pixels corresponding to the 12 cells with a value of 255 are background pixels. The pixels corresponding to the 13 cells with a value of 190, 160, 120, or 100 are pixels of the photographic object 50. In order to extract the image of the object 50 from the photographic image 60, the control unit 12 may perform the following steps: Figure 4B The mask image 70 is generated as shown in the example. The numerical value recorded in each cell of the mask image 70 indicates the difference between the mask part and the transmission part. The pixel corresponding to the cell with a numerical value of 1 corresponds to the transmission part. The transmission part corresponds to the pixel extracted from the photographic image 60 as the image of the object 50 when the mask image 70 is superimposed on the photographic image 60. The pixel corresponding to the cell with a numerical value of 0 corresponds to the mask part. The mask part corresponds to the pixel not extracted from the photographic image 60 when the mask image 70 is superimposed on the photographic image 60.

[0054] As a comparative example, it is determined whether each pixel of a photographic image is a pixel of an object or a pixel of a background based on the brightness of each pixel. In this case, when the brightness of each pixel of the photographic image is above a threshold value, the pixel is determined to be a pixel of the background. In addition, when the brightness of each pixel of the photographic image is less than a threshold value, the pixel is determined to be a pixel of the object. In the comparative example, when the background is close to black, it is difficult to distinguish between pixels of the object and pixels of the background. Assuming that when the brightness of each pixel is low, it is determined that the pixel is the background, when the brightness of the pixel of the background is close to the brightness of the pixel of the object, it is also difficult to distinguish between pixels of the object and pixels of the background. As a result, it is difficult for the transmission part of the mask image to match the shape of the image of the object. That is, the accuracy of extracting the image of the object is reduced.

[0055] Therefore, the data acquisition system 1 according to the present embodiment causes the imaging device 30 to perform imaging so that the light emitted from the light emitting panel 20 becomes the background of the object 50. Thus, it is easy to distinguish the background from the object 50. As a result, the transmission portion of the mask image 70 for extracting the image of the object 50 is easy to match the shape of the image of the object 50. In other words, the accuracy of extracting the image of the object 50 is increased.

[0056] A specific operation example of the data acquisition system 1 will be described below.

[0057] like Figure 5 As shown, the control unit 12 of the data acquisition device 10 acquires training data for generating a learned model for recognizing the object 50 placed on the light emitting panel 20 . Figure 5 The object 50 shown as an example is a bolt-shaped component. The object 50 is not limited to a bolt, but may be other various components, and is not limited to a component, but may be other various articles.

[0058] The control unit 12 obtains a state where the light emitting panel 20 is lit and the object 50 is not placed on the light emitting panel 20. Fig. 6A Illustrated photographic image 60. Fig. 6A The captured image 60 includes a light-on image 24 in which the light-emitting panel 20 is captured. The control unit 12 obtains a captured image in which the light-emitting panel 20 is captured in a light-on state and the object 50 is placed on the light-emitting panel 20. Figure 6B The photographic image 60 is shown as an example. Figure 6B The photographed image 60 includes an object image 62 in which the object 50 is photographed as a foreground, and a light-on image 24 in which the light-emitting panel 20 is photographed in a lighted state as a background.

[0059] The control unit 12 obtains Fig. 6A The photographic image 60 not including the object image 62 and Figure 6B The difference of the photographic image 60 including the object image 62 is as follows: Figure 6C The mask image 70 is generated as shown. The mask image 70 is also called mask data. In other words, when the control unit 12 obtains at least one photographic image 60, it can generate mask data of the object 50 based on the photographic image 60 obtained by photographing the object 50 located in the light-emitting panel 20 and in front of the light-emitting panel 20 in the at least one photographic image 60 and the photographic image 60 obtained by photographing the light-emitting panel 20 in a state where the light-emitting panel 20 is illuminated and the light-emitting panel 20 in a state where the object 50 is not located in front of the light-emitting panel 20.

[0060] Fig. 6A The photographed image 60 not including the object image 62 is also referred to as a background image. The background image may be a photographed image 60 in which only the light emitting panel 20 is photographed, or may be a photographed image 60 in which the light emitting panel 20 and some indicators are photographed together. Figure 6B The image including the object image 62 is also referred to as a foreground image. In this case, the control unit 12 can generate mask data based on the foreground image and the background image.

[0061] The mask image 70 includes a mask portion 72 and a transmissive portion 74. By improving the contrast between the light-emitting panel 20 in the light-emitting state and the object 50, the accuracy of the shape of the mask portion 72 in the mask image 70 can be improved. The control unit 12 can control the light-emitting panel 20 so as to improve the contrast between the light-emitting panel 20 in the light-emitting state and the object 50. The control unit 12 can determine the light color of the light-emitting panel 20 based on the color of the object 50.

[0062] The light emitting panel 20 and the photographing device 30 may also be placed in a dark room to improve the contrast between the light emitting panel 20 in the lighted state and the object 50. By placing the light emitting panel 20 and the photographing device 30 in a dark room, the control unit 12 can obtain a photographic image 60 in a state where the ambient light does not irradiate the object 50 and the light emitting panel 20.

[0063] The control unit 12 may also control the illumination light 42 of the illumination device 40 so as to increase the contrast between the light-emitting panel 20 in the illuminated state and the object 50. For example, the control unit 12 may set the light emission brightness of the light-emitting panel 20 so that the brightness of the pixel in which the light-emitting panel 20 is photographed in the photographed image 60 is greater than the brightness of the pixel in which the object 50 is photographed.

[0064] The photographing device 30 can photograph an off-time image when the object 50 is placed on the light-emitting panel 20 and the light-emitting panel 20 is off. The photographing device 30 can photograph an on-time image when the object 50 is placed on the light-emitting panel 20 and the light-emitting panel 20 is on. The control unit 12 can generate a mask image 70 as mask data based on the difference between the off-time image and the on-time image. In other words, the control unit 12 can further generate mask data of the object 50 based on a differential image between a photographed image 60 when the light-emitting panel 20 emits light and a photographed image 60 when the light-emitting panel 20 does not emit light.

[0065] The control unit 12 may also generate mask data based only on the foreground image. For example, the control unit 12 may generate mask data of the object 50 by distinguishing between the portion of the foreground image where the light emitting panel 20 is photographed and the portion where the object 50 is photographed. In other words, when the control unit 12 obtains at least one photographic image, it may generate mask data of the object 50 based on a photographic image 60 obtained by photographing the light emitting panel 20 and the object 50 located in front of the light emitting panel 20 in the at least one photographic image while the light emitting panel 20 is emitting light.

[0066] The control unit 12 uses the generated mask image 70 to extract the object image 62 from the captured image 60 to generate an extracted image 64 (see Figure 7CSpecifically, the control unit 12 obtains a state where the light emitting panel 20 is off and the object 50 is placed on the light emitting panel 20. Fig. 7A The photographic image 60 is shown as an example. Fig. 7A The photographed image 60 includes an object image 62 in which the object 50 is photographed as a foreground, and includes an off image 22 in which the light emitting panel 20 is photographed in an off state as a background.

[0067] The control unit 12 can extract the image data of the object 50 from the photographic image 60 used to generate the mask data to generate the extracted image 64. The control unit 12 can extract the image data of the object 50 from the image of the object 50 photographed at the same position as when the photographic image 60 was photographed, based on the mask data of the object 50, to generate the extracted image 64.

[0068] The control unit 12 takes Figure 7B The mask image 70 shown is applied to Fig. 7A The photographic image 60 is used to extract the object image 62, thereby generating Figure 7C The extracted image 64 is shown. The extracted image 64 includes a foreground composed of pixels in which the object 50 is photographed, and a background composed of transparent pixels.

[0069] The control unit 12 can generate training data using the extracted image 64. Specifically, Figure 8 As illustrated, the control unit 12 may generate an image that is a combination of the extracted image 64 and an arbitrary background image 82 as a synthesized image 80. The control unit 12 may output the synthesized image 80 as training data.

[0070] In addition, when generating the extracted image 64, the image obtained by photographing the object 50 may be an image of a state in which the ambient light contacts the object 50. In addition, when generating the extracted image 64, the image obtained by photographing the object 50 may be an image of a state in which the object 50 is arranged at a location different from the light-emitting panel 20.

[0071] Furthermore, when generating the extracted image 64, the control unit 12 may photograph the object 50 while controlling the lighting device 40. That is, in order to increase the diversity of the training data, the object 50 may be photographed under a lighting environment in which the position or brightness of the lighting light 42 is controlled. Furthermore, the object 50 may be photographed under a plurality of lighting environments.

[0072] <Data acquisition method procedure example>

[0073] The data acquisition device 10 may also execute the following steps: Fig. 9The data acquisition method of the steps of the flowchart shown as an example. The data acquisition method can also be implemented as a data acquisition program executed by a processor constituting the control unit 12 of the data acquisition device 10. The data acquisition program can be stored in a non-transitory computer-readable medium.

[0074] The control unit 12 photographs the light emitting panel 20 through the photographing device 30 (step S1). Specifically, the control unit 12 can photograph the light emitting panel 20 through the photographing device 30 in a state where the light emitting panel 20 is turned on and emits light and no object 50 is placed on the light emitting panel 20. The control unit 12 can obtain an image obtained by photographing the light emitting panel 20 that is turned on and emits light.

[0075] The control unit 12 photographs the light emitting panel 20 through the photographing device 30 in a state where the object 50 is placed on the light emitting panel 20 and the light emitting panel 20 is lit and emits light (step S2). The control unit 12 can obtain the image photographed by the photographing device 30. The control unit 12 generates mask data based on the difference between the image of the light emitting panel 20 photographed in a state where the object 50 is not placed and the image of the light emitting panel 20 photographed in a state where the object 50 is placed (step S3). Specifically, the control unit 12 can generate the mask image 70 as the mask data.

[0076] The control unit 12 extracts the image of the object 50 from the photographic image 60 using the mask data and generates an extracted image 64 (step S4). The control unit 12 generates training data using the extracted image 64 (step S5). After executing step S5, the control unit 12 ends. Fig. 9 Execution of the steps of the flowchart.

[0077] Summary

[0078] As described above, according to the data acquisition system 1, the data acquisition device 10, and the data acquisition method of the present embodiment, the contrast between the object 50 and the background can be improved in the photographic image 60 of the object 50. By improving the contrast, mask data for extracting the object 50 can be generated with high accuracy. By generating mask data with high accuracy, it is unnecessary to manually correct the image of the object 50. As a result, annotation can be simplified.

[0079] (Other Implementations)

[0080] Other embodiments will be described below.

[0081] <Influence of the side surface 54 of the object 50>

[0082] like Fig.10As illustrated, the object 50 may include an upper surface 52 and a side surface 54. When the light emitting panel 20 is turned on and emits light, the light emitted from the light emitting panel 20 may be reflected from the side surface 54 and enter the photographing device 30. When the light reflected from the side surface 54 enters the photographing device 30, the side surface 54 of the object 50 may appear to emit light in the photographed image 60.

[0083] Specifically, if Fig.11A As illustrated, when the color of light emitted by the light emitting panel 20 is the same as or similar to the color of the side surface 54 of the object 50, the light emitting panel 20 and the side surface 54 of the object 50 are difficult to distinguish in the photographic image 60. In this case, in the mask image 70, only the upper surface 52 of the object 50 is set in the transmission portion 74, and the side surface 54 is set in the mask portion 72.

[0084] On the other hand, Fig. 11B As illustrated, when the color of the light emitted by the light emitting panel 20 is greatly different from the color of the side surface 54 of the object 50, the light emitting panel 20 and the side surface 54 of the object 50 can be easily distinguished in the photographic image 60. For example, when the color of the light emitted by the light emitting panel 20 and the color of the side surface 54 of the object 50 are in a complementary color relationship, the light emitting panel 20 and the side surface 54 of the object 50 can be easily distinguished in the photographic image 60. In this case, in the mask image 70, the upper surface 52 and the side surface 54 of the object 50 can be set in the transmission portion 74.

[0085] As described above, by making the light emitting panel 20 emit light in at least two colors and generating mask data for each color, the influence of the reflected light from the side surface 54 can be reduced.

[0086] For example, the control unit 12 may cause the light emitting panel 20 to emit the same color as the side surface 54 of the object 50 as the first color, and cause the light emitting panel 20 to emit a color different from the side surface 54 as the second color. Fig.11A The illustrated light emitting panel 20 emits light in a first color. Fig. 12A Example based on Fig.11A The illustrated example is an image of mask data generated by an image obtained by photographing the light emitting panel 20. Fig. 12A The image of the mask data illustrated is an image when the light emitting panel 20 is caused to emit light in the first color, and is referred to as a first mask image 70A. Fig. 11B The illustrated light emitting panel 20 emits light in the second color. Fig. 12B Example based on Fig. 11B The illustrated example is an image of mask data generated by capturing an image of the light emitting panel 20 . Fig. 12BThe image of the mask data illustrated is an image when the light emitting panel 20 is caused to emit light in the second color, and is referred to as a second mask image 70B.

[0087] exist Fig. 12A The first mask image 70A and Fig. 12B In the second mask image 70B, cells surrounded by a thicker frame than other cells represent pixels corresponding to the side surface 54 of the object 50. Fig. 12A In the first mask image 70A, the pixel corresponding to the side surface 54 becomes the mask portion 72. Fig. 12B In the second mask image 70B, the pixel corresponding to the side surface 54 becomes the transmission portion 74. In other words, the pixel corresponding to the side surface 54 becomes the mask portion 72 or the transmission portion 74 depending on whether the light emitting panel 20 emits light in the first color or the second color.

[0088] Here, the control unit 12 can Fig. 12A The first mask image 70A is Fig. 12B Specifically, the control unit 12 calculates the logical sum of each pixel of the first mask image 70A and each pixel of the second mask image 70B to generate the mask image 70. Fig. 12C In other words, the control unit 12 can generate mask data of the object 50 based on a plurality of mask data corresponding to each luminous color of the photographic image 60 when the luminous panel 20 emits light in each luminous color. Fig. 12C In the mask image 70A, the pixel corresponding to the side surface 54 of the object 50 becomes the transmission portion 74. If only the first mask image 70A is generated, the mask data corresponding to the side surface 54 of the object 50 becomes erroneous data. By making the light emitting panel 20 emit at least two different colors and generating mask data in each color, it is less likely that errors in the mask data will occur on the side surface 54 of the object 50.

[0089] like Fig.13 As shown in the flowchart of , the data acquisition device 10 may also execute a data acquisition method including the step of lighting the light emitting panel 20 in multiple colors. The data acquisition method may also be implemented as a data acquisition program executed by a processor constituting the control unit 12 of the data acquisition device 10. The data acquisition program may be stored in a non-transitory computer-readable medium.

[0090] The control unit 12 photographs the light emitting panel 20 through the photographing device 30 (step S11). Specifically, the control unit 12 may photograph the light emitting panel 20 through the photographing device 30 in a state where the light emitting panel 20 is lit to emit light of the first color and the second color respectively and no object 50 is placed on the light emitting panel 20. The control unit 12 may also obtain an image obtained by photographing the light emitting panel 20 that is lit to emit light of the first color and the second color respectively.

[0091] The control unit 12 photographs the light emitting panel 20 with the photographing device 30 when the object 50 is placed on the light emitting panel 20 and the light emitting panel 20 is lit to emit the first color (step S12). The control unit 12 can obtain the image photographed by the photographing device 30 as the first lit image. The control unit 12 generates the first mask image 70A based on the first lit image (step S13).

[0092] The control unit 12 photographs the light emitting panel 20 with the photographing device 30 when the object 50 is placed on the light emitting panel 20 and the light emitting panel 20 is lit to emit the second color (step S14). The control unit 12 can obtain the image photographed by the photographing device 30 as the second lit image. The control unit 12 generates the second mask image 70B based on the second lit image (step S15).

[0093] The control unit 12 calculates the logical sum of the first mask image 70A and the second mask image 70B to generate the mask image 70 (step S16). Specifically, the control unit 12 may calculate the logical sum of each pixel of the first mask image 70A and each pixel of the second mask image 70B to generate an image in which the calculation results of each pixel are arranged as the mask image 70. After executing step S16, the control unit 12 ends. Fig.13 Execution of the steps of the flowchart.

[0094] <Data acquisition station>

[0095] The data acquisition system 1 may include a data acquisition station for acquiring data. The data acquisition station may include a light emitting panel 20 and a plate for placing an object 50 on the light emitting surface of the light emitting panel 20. The plate for placing the object 50 is configured to transmit light emitted from the light emitting panel 20, and is also called a light-transmitting member. The light-transmitting member may be configured so that the object 50 does not directly contact the light emitting surface. The light-transmitting member may be arranged at a distance from the light emitting surface, or may be arranged to contact the light emitting surface.

[0096] The data acquisition station may further include a dark room for housing the light emitting panel 20 and the light-transmitting member. In addition, the data acquisition station may further include an illumination device 40 configured to illuminate the object 50 .

[0097] (Configuration Example of Robot Control System 100)

[0098] like Fig.14 As shown, a robot control system 100 according to one embodiment includes a robot 2 and a robot control device 110. In this embodiment, the robot 2 moves a work object 8 from a work start location 6 to a work target location 7. In other words, the robot control device 110 controls the robot 2 so that the work object 8 moves from the work start location 6 to the work target location 7. The work object 8 is also referred to as a work object. The robot control device 110 controls the robot 2 based on information related to a space in which the robot 2 performs work. Information related to a space is also referred to as space information.

[0099] <Robot control device 110>

[0100] The robot control device 110 acquires a learned model based on learning using the training data generated by the data acquisition device 10. The robot control device 110 recognizes the work object 8, the work start point 6, the work target point 7, etc., which exist in the space where the robot 2 performs work, based on the image photographed by the camera 4 and the learned model. In other words, the robot control device 110 acquires a learned model generated to recognize the work object 8, etc., based on the image photographed by the camera 4.

[0101] In order to provide control and processing capabilities for performing various functions, the robot control device 110 can be configured to include at least one processor. Each component of the robot control device 110 can also be configured to include at least one processor. Multiple components of each component of the robot control device 110 can also be implemented by one processor. The entire robot control device 110 can also be implemented by one processor. The processor is capable of executing programs that implement various functions of the robot control device 110. The processor can be implemented as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor can be implemented as an integrated circuit and a discrete circuit that can be connected in a communicative manner. The processor can be implemented based on various other known technologies.

[0102] The robot control device 110 may include a storage unit. The storage unit may include an electromagnetic storage medium such as a magnetic disk, or may include a memory such as a semiconductor memory or a magnetic memory. The storage unit stores various information and programs executed by the robot control device 110. The storage unit may also be configured as a non-temporary readable medium. The storage unit may function as a working memory of the robot control device 110. At least a portion of the storage unit may also be configured separately from the robot control device 110.

[0103] <Robot 2>

[0104] The robot 2 has an arm 2A and an end effector 2B. The arm 2A can be configured as a 6-axis or 7-axis vertical multi-joint robot, for example. The arm 2A can also be configured as a 3-axis or 4-axis horizontal multi-joint robot or a SCARA robot. The arm 2A can also be configured as a 2-axis or 3-axis orthogonal robot. The arm 2A can also be configured as a parallel linkage robot, etc. The number of axes constituting the arm 2A is not limited to the number illustrated. In other words, the robot 2 has an arm 2A connected by a plurality of joints, and moves by driving the joints.

[0105] The end effector 2B may include, for example, a gripping manipulator configured to grip the work object 8. The gripping manipulator may have a plurality of fingers. The number of fingers of the gripping manipulator may be two or more. The fingers of the gripping manipulator may have one or more joints. The end effector 2B may also include a suction manipulator configured to suck up the work object 8. The end effector 2B may also include a shoveling hand configured to scoop up the work object 8. The end effector 2B may also be configured to include tools such as a drill, and may perform various processing operations such as drilling holes in the work object 8. The end effector 2B is not limited to these examples, and may also be configured to be capable of performing various other actions. Fig.14 In the illustrated structure, the end effector 2B includes a gripping robot.

[0106] The robot control device 110 can control the position of the end effector 2B by moving the arm 2A of the robot 2. The end effector 2B may also have an axis that serves as a reference for the direction of the action on the work object 8. When the end effector 2B has an axis, the robot control device 110 can control the direction of the axis of the end effector 2B by moving the arm 2A of the robot 2. The robot control device 110 controls the start and end of the action of the end effector 2B on the work object 8. The robot control device 110 controls the position of the end effector 2B or the direction of the axis of the end effector 2B while controlling the action of the end effector 2B, thereby moving or processing the work object 8. Fig.14 In the illustrated structure, the robot control device 110 causes the end effector 2B to hold the work object 8 at the work start point 6 and moves the end effector 2B to the work target point 7. The robot control device 110 causes the end effector 2B to release the work object 8 at the work target point 7. In this way, the robot control device 110 can move the work object 8 from the work start point 6 to the work target point 7 through the robot 2.

[0107] <Sensor 3>

[0108] like Fig.14As shown, the robot control system 100 is further provided with a sensor 3. The sensor 3 detects physical information of the robot 2. The physical information of the robot 2 may include information related to the actual position or posture of each component of the robot 2, or the speed or acceleration of each component of the robot 2. The physical information of the robot 2 may include information related to the force acting on each structural part of the robot 2. The physical information of the robot 2 may include information related to the current flowing in the motor driving each component of the robot 2 or the torque of the motor. The physical information of the robot 2 represents the result of the actual action of the robot 2. In other words, the robot control system 100 can grasp the result of the actual action of the robot 2 by obtaining the physical information of the robot 2.

[0109] The sensor 3 may include a force sensor or a tactile sensor that detects a force, a distributed pressure, or a sliding motion, etc., acting on the robot 2 as physical information of the robot 2. The sensor 3 may include a motion sensor that detects a position or a posture, or a speed or an acceleration of the robot 2 as physical information of the robot 2. The sensor 3 may include a current sensor that detects a current flowing in a motor that drives the robot 2 as physical information of the robot 2. The sensor 3 may include a torque sensor that detects a torque of a motor that drives the robot 2 as physical information of the robot 2.

[0110] The sensor 3 may be provided at a joint of the robot 2 or a joint driving unit that drives the joint. The sensor 3 may also be provided at the arm 2A or the end effector 2B of the robot 2.

[0111] The sensor 3 outputs the detected physical information of the robot 2 to the robot control device 110. The sensor 3 detects and outputs the physical information of the robot 2 at a given timing. The sensor 3 outputs the physical information of the robot 2 as time-series data.

[0112] <Camera 4>

[0113] exist Fig.14In the structural example shown, the robot control system 100 has two cameras 4. The camera 4 photographs objects or people located in the influence range 5 that may affect the movement of the robot 2. The image photographed by the camera 4 may include monochrome brightness information or brightness information of each color represented by RGB or the like. The influence range 5 includes the movement range of the robot 2. The influence range 5 is a range that further extends the movement range of the robot 2 to the outside. The influence range 5 can be set to be able to stop the robot 2 before a person moving from the outside of the movement range of the robot 2 to the inside of the movement range enters the inside of the movement range of the robot 2. The influence range 5 can also be set, for example, to a range extending from a given distance from the boundary of the movement range of the robot 2 to the outside. The camera 4 can also be set to be able to photograph the influence range 5 or the movement range of the robot 2 or the area around them from a bird's-eye view. The number of cameras 4 is not limited to two, and can be one or more than three.

[0114] (Operation Example of Robot Control System 100)

[0115] The robot control device 110 obtains the learned model in advance. The robot control device 110 may store the learned model in the storage unit. The robot control device 110 obtains an image of the work object 8 from the camera 4. The robot control device 110 inputs the image of the work object 8 as input information to the learned model. The robot control device 110 obtains output information output according to the input information from the learned model. The robot control device 110 recognizes the work object 8 based on the output information and performs the operation of holding or moving the work object 8.

[0116] Summary

[0117] As described above, the robot control system 100 can acquire a learned model based on learning using the training data generated by the data acquisition system 1 and recognize the work object 8 using the learned model.

[0118] The above describes the implementation of the data acquisition system 1 and the robot control system 100, but as an implementation of the present disclosure, in addition to the method or program for implementing the system or device, it is also possible to adopt an implementation as a storage medium with a program recorded thereon (as an example, a CD, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card, etc.).

[0119] In addition, the installation method of the program is not limited to application programs such as object codes compiled by a compiler and program codes executed by an interpreter, and may also be a program module compiled into an operating system. Furthermore, the program may be configured so that all processing is performed only in the CPU on the control substrate. The program may also be configured so that part or all of it is performed by other processing units installed in an expansion board or expansion unit attached to the substrate as needed.

[0120] Although the embodiments of the present disclosure have been described based on the drawings and embodiments, it should be noted that those skilled in the art may make various deformations or modifications based on the present disclosure. Therefore, it should be noted that these deformations or changes are included in the scope of the present disclosure. For example, the functions included in each component, etc. can be reconfigured to be logically consistent, and multiple components, etc. can be combined into one or divided.

[0121] All structural elements and / or all methods or steps of processing disclosed in the present disclosure can be combined in any combination except for the mutually exclusive combination of these features. In addition, the features recorded in the present disclosure can be replaced with the same purpose, equivalent purpose, or alternative features that work for similar purposes as long as they are not explicitly denied. Therefore, unless explicitly denied, each feature disclosed is only an example of a series of identical or equal features included.

[0122] Furthermore, the embodiments of the present disclosure are not limited to any specific configuration in the above-mentioned embodiments. The embodiments of the present disclosure can be extended to all new features or their combinations, or all new methods or processing steps or their combinations recorded in the present disclosure.

[0123] In the present disclosure, the descriptions of "first" and "second" etc. are identifiers for distinguishing the structure. The structures distinguished by the descriptions of "first" and "second" etc. in the present disclosure can exchange the numbers in the structures. For example, the first mask image 70A can exchange "first" and "second" as identifiers with the second mask image 70B. The identifiers are exchanged at the same time. The structures can also be distinguished after the identifiers are exchanged. The identifiers can be deleted. The structures from which the identifiers are deleted are distinguished by the figure marks. The descriptions of the identifiers of "first" and "second" etc. in the present disclosure alone cannot be used to interpret the order of the structures or the basis for the existence of identifiers with small numbers.

[0124] In one embodiment, (1) a data acquisition device includes a control unit configured to control a light emitting panel and to acquire at least one photographic image obtained by photographing a light emitting surface of the light emitting panel. The control unit generates mask data of the object based on a photographic image of the light emitting panel and an object located in front of the light emitting panel in a state where the light emitting panel is illuminated, among the at least one photographic image.

[0125] (2) In the data acquisition device of (1) above, the control unit may determine the light emission color of the light emitting panel based on the color of the object.

[0126] (3) In the data acquisition device of (1) or (2) above, the control unit may cause the light-emitting panel to emit light in a plurality of light-emitting colors, and generate mask data of the object by generating a plurality of mask data corresponding to each light-emitting color based on a photographic image when the light-emitting panel emits light in each light-emitting color.

[0127] (4) In the data acquisition device described in any one of (1) to (3) above, the control unit may generate mask data of the object based on a photographic image obtained by photographing the light-emitting panel without the object in the at least one photographic image.

[0128] (5) In the data acquisition device described in any one of (1) to (4) above, the control unit may generate mask data of the object based on a difference image between a photographic image when the light-emitting panel emits light and a photographic image when the light-emitting panel does not emit light.

[0129] (6) In the data acquisition device according to any one of (1) to (5) above, the control unit may acquire a photographic image in a state where the object and the light-emitting panel are not irradiated with ambient light.

[0130] (7) In the data acquisition device described in any one of (1) to (6) above, the control unit may set the light emission brightness of the light emission panel so that the brightness of the light emission panel is greater than the brightness of the object in the photographic image.

[0131] (8) In the data acquisition device described in any one of (1) to (7) above, the control unit may extract image data of the object from an image obtained by photographing the object at the same position as when the photographic image was photographed, based on mask data of the object.

[0132] (9) In the data acquisition device according to (8) above, the control unit may control the illumination light that illuminates the object.

[0133] In one embodiment, (10) a data acquisition method includes: causing a light emitting panel to emit light; and generating mask data of the object based on a photographic image obtained by photographing the object located in front of the light emitting panel and the light emitting panel.

[0134] (11) The data acquisition method of (10) above may further include extracting image data of the object from an image obtained by photographing the object at the same position as when the photographed image was photographed, based on mask data of the object.

[0135] In one embodiment, (12) a data acquisition station includes: a light emitting panel that emits light in a predetermined color; and a light transmitting member that is located between an object disposed in front of the light emitting panel and the light emitting panel.

[0136] (13) The data acquisition station described in (12) above may further include a darkroom that accommodates the light emitting panel and the light-transmitting member.

[0137] (14) The data acquisition station according to (12) or (13) above may further include a lighting device configured to illuminate the object.

[0138] (15) In the data acquisition station described in any one of (12) to (14) above, the light emitting panel may emit light of one of given colors.

[0139] -Explanation of symbols-

[0140] 1 Data acquisition system

[0141] 10 Data acquisition device (12: control unit, 14: storage unit, 16: interface)

[0142] 20 Illuminated panel (22: extinguish image, 24: light up image)

[0143] 30 Photographic Installation

[0144] 40 lighting device (42: lighting light)

[0145] 50 Object (52: top surface, 54: side surface)

[0146] 60 Photographic image (62: image of the object, 64: extracted image of the object)

[0147] 70 Mask image (70A: first mask image, 70B: second mask image, 72: mask portion, 74: transmission portion)

[0148] 80 Composite image (82: Background image)

[0149] 100 Robot control system (2: robot, 2A: arm, 2B: end effector, 3: sensor, 4: camera, 5: influence range, 6: work start point, 7: work target point, 8: work object, 110: robot control device).

Claims

1. A data acquisition device, A control unit is provided, the control unit being configured to be able to control the light emitting panel and to be able to obtain at least one photographic image obtained by photographing the light emitting surface of the light emitting panel, The control unit generates mask data of the object based on a photographic image obtained by photographing the light emitting panel and an object located in front of the light emitting panel in a state where the light emitting panel is caused to emit light, among the at least one photographic image.

2. The data acquisition device according to claim 1, wherein: The control unit determines a light emission color of the light emitting panel based on a color of the object.

3. The data acquisition device according to claim 1 or 2, wherein: The control unit causes the light emitting panel to emit light in a plurality of light emitting colors. The control unit generates the mask data of the object based on a plurality of mask data corresponding to each emission color of a photographed image when the light emitting panel emits light in each emission color.

4. The data acquisition device according to claim 1, wherein: The control unit generates mask data of the object based on a photographic image obtained by photographing the light emitting panel in a state where the object is not arranged, among the at least one photographic image.

5. The data acquisition device according to claim 1 or 2, wherein: The control unit generates mask data of the object based on a difference image between an image captured when the light emitting panel emits light and an image captured when the light emitting panel does not emit light.

6. The data acquisition device according to claim 1 or 2, wherein: The control unit acquires a photographic image in a state where the object and the light emitting panel are not irradiated with ambient light.

7. The data acquisition device according to claim 1 or 2, wherein: The control unit sets the light emission brightness of the light emission panel so that the brightness of the light emission panel is greater than the brightness of the object in the photographic image.

8. The data acquisition device according to claim 1 or 2, wherein: The control unit extracts image data of the object from an image obtained by photographing the object at the same position as when the photographic image was photographed, based on mask data of the object.

9. The data acquisition device according to claim 8, wherein: The control unit controls illumination light that illuminates the object.

10. A data acquisition method, comprising: causing the light-emitting panel to emit light; as well as Mask data of the object is generated based on a photographic image obtained by photographing the object located in front of the light emitting panel and the light emitting panel.

11. The data acquisition method according to claim 10, wherein: Also includes: Based on the mask data of the object, image data of the object is extracted from an image obtained by photographing the object at the same position as when the photographed image was photographed.

12. A data acquisition station, comprising: a light-emitting panel that emits light in a given color; and The light-transmitting member is located between an object arranged in front of the light-emitting panel and the light-emitting panel.

13. The data acquisition station according to claim 12, wherein: The invention further comprises a darkroom for housing the light emitting panel and the light transmitting member.

14. The data acquisition station according to claim 12 or 13, wherein: The device further includes a lighting device configured to illuminate the object.

15. The data acquisition station according to claim 12, wherein: The light emitting panel emits light of one of given colors.

Images