Data acquisition device, data acquisition method, and data acquisition station

Abstract

This data acquisition device is provided with a control unit that is configured so as to be able to control a display device having a plurality of pixels that are controlled to be in one of an on state and an off state. The display device is configured so as to be able to acquire a captured image obtained by capturing an image of an object located in front of the display device and the display device. The control unit controls the state of each pixel to one of a turned-on state and a turned-off state such that the number of the pixels in the turned-on state is increased and the number of the pixels in the turned-on state reflected in the captured image is reduced. Or reducing the number of pixels in the extinguished state and increasing the number of pixels in the extinguished state shown in the captured image; mask data of the object is generated on the basis of the arrangement of the pixels in the lighted state.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority of Japanese Patent Application No. 2022-88692 (filed on May 31, 2022), the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to a data acquisition device, a data acquisition method, and a data acquisition station. Background Art

[0004] Conventionally, there is known a system that generates learning data for learning in semantic segmentation or the like (see, for example, Patent Document 1).

[0005] Prior Art Literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2020-102041 Summary of the invention

[0008] A data acquisition device according to an embodiment of the present disclosure includes a control unit configured to control a display device having a plurality of pixels controlled to be in one of a lit state and an unlit state, and configured to acquire a photographic image obtained by photographing an object located in front of the display device and the display device. The control unit controls the state of each pixel to be in one of the lit state and the unlit state so that the number of pixels in the lit state is increased and the number of pixels in the lit state reflected in the photographic image is decreased, or the number of pixels in the unlit state is decreased and the number of pixels in the unlit state reflected in the photographic image is increased. The control unit generates mask data of the object based on the configuration of the pixels in the lit state.

[0009] An embodiment of the present disclosure involves a data acquisition method that includes the following processing: in a display device having a plurality of pixels that are controlled to be in a state either of a lit state and an unlit state, controlling the state of each pixel to be in a state either of the lit state and the unlit state so as to increase the number of pixels in the lit state and decrease the number of pixels in the lit state reflected in a photographic image, or to decrease the number of pixels in the unlit state and increase the number of pixels in the unlit state reflected in the photographic image, the photographic image being obtained by photographing an object located in front of the display device and the display device; and generating mask data of the object based on the configuration of the pixels in the lit state.

[0010] A data acquisition station according to one embodiment of the present disclosure includes: a display device having a plurality of pixels; and a light-transmitting member located between an object disposed in front of the display device and the display device. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a block diagram showing a configuration example of a data acquisition system according to one embodiment.

[0012] Figure 2 It is a top view showing a configuration example of a data acquisition system.

[0013] Figure 3 yes Figure 2 AA section view.

[0014] Figure 4A This is a diagram showing an example of the brightness of each pixel in a photographic image of an object.

[0015] Figure 4B Is based on Figure 4A Figure 3 is a diagram showing an example of a mask image generated from a photographic image.

[0016] Figure 5 This is a plan view showing an example of an object located on the light emitting panel.

[0017] Figure 6 This is a diagram showing an example of a light emitting panel including a lighting range and an extinguishing range.

[0018] Fig. 7A This is a diagram showing an example of a light emitting panel that includes pixels in a lighting range and pixels in a non-lighting range and is controlled so as to enlarge the lighting range.

[0019] Figure 7B This is a diagram showing an example of a light emitting panel which includes pixels in a lighting range and pixels in a non-lighting range and is controlled so as to narrow the lighting range.

[0020] Figure 8 This is a diagram showing an example of a light emitting panel in which a range where an object is located coincides with a lighting range.

[0021] Fig. 9 It is assumed that Figure 8 A stereoscopic view of a light emitting panel when an object is moved in a normal direction of the light emitting panel.

[0022] Fig. 10A This is a diagram showing an example of a photographic image of an object located on a light-emitting panel in a turned-off state.

[0023] Fig. 10B This is a diagram showing an example of a mask image.

[0024] Fig. 10C It means that Fig. 10B The mask image is applied to Fig. 10A A diagram showing an example of an extracted image obtained by extracting an image of an object from a photographic image.

[0025] Fig.11 It means that Fig. 10C An example of training data generated by overlapping an extracted image with a background image.

[0026] Fig.12 This is a flowchart showing an example of the procedure of the data acquisition method.

[0027] Fig.13 This is a flowchart showing an example of a procedure for determining a lighting range.

[0028] Fig.14 This is a diagram showing an example in which the lighting range is enlarged in one direction.

[0029] Fig.15 This is a diagram showing an example of moving a strip-shaped lighting range.

[0030] Fig.16 This is a diagram showing an example in which each section of the light emitting panel is caused to emit light in sequence in a given pattern.

[0031] Fig.17 This is a schematic diagram showing a configuration example of a robot control system. DETAILED DESCRIPTION

[0032] (Configuration Example of Data Acquisition System 1)

[0033] A data acquisition system 1 involved in one embodiment of the present disclosure acquires training data for generating a learned model, which outputs a recognition result of a recognition object contained in input information. The learned model can be constructed to include a CNN (Convolution Neural Network) having multiple layers. For the information input to the learned model, convolution based on a given weighting coefficient is performed in each layer of the CNN. During the learning of the learned model, the weighting coefficient is updated. The learned model can be constructed to include a fully combined layer. The learned model can also be composed of VGG16 or ResNet50. The learned model can also be constructed as a transformer. The learned model is not limited to these examples, and can also be constructed as various other models.

[0034] like Figure 1 , Figure 2 as well as Figure 3As shown, a data acquisition system 1 involved in one embodiment of the present disclosure includes a data acquisition device 10, a light emitting panel 20, and a photographing device 30. The light emitting panel 20 has a light emitting surface, and is configured to be able to place an object 50 for acquiring training data on the light emitting surface. The photographing device 30 is configured to photograph the object 50 placed on the light emitting panel 20 and the light emitting panel 20. The data acquisition device 10 controls the light emitting state of the light emitting panel 20. The data acquisition device 10 acquires an image obtained by the photographing device 30 photographing the light emitting panel 20 and the object 50. An image of the light emitting panel 20 and the object 50 is also called a photographed image. The data acquisition device 10 is configured to be able to acquire a photographed image. The data acquisition device 10 generates training data of the object 50 based on the photographed image and the light emitting state of the light emitting panel 20 when the photographic image is photographed, and acquires the training data.

[0035] <Data acquisition device 10>

[0036] The data acquisition device 10 includes a control unit 12 , a storage unit 14 , and an interface 16 .

[0037] The control unit 12 may be configured to include at least one processor in order to provide control and processing capabilities for performing various functions. The processor may execute programs that implement various functions of the control unit 12. The processor may be implemented as a single integrated circuit. An integrated circuit is also referred to as an IC (Integrated Circuit). The processor may be implemented as an integrated circuit and a discrete circuit that can be connected to multiple communications. The processor may be implemented based on various other known technologies.

[0038] The storage unit 14 may include an electromagnetic storage medium such as a magnetic disk, or may include a memory such as a semiconductor memory or a magnetic memory. The storage unit 14 stores various information. The storage unit 14 stores programs executed by the control unit 12, etc. The storage unit 14 may also be configured as a non-temporary readable medium. The storage unit 14 may function as a working memory of the control unit 12. At least a part of the storage unit 14 may also be configured separately from the control unit 12.

[0039] The interface 16 is configured to input and output information or data between the light emitting panel 20 and the imaging device 30. The interface 16 may be configured to include a communication device configured to communicate by wire or wirelessly. The communication device may be configured to communicate in a communication manner based on various communication standards. The interface 16 may be configured using known communication technologies.

[0040] The interface 16 may be configured to include a display device. The display device may include various displays such as a liquid crystal display. The interface 16 may be configured to include a sound output device such as a speaker. The interface 16 is not limited to these and may also be configured to include various other output devices.

[0041] The interface 16 may be configured to include an input device for receiving input from a user. The input device may include, for example, a keyboard or physical keys, or may include a touch panel or touch sensor or a pointing device such as a mouse. The input device is not limited to these examples, and may also be configured to include various other devices.

[0042] <Light-emitting panel 20>

[0043] The light-emitting panel 20 has a light-emitting surface. The light-emitting panel 20 has a plurality of pixels arranged on the light-emitting surface. The light-emitting panel 20 can be configured to control the state of each pixel to be a lit state or an unlit state. Each pixel of the light-emitting panel 20 can be configured as a self-luminous element. Each pixel of the light-emitting panel 20 can be configured to be lit when the shutter is opened and unlit when the shutter is closed by combining a shutter and a backlight that are opened and closed. The light-emitting panel 20 can be, for example, a liquid crystal panel or various display devices such as an organic EL (Electro-luminescence) or an inorganic EL panel.

[0044] <Photographic device 30>

[0045] The photographing device 30 may be configured to include various imaging elements or cameras. The photographing device 30 is configured to be able to photograph the light-emitting surface of the light-emitting panel 20 and the object 50 placed on the light-emitting surface. In other words, the photographing device 30 is configured to be able to photograph the object 50 located in front of the light-emitting panel 20 as viewed from the photographing device 30 together with the light-emitting panel 20. The photographing device 30 may also be configured to photograph the light-emitting surface of the light-emitting panel 20 from various directions. The photographing device 30 may be configured so that the normal direction of the light-emitting surface of the light-emitting panel 20 coincides with the optical axis of the photographing device 30.

[0046] The data acquisition system 1 may further include a dark room for storing the light emitting panel 20 and the photographing device 30. When the light emitting panel 20 and the photographing device 30 are stored in the dark room, the side of the object 50 facing the photographing device 30 is not irradiated by ambient light. When the side of the object 50 facing the photographing device 30 is not irradiated by ambient light, the image of the object 50 photographed by the photographing device 30 becomes an image of a color close to black or black. When pixels of the light emitting panel 20 that extend to a range outside the range where the object 50 exists are lit, the image of the object 50 photographed by the photographing device 30 becomes an image showing the outline of the object 50.

[0047] The data acquisition system 1 may further include an illumination device 40 that emits illumination light for illuminating the object 50. The illumination device 40 may be configured to emit illumination light as light of various colors. When the data acquisition system 1 includes the illumination device 40, the photographing device 30 may photograph the object 50 in a state where the object 50 is illuminated by the illumination light and the ambient light. When the data acquisition system 1 includes the illumination device 40 and a dark room, the photographing device 30 may photograph the object 50 in a state where the object 50 is illuminated by the illumination light. When the data acquisition system 1 does not include the illumination device 40, the photographing device 30 may photograph the object 50 in a state where the object 50 is illuminated by the ambient light.

[0048] (Operation Example of Data Acquisition System 1)

[0049] In the data acquisition system 1, the data acquisition device 10 acquires training data used for learning to generate a learned model for recognizing the object 50 from an image obtained by photographing the object 50. The image of the object 50 includes the background of the object 50. The control unit 12 of the data acquisition device 10 may, for example, Figure 4A As shown, an image of an object 50 is extracted from a photographic image 60 having 25 pixels arranged in 5×5 and training data is obtained. The numerical value recorded in the cell corresponding to each pixel corresponds to the brightness of each pixel when the color of each pixel is represented by grayscale. The numerical value represents the brightness in 256 stages from 0 to 255. The larger the numerical value, the closer the pixel is to white. When the numerical value is 0, the color of the pixel corresponding to the cell is black. When the numerical value is 255, the color of the pixel corresponding to the cell is white.

[0050] exist Figure 4A In the above, the pixels corresponding to the 12 cells with a value of 255 are background pixels. The pixels corresponding to the 13 cells with a value of 190, 160, 120, or 100 are pixels in which the object 50 is photographed. In order to extract the image of the object 50 from the photographed image 60, the control unit 12 may perform the following operations: Figure 4B The mask image 70 is generated as shown in the example. The numerical value recorded in each cell of the mask image 70 indicates the difference between the mask part and the transmission part. The pixel corresponding to the cell with a numerical value of 1 corresponds to the transmission part. When the mask image 70 is superimposed on the photographic image 60, the transmission part corresponds to the pixel extracted from the photographic image 60 as the image of the object 50. The pixel corresponding to the cell with a numerical value of 0 corresponds to the mask part. The mask part corresponds to the pixel not extracted from the photographic image 60 when the mask image 70 is superimposed on the photographic image 60. The mask image 70 is used as mask data for extracting the image of the object 50 from the photographic image 60.

[0051] As a comparative example, it is determined whether each pixel of a photographic image is a pixel of an object or a pixel of a background based on the brightness of each pixel. In this case, when the brightness of each pixel of the photographic image is above a threshold value, the pixel is determined to be a pixel of the background. In addition, when the brightness of each pixel of the photographic image is less than the threshold value, the pixel is determined to be a pixel of the object. In the comparative example, when the background is close to black, it is difficult to distinguish between the pixel of the object and the pixel of the background. Assuming that the pixel is determined to be the background when the brightness of each pixel is low, it is also difficult to distinguish between the pixel of the object and the pixel of the background when the brightness of the pixel of the background is close to the brightness of the pixel of the object. As a result, it is difficult for the transmission part of the mask image to match the shape of the image of the object. That is, the accuracy of extracting the image of the object is reduced.

[0052] Therefore, the data acquisition device 10 according to the present embodiment generates a mask image 70 of the object 50 as mask data for extracting the image of the object 50 based on the image of the object 50 and the state of each pixel of the light emitting panel 20 when the image is captured. Specifically, when only pixels located behind the object 50 as viewed from the imaging device 30 among the pixels of the light emitting panel 20 are lit, the range where the lit pixels are located coincides with the range where the object 50 is located. By generating mask data based on the range where the lit pixels are located, the transmission portion of the mask image 70 for extracting the image of the object 50 easily coincides with the shape of the image of the object 50. As a result, the accuracy of extracting the image of the object 50 is increased.

[0053] In other words, the control unit 12 of the data acquisition device 10 is configured to control a display device having a plurality of pixels controlled to be in a state of either a lit state or an unlit state, and is configured to acquire a photographic image 60 obtained by photographing the object 50 and the display device located in front of the display device. The control unit 12 controls the state of each pixel to be in a state of either a lit state or an unlit state, and generates mask data of the object 50 based on the arrangement of the pixels in the lit state so that the number of pixels in the lit state is increased and the number of pixels in the lit state reflected in the photographic image 60 is reduced.

[0054] A specific operation example of the data acquisition system 1 will be described below.

[0055] The control unit 12 of the data acquisition device 10 acquires training data for generating a learned model for recognizing the object 50. In order to acquire the training data of the object 50, Figure 5 As shown, it is assumed that an object 50 is placed on the light emitting panel 20 . Figure 5The object 50 shown as an example is a bolt-shaped component. The object 50 is not limited to a bolt, but may be other various components, and is not limited to a component, but may be other various articles.

[0056] like Figure 6 As shown, the control unit 12 determines the initial arrangement of the pixels to be lit so that the shape of the lighting range 24 of the light emitting panel 20 when viewed from the photographing device 30 is close to the shape of the object 50. The initial setting of the lighting range 24 is also referred to as the initial lighting range. The control unit 12 can recognize the shape of the object 50 from the image photographed in a state where the object 50 is placed on the light emitting panel 20, thereby setting the initial lighting range. The control unit 12 can also set the initial lighting range by various other methods.

[0057] The control unit 12 may determine the pixels to be lit so that the lighting range 24 of the light emitting panel 20 is wider than the object 50. That is, the control unit 12 may determine the pixels to be lit so that a portion of the lighting range 24 of the light emitting panel 20 can be seen from the photographing device 30. When the lighting range 24 is wider than the object 50, the control unit 12 may reduce the lighting range 24 reflected in the image based on the image photographed by the photographing device 30 (extend the extinguishing range 22 inward), thereby making the lighting range 24 close to the shape of the object 50.

[0058] The control unit 12 may determine the pixels to be lit so that the lighting range 24 of the light emitting panel 20 is narrower than the object 50. In other words, the control unit 12 may determine the pixels to be lit so that the lighting range 24 of the light emitting panel 20 cannot be seen from the photographing device 30. In the case where the lighting range 24 is narrower than the object 50, the control unit 12 may expand the lighting range 24 outward based on the image photographed by the photographing device 30 until the lighting range 24 is reflected in the image, thereby narrowing the lighting range 24 (expanding the extinguishing range 22 inward), so that the lighting range 24 is close to the shape of the object 50.

[0059] When the control unit 12 expands the lighting range 24, Fig. 7A As shown in the example, the cell in which "1" indicating the transmission part located inside the mask image 70 is recorded can be expanded toward the cell in which "0" indicating the mask part located outside the mask image 70 is recorded by morphological processing. Figure 7B As illustrated, the cell in which “0” indicating the mask portion located outside the mask image 70 is written can be shrunk toward the cell in which “1” indicating the transmission portion located inside the mask image 70 by morphological processing.

[0060] like Figure 8As shown, the control unit 12 controls the lighting range 24 so that the lighting range 24 cannot be seen from the photographing device 30, and only the object 50 and the extinguishing range 22 can be seen from the photographing device 30. Fig. 9 As shown in FIG. 1 , the control unit 12 maximizes the lighting range 24 of the pixels located behind the object 50 as viewed from the photographing device 30. That is, the control unit 12 controls the state of each pixel to be either a lighting state or an unlighted state, so as to increase the number of pixels in the lighting state and reduce the number of pixels in the lighting state reflected in the photographic image 60. By controlling the state of each pixel as described above, the control unit 12 Figure 8 as well as Fig. 9 As shown, the shape of lighting range 24 can be made close to the shape of object 50 .

[0061] The control unit 12 may generate mask data based on the arrangement of the lighted pixels constituting the lighted range 24 when the lighted range 24 is maximized and the lighted range 24 shown in the photographic image 60 is minimized. The control unit 12 may determine that the lighted range 24 is maximized and the lighted range 24 shown in the photographic image 60 is minimized on the condition that the difference between the number of lighted pixels when a part of the lighted range 24 is shown in the photographic image 60 and the number of lighted pixels when the lighted range 24 is not shown in the photographic image 60 is within a given value.

[0062] The control unit 12 can converge the setting of the lighting range 24 by repeatedly performing the steps of enlarging and reducing the lighting range 24. The control unit 12 can determine that the setting of the lighting range 24 has converged when the number of times the enlargement and reduction of the lighting range 24 is repeated reaches a given number or more. In other words, the control unit 12 can determine that the setting of the lighting range 24 has converged when the number of repetitions of enlarging and reducing the lighting range 24 reaches or exceeds a determination threshold.

[0063] The control unit 12 can determine whether a change occurs in the photographic image 60 between a state where the lighted pixel is reflected and a state where the lighted pixel is not reflected by enlarging or reducing only one pixel for the pixels located at the outline of the object 50. The control unit 12 can set the lighting range 24 by enlarging or reducing only one pixel for all the pixels located at the outline of the object 50 so that a change occurs in the photographic image 60 between a state where the lighted pixel is reflected and a state where the lighted pixel is not reflected. In this case, it is determined that the setting of the lighting range 24 has converged.

[0064] The control unit 12 uses the generated mask image 70 to extract the object image 62 from the captured image 60 to generate an extracted image 64 (see Fig. 10C Specifically, the control unit 12 obtains a state where the light emitting panel 20 is off and the object 50 is placed on the light emitting panel 20. Fig. 10A The photographic image 60 is shown as an example. Fig. 10A The photographed image 60 includes an object image 62 in which the object 50 is photographed as a foreground, and includes a turn-off range 22 in which the light emitting panel 20 is turned off as a background.

[0065] The control unit 12 can extract the image data of the object 50 from the photographic image 60 used to generate the mask data to generate the extracted image 64. The control unit 12 can extract the image data of the object 50 from an image obtained by photographing the object 50 at the same position as when the photographic image 60 was photographed, based on the mask data of the object 50, to generate the extracted image 64.

[0066] The control unit 12 takes Fig. 10B The mask image 70 shown is applied to Fig. 10A The photographic image 60 is used to extract the object image 62, thereby generating Fig. 10C The extracted image 64 is shown. The mask image 70 includes a mask portion 72 and a transmissive portion 74. In the photographic image 60, a portion corresponding to the transmissive portion 74 is extracted as the object image 62. The extracted image 64 includes a foreground composed of pixels of the photographic object 50 and a background composed of transparent pixels.

[0067] The control unit 12 can generate training data using the extracted image 64. Specifically, Fig.11 As illustrated, the control unit 12 may generate an image that is a combination of the extracted image 64 and an arbitrary background image 82 as a synthesized image 80. The control unit 12 may output the synthesized image 80 as training data.

[0068] <Data acquisition method procedure example>

[0069] The data acquisition device 10 may also execute Fig.12 The data acquisition method of the steps of the flowchart shown as an example. The data acquisition method can also be implemented as a data acquisition program executed by a processor constituting the control unit 12 of the data acquisition device 10. The data acquisition program can be stored in a non-transitory computer-readable medium.

[0070] The control unit 12 acquires an initial lighting range corresponding to a state where the object 50 is placed on the light emitting panel 20 (step S1 ). The control unit 12 lights up pixels in the initial lighting range of the light emitting panel 20 (step S2 ).

[0071] The control unit 12 determines the lighting range 24 based on the photographic image 60 of the photographing device 30 so as to increase the number of lit pixels of the light emitting panel 20 and reduce the number of lit pixels reflected in the photographic image 60 (step S3). The control unit 12 determines the lighting range 24 as the arrangement of the lit pixels of the light emitting panel 20.

[0072] The control unit 12 generates mask data based on the determined lighting range 24 (step S4). Specifically, the control unit 12 sets the pixels corresponding to the positions of the lighting pixels of the light emitting panel 20 as the transmission parts, and sets the pixels corresponding to the positions of the extinguishing pixels of the light emitting panel 20 as the mask parts in the mask data.

[0073] The control unit 12 extracts the image of the object 50 from the photographic image 60 using the mask data to generate training data (step S5). After the control unit 12 executes step S5, the process ends. Fig.12 Execution of the steps of the flowchart.

[0074] As Fig.12 In the step S3 of determining the lighting range 24, the control unit 12 may also execute Fig.13 Steps of the illustrated flowchart.

[0075] The control unit 12 determines whether there are lit pixels in the photographic image 60 (step S11). If there are no lit pixels in the photographic image 60 (step S11: No), the control unit 12 enlarges the lighting range 24 considering the possibility that the pixels behind the object 50 include unlit pixels (step S12). If there are lit pixels in the photographic image 60 (step S11: Yes), the control unit 12 narrows the lighting range 24 to reduce the number of lit pixels (step S13). After executing step S12 or S13, the control unit 12 proceeds to step S14.

[0076] The control unit 12 determines again whether there are lit pixels in the photographic image 60 (step S14). When the photographic image 60 shows lit pixels (step S14: Yes), the control unit 12 returns to the step S13 and further narrows the lighting range 24. When the photographic image 60 does not show lit pixels (step S14: No), the control unit 12 determines whether the number of repetitions of the steps of enlarging and reducing the lighting range 24 in steps S12 and S13 is greater than the determination threshold (step S15). When the number of repetitions is not greater than the determination threshold (step S15: No), that is, when the number of repetitions is less than the determination threshold, the control unit 12 may deem that there is a high possibility that the pixels located behind the object 50 include unlit pixels, and return to the step of enlarging the lighting range 24 in step S12. When the number of repetitions is greater than the determination threshold (step S15: Yes), the control unit 12 deems that there is a low possibility that the pixels located behind the object 50 include unlit pixels, and ends the process. Fig.13 The lighting range 24 is determined by executing the steps of the flowchart.

[0077] The control unit 12 may determine whether the difference between the number of lit pixels when the lit pixels appear in the photographic image 60 and the number of lit pixels when the lit pixels do not appear in the photographic image 60 is less than a given value in the determination step of step S15. The control unit 12 may determine that the lit range 24 is the largest and the lit range 24 shown in the photographic image 60 is the smallest when the difference between the number of lit pixels when the lit pixels appear in the photographic image 60 and the number of lit pixels when the lit pixels do not appear in the photographic image 60 is less than a given value.

[0078] Summary

[0079] As described above, according to the data acquisition system 1, the data acquisition device 10, and the data acquisition method involved in the present embodiment, the number of lit pixels in the pixels located behind the object 50 is increased so that the lit pixels are not reflected in the photographic image 60. Thus, the lighting range 24 matching the shape of the object 50 is set. By generating mask data of the object 50 based on the set lighting range 24, the accuracy of the mask data can be improved. By generating mask data with high accuracy, it is unnecessary to manually correct the image of the object 50. As a result, the annotation can be simplified.

[0080] (Other Implementations)

[0081] Other embodiments will be described below.

[0082] <Example of generating mask data>

[0083] The control unit 12 may set data indicating that the object 50 exists at the position of each pixel included in the mask data as a mask portion. In addition, the control unit 12 may set data indicating that the object 50 does not exist at the position of each pixel included in the mask data as a transmission portion.

[0084] When the control unit 12 changes the state of a given pixel of the light-emitting panel 20 to a lit state or an unlit state, while the portion corresponding to the given pixel of the light-emitting panel 20 in the photographic image 60 does not change, the control unit 12 can set data indicating that an object 50 exists in a pixel of the mask data corresponding to the given pixel of the light-emitting panel 20.

[0085] When the state of a given pixel of the light-emitting panel 20 is changed to a lit state or an unlit state, the control unit 12 can set data indicating that the object 50 does not exist in the pixel of the mask data corresponding to the given pixel of the light-emitting panel 20 when a portion corresponding to the given pixel of the light-emitting panel 20 in the photographic image 60 changes.

[0086] This makes it possible to generate mask data with high accuracy.

[0087] The control unit 12 may execute calibration to associate the position of each pixel of a display device such as the light emitting panel 20 with the position of each pixel of the photographic image 60. This can improve the accuracy of the mask data.

[0088] The control unit 12 may change the lighting range 24 of the light emitting panel 20 in various modes in order to identify the pixel located behind the object 50. The control unit 12 may change the state of a given pixel by performing expansion processing or contraction processing based on the arrangement of pixels in the lighting state and the unlighted state of the light emitting panel 20.

[0089] In addition, the control unit 12 may collectively change the states of the plurality of pixels so as to set a given pixel of the light emitting panel 20 as the object of changing the state as described above. Fig.14 As shown in the example, the control unit 12 can control the state of each pixel of the light emitting panel 20 so that the lighting range 24 extends in a given direction such as the vertical direction or the horizontal direction. Fig.15 As shown, the control unit 12 may control the state of each pixel of the light emitting panel 20 to move the strip-shaped lighting range 24. In this case, each pixel of the light emitting panel 20 is lit or extinguished for each vertical or horizontal row.

[0090] The control unit 12 may determine the range of the lit pixels shown in the photographic image 60 each time the lighting range 24 of the light emitting panel 20 is changed. The control unit 12 may determine the lighting range 24 so that the number of lit pixels is maximized and the number of lit pixels shown in the photographic image 60 is minimized based on the range of the lit pixels shown in the photographic image 60 determined in each lighting range 24 after the change.

[0091] Thus, the influence of light emitted from pixels in adjacent or nearby rows can be reduced. As a result, the accuracy of determining whether a lit pixel is reflected in the photographic image 60 can be improved. The rows that are uniformly controlled to be lit or extinguished are not limited to vertical or horizontal lines, but can also be inclined lines. The number of rows that are uniformly controlled to be lit or extinguished can be one or more than two. In other words, the control unit 12 can uniformly change the state of each of a plurality of pixels arranged in at least one column as a given pixel of the light-emitting panel 20.

[0092] like Fig.16 As illustrated, the control unit 12 can divide the light emitting panel 20 into a plurality of sections and control the state of each pixel of the light emitting panel 20 so as to change the lighting or extinguishing pattern combining the sections. Fig.16In the example, the control unit 12 divides the light emitting panel 20 into six sections and sets a lighting range 24 or a non-lighting range 22 for each section. In other words, as a given pixel of the light emitting panel 20, the control unit 12 can collectively change the states of a plurality of pixels included in a given block.

[0093] exist Fig.16 In the table below the light emitting panel 20, the pattern of the combination of the states of each partition is represented as a combination of 0 or 1. The lighting range 24 corresponds to the cell recorded with 1. The extinguishing range 22 corresponds to the cell recorded with 0. Fig.16 The state of the light emitting panel 20 shown is represented by "001010" as shown in the first row of the table.

[0094] The control unit 12 may sequentially change the combination of states of the sections of the light emitting panel 20 as shown in the table. The control unit 12 may determine the range of lit pixels shown in the photographic image 60 in each combination of states of the light emitting panel 20. The control unit 12 may determine the lighting range 24 based on the range of lit pixels shown in the photographic image 60 determined in each combination so that the number of lit pixels is maximized and the number of lit pixels shown in the photographic image 60 is minimized.

[0095] Thus, the control unit 12 can determine whether or not there is an influence of light emitted from pixels in adjacent or nearby rows. As a result, it is possible to improve the accuracy of determining whether or not a lit pixel is reflected in the captured image 60.

[0096] In the above-mentioned embodiment, the control unit 12 generates mask data of the object 50 by controlling the state of each pixel to be either a lit state or an unlit state, so as to increase the number of lit pixels and reduce the number of lit pixels reflected in the photographic image 60. Conversely, the control unit 12 may control the state of each pixel to be either a lit state or an unlit state, so as to reduce the number of unlit pixels and increase the number of unlit pixels reflected in the photographic image 60. For example, in Fig.12 In step S3, the control unit 12 may determine the light-off range 22 based on the photographic image 60 so as to reduce the number of light-off pixels and increase the number of light-off pixels reflected in the photographic image 60, instead of determining the light-on range 24 based on the photographic image 60 so as to increase the number of light-on pixels and reduce the number of light-on pixels reflected in the photographic image 60.

[0097] The control unit 12 can determine whether the object 50 is reflected in each pixel of the photographic image 60 or the extinguished pixel is reflected, while controlling the state of each pixel based on the number of extinguished pixels. The control unit 12 can distinguish the object 50 from the extinguished pixel by, for example, image processing of the photographic image 60. The control unit 12 can also use a learned model for distinguishing the object 50 from the extinguished pixel. The control unit 12 can control the illumination of the object 50 by the illumination device 40 described later so that the difference between the brightness of the pixel reflecting the object 50 and the brightness of the pixel reflecting the extinguished pixel in the photographic image 60 becomes greater than a given value.

[0098] <Data Acquisition Station>

[0099] The data acquisition system 1 may include a data acquisition station for acquiring data. The data acquisition station may include a light emitting panel 20 and a plate for placing an object 50 on the light emitting surface of the light emitting panel 20. The plate for placing the object 50 is configured to transmit light emitted from the light emitting panel 20, and is also called a light-transmitting member. The light-transmitting member may be configured so that the object 50 does not directly contact the light emitting surface. The light-transmitting member may be arranged at a distance from the light emitting surface, or may be arranged to contact the light emitting surface.

[0100] The data acquisition station may further include a dark room for housing the light emitting panel 20 and the light-transmitting member. In addition, the data acquisition station may further include an illumination device 40 configured to illuminate the object 50 .

[0101] (Configuration Example of Robot Control System 100)

[0102] like Fig.17 As shown, a robot control system 100 according to one embodiment includes a robot 2 and a robot control device 110. In this embodiment, the robot 2 moves a work object 8 from a work start location 6 to a work target location 7. In other words, the robot control device 110 controls the robot 2 so that the work object 8 moves from the work start location 6 to the work target location 7. The work object 8 is also referred to as a work object. The robot control device 110 controls the robot 2 based on information related to a space in which the robot 2 performs work. Information related to a space is also referred to as space information.

[0103] <Robot control device 110>

[0104] The robot control device 110 acquires a learned model based on learning using the training data generated by the data acquisition device 10. The robot control device 110 recognizes the work object 8, the work start point 6, the work target point 7, etc., which exist in the space where the robot 2 performs work, based on the image photographed by the camera 4 and the learned model. In other words, the robot control device 110 acquires a learned model generated to recognize the work object 8, etc., based on the image photographed by the camera 4.

[0105] In order to provide control and processing capabilities for performing various functions, the robot control device 110 can be configured to include at least one processor. Each component of the robot control device 110 can also be configured to include at least one processor. Multiple components of each component of the robot control device 110 can also be implemented by one processor. The entire robot control device 110 can also be implemented by one processor. The processor is capable of executing programs that implement various functions of the robot control device 110. The processor can be implemented as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor can be implemented as an integrated circuit and a discrete circuit that can be connected in a communicative manner. The processor can be implemented based on various other known technologies.

[0106] The robot control device 110 may include a storage unit. The storage unit may include an electromagnetic storage medium such as a magnetic disk, or may include a memory such as a semiconductor memory or a magnetic memory. The storage unit stores various information and programs executed by the robot control device 110. The storage unit may also be configured as a non-temporary readable medium. The storage unit may function as a working memory of the robot control device 110. At least a portion of the storage unit may also be configured separately from the robot control device 110.

[0107] <Robot 2>

[0108] The robot 2 has an arm 2A and an end effector 2B. The arm 2A can be configured as a 6-axis or 7-axis vertical multi-joint robot, for example. The arm 2A can also be configured as a 3-axis or 4-axis horizontal multi-joint robot or a SCARA robot. The arm 2A can also be configured as a 2-axis or 3-axis orthogonal robot. The arm 2A can also be configured as a parallel linkage robot, etc. The number of axes constituting the arm 2A is not limited to the number illustrated. In other words, the robot 2 has an arm 2A connected by a plurality of joints, and moves by driving the joints.

[0109] The end effector 2B may include, for example, a gripping manipulator configured to grip the work object 8. The gripping manipulator may have a plurality of fingers. The number of fingers of the gripping manipulator may be two or more. The fingers of the gripping manipulator may have one or more joints. The end effector 2B may also include a suction manipulator configured to suck up the work object 8. The end effector 2B may also include a shoveling hand configured to scoop up the work object 8. The end effector 2B may also be configured to include tools such as a drill, and may perform various processing operations such as drilling holes in the work object 8. The end effector 2B is not limited to these examples, and may also be configured to be capable of performing various other actions. Fig.17 In the illustrated structure, the end effector 2B includes a gripping robot.

[0110] The robot control device 110 can control the position of the end effector 2B by moving the arm 2A of the robot 2. The end effector 2B may also have an axis that serves as a reference for the direction of the action on the work object 8. When the end effector 2B has an axis, the robot control device 110 can control the direction of the axis of the end effector 2B by moving the arm 2A of the robot 2. The robot control device 110 controls the start and end of the action of the end effector 2B on the work object 8. The robot control device 110 controls the position of the end effector 2B or the direction of the axis of the end effector 2B while controlling the action of the end effector 2B, thereby moving or processing the work object 8. Fig.17 In the illustrated structure, the robot control device 110 causes the end effector 2B to hold the work object 8 at the work start point 6 and moves the end effector 2B to the work target point 7. The robot control device 110 causes the end effector 2B to release the work object 8 at the work target point 7. In this way, the robot control device 110 can move the work object 8 from the work start point 6 to the work target point 7 through the robot 2.

[0111] <Sensor 3>

[0112] like Fig.17As shown, the robot control system 100 is further provided with a sensor 3. The sensor 3 detects physical information of the robot 2. The physical information of the robot 2 may include information related to the actual position or posture of each component of the robot 2, or the speed or acceleration of each component of the robot 2. The physical information of the robot 2 may include information related to the force acting on each structural part of the robot 2. The physical information of the robot 2 may include information related to the current flowing in the motor driving each component of the robot 2 or the torque of the motor. The physical information of the robot 2 represents the result of the actual action of the robot 2. In other words, the robot control system 100 can grasp the result of the actual action of the robot 2 by obtaining the physical information of the robot 2.

[0113] The sensor 3 may include a force sensor or a tactile sensor that detects a force, a distributed pressure, or a sliding motion, etc., acting on the robot 2 as physical information of the robot 2. The sensor 3 may include a motion sensor that detects a position or a posture, or a speed or an acceleration of the robot 2 as physical information of the robot 2. The sensor 3 may include a current sensor that detects a current flowing in a motor that drives the robot 2 as physical information of the robot 2. The sensor 3 may include a torque sensor that detects a torque of a motor that drives the robot 2 as physical information of the robot 2.

[0114] The sensor 3 may be provided at a joint of the robot 2 or a joint driving unit that drives the joint. The sensor 3 may also be provided at the arm 2A or the end effector 2B of the robot 2.

[0115] The sensor 3 outputs the detected physical information of the robot 2 to the robot control device 110. The sensor 3 detects and outputs the physical information of the robot 2 at a given timing. The sensor 3 outputs the physical information of the robot 2 as time-series data.

[0116] <Camera 4>

[0117] exist Fig.17In the structural example shown, the robot control system 100 has two cameras 4. The camera 4 photographs objects or people located in the influence range 5 that may affect the movement of the robot 2. The image photographed by the camera 4 may include monochrome brightness information or brightness information of each color represented by RGB or the like. The influence range 5 includes the movement range of the robot 2. The influence range 5 is a range that further extends the movement range of the robot 2 to the outside. The influence range 5 can be set to be able to stop the robot 2 before a person moving from the outside of the movement range of the robot 2 to the inside of the movement range enters the inside of the movement range of the robot 2. The influence range 5 can also be set, for example, to a range extending from a given distance from the boundary of the movement range of the robot 2 to the outside. The camera 4 can also be set to be able to photograph the influence range 5 or the movement range of the robot 2 or the area around them from a bird's-eye view. The number of cameras 4 is not limited to two, and can be one or more than three.

[0118] (Operation Example of Robot Control System 100)

[0119] The robot control device 110 obtains the learned model in advance. The robot control device 110 may store the learned model in the storage unit. The robot control device 110 obtains an image of the work object 8 from the camera 4. The robot control device 110 inputs the image of the work object 8 as input information to the learned model. The robot control device 110 obtains output information output according to the input information from the learned model. The robot control device 110 recognizes the work object 8 based on the output information and performs the operation of holding or moving the work object 8.

[0120] Summary

[0121] The robot control system 100 can acquire a learned model based on learning using the training data generated by the data acquisition system 1 , and recognize the work object 8 using the learned model.

[0122] The above describes the implementation of the data acquisition system 1 and the robot control system 100, but as an implementation of the present disclosure, in addition to the method or program for implementing the system or device, it is also possible to adopt an implementation as a storage medium with a program recorded thereon (as an example, a CD, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card, etc.).

[0123] In addition, the installation method of the program is not limited to application programs such as object codes compiled by a compiler and program codes executed by an interpreter, and may also be a program module compiled into an operating system. Furthermore, the program may be configured so that all processing is performed only in the CPU on the control substrate. The program may also be configured so that part or all of it is performed by other processing units installed in an expansion board or expansion unit attached to the substrate as needed.

[0124] Although the embodiments of the present disclosure have been described based on the drawings and embodiments, it should be noted that those skilled in the art may make various deformations or modifications based on the present disclosure. Therefore, it should be noted that these deformations or changes are included in the scope of the present disclosure. For example, the functions included in each component, etc. can be reconfigured to be logically consistent, and multiple components, etc. can be combined into one or divided.

[0125] All structural elements and / or all methods or steps of processing disclosed in the present disclosure can be combined in any combination except for the mutually exclusive combination of these features. In addition, the features recorded in the present disclosure can be replaced with the same purpose, equivalent purpose, or alternative features that work for similar purposes as long as they are not explicitly denied. Therefore, unless explicitly denied, each feature disclosed is only an example of a series of identical or equal features included.

[0126] Furthermore, the embodiments of the present disclosure are not limited to any specific configuration in the above-mentioned embodiments. The embodiments of the present disclosure can be extended to all new features or their combinations, or all new methods or processing steps or their combinations recorded in the present disclosure.

[0127] In one embodiment, (1) a data acquisition device includes a control unit configured to control a display device having a plurality of pixels controlled to be in one of a lit state and an unlit state, and configured to acquire a photographic image obtained by photographing an object located in front of the display device and the display device. The control unit controls the state of each pixel to be in one of the lit state and the unlit state so that the number of pixels in the lit state is increased and the number of pixels in the lit state reflected in the photographic image is decreased, or the number of pixels in the unlit state is decreased and the number of pixels in the unlit state reflected in the photographic image is increased, and mask data of the object is generated based on the arrangement of the pixels in the lit state.

[0128] (2) In the data acquisition device of the above-mentioned (1), the control unit can generate mask data of the object based on the configuration of the pixels in the lit state when the number of pixels in the lit state is maximum and the number of pixels in the lit state reflected in the photographic image is minimum, or the configuration of the pixels in the unlit state when the number of pixels in the unlit state is minimum and the number of pixels in the unlit state reflected in the photographic image is maximum.

[0129] (3) In the data acquisition device of (1) or (2) above, the control unit can change the state of a given pixel, and when the portion of the photographic image corresponding to the given pixel does not change, set data indicating that the object exists in the portion of the mask data corresponding to the given pixel, and when the portion of the photographic image corresponding to the given pixel changes, set data indicating that the object does not exist in the portion of the mask data corresponding to the given pixel.

[0130] (4) In the data acquisition device of (3) above, the control unit may collectively change the states of a plurality of pixels as the predetermined pixel.

[0131] (5) In the data acquisition device of (4) above, the control unit may collectively change the states of a plurality of pixels arranged in at least one row as the predetermined pixel.

[0132] (6) In the data acquisition device of (4) above, the control unit may collectively change the states of a plurality of pixels included in a predetermined block as the predetermined pixel.

[0133] (7) In the data acquisition device described in any one of (3) to (6), the control unit may change the state of the given pixel by performing expansion processing or contraction processing based on the arrangement of the pixels in the lighting state and the unlighting state.

[0134] (8) In the data acquisition device according to any one of (1) to (7) above, the control unit may execute calibration to associate the position of each pixel of the display device with the position of each pixel of the photographic image.

[0135] (9) In the data acquisition device described in any one of (1) to (8) above, the control unit may extract image data of the object from an image obtained by photographing the object at the same position as when the photographic image was photographed, based on mask data of the object.

[0136] (10) In the data acquisition device described in (9) above, the control unit may control the illumination light that illuminates the object.

[0137] In one embodiment, (11) the data acquisition method includes the following processing: in a display device having a plurality of pixels that are controlled to be in a state either of a lit state and an unlit state, controlling the state of each pixel to be in a state either of the lit state and the unlit state, so that the number of pixels in the lit state is increased and the number of pixels in the lit state reflected in a photographic image is decreased, or the number of pixels in the unlit state is decreased and the number of pixels in the unlit state reflected in the photographic image is increased, wherein the photographic image is obtained by photographing an object located in front of the display device and the display device; and generating mask data of the object based on the configuration of the pixels in the lit state.

[0138] (12) The data acquisition method of (11) above may further include extracting image data of the object from an image obtained by photographing the object at the same position as when the photographed image was photographed, based on mask data of the object.

[0139] In one embodiment, (13) a data acquisition station includes: a display device having a plurality of pixels; and a light-transmitting member located between an object disposed in front of the display device and the display device.

[0140] (14) The data acquisition station according to (13) above may further include a lighting device configured to illuminate the object.

[0141] -Explanation of symbols-

[0142] 1 Data acquisition system

[0143] 10 Data acquisition device (12: control unit, 14: storage unit, 16: interface)

[0144] 20 Luminous panel (22: off range, 24: on range)

[0145] 30 Photographic Installation

[0146] 40 lighting fixtures

[0147] 50 Objects

[0148] 60 Photographic image (62: object image, 64: extracted image)

[0149] 70 Mask image (72: mask part, 74: transparent part)

[0150] 80 Composite image (82: Background image)

[0151] 100 Robot control system (2: robot, 2A: arm, 2B: end effector, 3: sensor, 4: camera, 5: influence range, 6: work start point, 7: work target point, 8: work object, 110: robot control device).

Claims

1. A data acquisition device, comprising a control unit configured to control a display device having a plurality of pixels, and configured to acquire a photographic image obtained by photographing an object located in front of the display device and the display device, wherein the plurality of pixels are controlled to be in one of a lighting state and an unlighting state, The control unit controls the state of each pixel to be one of the lit state and the unlit state, so that the number of pixels in the lit state is increased and the number of pixels in the lit state reflected in the photographic image is reduced, or the number of pixels in the unlit state is reduced and the number of pixels in the unlit state reflected in the photographic image is increased, The control unit generates mask data of the object based on the arrangement of the pixels in the lighting state.

2. The data acquisition device according to claim 1, wherein: The control unit generates mask data of the object based on the configuration of the pixels in the lit state when the number of pixels in the lit state is maximum and the number of pixels in the lit state reflected in the photographic image is minimum, or the configuration of the pixels in the unlit state when the number of pixels in the unlit state is minimum and the number of pixels in the unlit state reflected in the photographic image is maximum.

3. The data acquisition device according to claim 1 or 2, wherein: The control unit: Change the state of a given pixel, When the portion corresponding to the given pixel in the photographic image does not change, data indicating that the object exists in the portion corresponding to the given pixel in the mask data is set; When a portion corresponding to the predetermined pixel in the photographic image changes, data indicating that the object does not exist in the portion corresponding to the predetermined pixel in the mask data is set.

4. The data acquisition device according to claim 3, wherein: The control unit collectively changes the states of each of a plurality of pixels as the predetermined pixel.

5. The data acquisition device according to claim 4, wherein: The control unit collectively changes the states of each of a plurality of pixels arranged in at least one column as the predetermined pixel.

6. The data acquisition device according to claim 4, wherein: The control unit collectively changes the states of each of a plurality of pixels included in a given block as the given pixel.

7. The data acquisition device according to claim 3, wherein: The control unit changes the state of the given pixel by performing an expansion process or a contraction process based on the arrangement of the pixels in the lighting state and the extinguishing state.

8. The data acquisition device according to claim 1 or 2, wherein: The control unit executes calibration to associate a position of each pixel of the display device with a position of each pixel of the photographic image.

9. The data acquisition device according to claim 1 or 2, wherein: The control unit extracts image data of the object from an image obtained by photographing the object at the same position as when the photographed image was photographed, based on mask data of the object.

10. The data acquisition device according to claim 9, wherein: The control unit controls illumination light that illuminates the object.

11. A data acquisition method, comprising the following processing: In a display device having a plurality of pixels that are controlled to be in either a lit state or an unlit state, the state of each pixel is controlled to be in either the lit state or the unlit state so that the number of pixels in the lit state is increased and the number of pixels in the lit state that appear in a photographic image is decreased, or the number of pixels in the unlit state is decreased and the number of pixels in the unlit state that appear in the photographic image is increased, the photographic image being obtained by photographing an object located in front of the display device and the display device; and Mask data of the object is generated based on the arrangement of the pixels in the lighting state.

12. The data acquisition method according to claim 11, wherein: The method further includes extracting image data of the object from an image obtained by photographing the object at the same position as when the photographed image was photographed, based on mask data of the object.

13. A data acquisition station, comprising: A display device having a plurality of pixels; and The light-transmitting member is located between an object arranged in front of the display device and the display device.

14. The data acquisition station according to claim 13, wherein: The data acquisition station further includes a lighting device configured to illuminate the object.

Images