System, information processing device and program

Abstract

To easily generate an image capturing no background for an object.SOLUTION: A system of an embodiment includes: first and second light sources that irradiate an object with two kinds of light having different color main components from different directions, respectively; an imaging device that images an area including the object from a direction different from an irradiation direction of each light from the first and second light sources; and an information processing device that generates an image capturing no background of the object on the basis of the captured image obtained from the imaging device. The information processing device extracts a first image having a first color component as a main component and a second image having a second color component as a main component from the captured image, and detects a boundary between the object and the background of the object from an image obtained by adding a part of a binarized image of the first image and a part of a binarized image of the second image to generate the image capturing no background of the object.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a system, an information processing device, and a program. [Background technology]

[0002] For example, there is a system known that judges the movement of an object, such as a human hand, from a photographed image of the object (e.g., whether the hand has been washed properly). For this movement judgment, a model created by previously learning the classification of movements using deep learning or the like is sometimes used. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2016 / 194939 [Patent Document 2] Japanese Patent Application Publication No. 2019-207451 Summary of the Invention [Problem to be solved by the invention]

[0004] When using a model trained on training images that include not only an object such as a human hand, but also the entire image including the sink in the bathroom that is reflected in the background, the judgment is also made based on the entire image including the sink in the background. In this case, even if the hand movements appear to the human eye to be the same, they may be judged as different movements due to the influence of the shape or color of the sink in the background.

[0005] For this reason, it is desirable to be able to distinguish between the image portion of the object and the image portion of the background, and obtain an image in which the background of the object is not visible.

[0006] However, objects such as sinks in the background often reflect light specularly, like a mirror, and have a wide variety of distances and colors. Furthermore, the color of objects such as human hands varies greatly from person to person, and the color of the sleeves of clothing that appear along with the hands also varies widely. Therefore, it is not easy to distinguish between the image of the object and the image of the background and obtain an image in which the background of the object is not visible.

[0007] The problem to be solved by the invention is to provide a system, an information processing device, and a program that can easily generate an image in which the background of an object is not visible. [Means for solving the problem]

[0008] A system according to an embodiment includes first and second light sources that irradiate an object with two types of light having different primary color components from different directions, an imaging device that captures an image of an area including the object from a direction different from the irradiation directions of the light from the first and second light sources, and an information processing device that generates an image that does not show the background of the object based on the captured image obtained from the imaging device. The information processing device extracts a first image having a first color component as a primary component and a second image having a second color component as a primary component from the captured image, and generates an image that does not show the background of the object by detecting a boundary between the object and its background from an image obtained by combining a portion of an image obtained by binarizing the first image and a portion of an image obtained by binarizing the second image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram schematically illustrating an example of the configuration of a motion determining system 1 according to an embodiment. [Figure 2] FIG. 2 is a block diagram mainly showing the configuration of a movement determining device 10 included in the movement determining system 1. As shown in FIG. [Figure 3] FIG. 3 is a block diagram showing an example of a configuration of functions realized by the processor 11 in FIG. 2 executing a program. [Figure 4]FIG. 4 is a conceptual diagram showing the range photographed by the camera 3. As shown in FIG. [Figure 5] FIG. 5 is a conceptual diagram showing the relationship between photography by the camera 3 and illumination by the lights 22 and 23. In FIG. [Figure 6] FIG. 6 is a conceptual diagram showing the relationship between an object, which is a simplified representation of both hands 40 of a user, and the photographing by the camera 3 and the illumination by the lights 22 and 23. In FIG. [Figure 7] FIG. 7 is a flowchart showing an example of the operation of the image acquisition unit 31 and the image processing unit 32 of the control unit 30. [Figure 8] FIG. 8 is a diagram showing an example of an acquired color image (Ic). [Figure 9] FIG. 9 is a diagram showing an example of a color separation image (Ig) having a "G (green)" (G) primary component corresponding to the illumination 22. In FIG. [Figure 10] FIG. 10 is a diagram showing an example of a color separation image (Ir) having "R (red color)" as the main component corresponding to the illumination 23. [Figure 11] FIG. 11 is a diagram showing an example of luminance information (Ig2) obtained by binarizing the image (Ig) in FIG. [Figure 12] FIG. 12 is a diagram showing an example of luminance information (Ir2) obtained by binarizing the image (Ir) in FIG. [Figure 13] FIG. 13 is a diagram showing an example of an image in which the edge portions are displayed in white. [Figure 14] FIG. 14 is a diagram showing an example of an image in which the area where there was a background is displayed in black. [Figure 15] FIG. 15 is a flowchart showing an example of the operation performed by the operation determining unit 33 of the control unit 30. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described with reference to the drawings.

[0011] The system according to this embodiment includes an action determination system that determines whether a user has washed their hands properly in accordance with a predetermined procedure.

[0012] The motion determination system photographs both hands of a user as subjects (objects) at a sink or other location in a washing area, and determines the motion class of the user's hands based on the photographed image. For example, the motion determination system determines the pass / fail of each hand motion (e.g., lathering well, massaging the back of the hand, massaging between the fingers, etc.) classified according to the area being washed (e.g., the back of the hand, between the fingers, thumb, etc.) as a motion class. The motion determination system also determines whether the user washed their hands properly according to a predetermined procedure based on the order of the motion classes, the duration of the motion, etc.

[0013] FIG. 1 is a diagram schematically illustrating an example of the configuration of a motion determining system 1 according to an embodiment.

[0014] As shown in FIG. 1, the motion determination system 1 includes a support member 2, a camera 3 (imaging device), a distance sensor 4, an input / output device 5, a sink 20, and lights 22 and 23 (first and second light sources).

[0015] The sink 20 is a water tank where a user (not shown) washes their hands. The sink 20 has a recessed structure, and a drain groove is formed at the bottom. The sink 20 is also provided with a faucet 21. The faucet 21 discharges water for the user to wash their hands. The discharged water is discharged from the drain groove of the sink 20.

[0016] The support member 2 is installed above the sink 20 and supports various devices such as the camera 3, the input / output device 5, and the lights 22 and 23.

[0017] Camera 3 is installed facing downward above sink 20 and captures from above an area including the user's hands and the background illuminated by lights 22 and 23. That is, camera 3 captures an area including the user's hands and the background from a direction different from the illumination direction of the lights 22 and 23 as a two-dimensional image (captured image), and outputs color information (RGB (Red Green Blue)) for each pixel.

[0018] The lights 22 and 23 each emit light onto the user's hands. The lights 22 and 23 each emit two types of light with different main color components (main color elements) from different directions onto the user's hands (not shown). The reason for using two different types of light is to make it easier to confirm that the user's hands are within the range of both lights.

[0019] The main component of the color of the light emitted by the lighting 22 (first light source) and the main component of the color of the light emitted by the lighting 23 (second light source) each correspond to one of the three primary colors of light: red, green, or blue. In this example, the main component of the color of the light emitted by the lighting 22 is "G (green)," and the main component of the color of the light emitted by the lighting 23 is "R (red)." However, the combination of the two main color components is not limited to this example. The greater the wavelength difference between the two main color components, the easier they are to distinguish. In that sense, the combination of "B (blue)" and "R (red)" is the most desirable.

[0020] The distance sensor 4 is a sensor that measures the distance to the user's hand and outputs a signal indicating the measured distance to the outside.

[0021] The input / output device 5 is installed in a position visible to the user. In this example, the input / output device 5 is installed in front of the user washing their hands at the sink 20.

[0022] The input / output device 5 is an interface that receives instructions from an operator and displays various information to the operator. The input / output device 5 is made up of an operation unit that receives instructions and a display unit that displays information.

[0023] The input / output device 5 transmits a signal indicating an operation received from an operator to a later-described operation determining device 10 (information processing device) as the operation of the operation unit. For example, the operation unit is configured with a touch panel. The operation unit may also include a keyboard or a numeric keypad.

[0024] The input / output device 5 operates as a display unit to display an image from the motion determining device 10. For example, the display unit is configured with a liquid crystal monitor. The display unit may be formed integrally with a touch panel serving as an operation unit.

[0025] FIG. 2 is a block diagram mainly showing the configuration of a movement determining device 10 included in the movement determining system 1. As shown in FIG.

[0026] The movement determination device 10 corresponds to an information processing device (computer) such as a desktop PC, a notebook PC, a tablet PC, a smartphone, etc. The movement determination device 10 includes a processor 11, a ROM 12, a RAM 13, an NVM 14, a camera interface 15, a distance sensor interface 16, an input / output interface 17, a communication interface 18, a lighting interface 19, etc.

[0027] The processor 11 is communicatively connected to the ROM 12, RAM 13, NVM 14, camera interface 15, distance sensor interface 16, input / output interface 17, communication interface 18, and lighting interface 19.

[0028] The camera interface 15 is communicatively connected to the camera 3. The distance sensor interface 16 is communicatively connected to the distance sensor 4. The input / output interface 17 is communicatively connected to the input / output device 5. The lighting interface 19 is communicatively connected to the lighting devices 22 and 23.

[0029] The processor 11 controls the overall operation of the operation determination device 10. The processor 11 may include an internal cache and various interfaces. In this case, the processor 11 executes programs stored in advance in the internal cache, the ROM 12, or the NVM 14 to realize various processes.

[0030] Some of the various functions realized by the processor 11 executing the programs may be realized by hardware circuits. In this case, the processor 11 controls the functions executed by the hardware circuits.

[0031] The ROM 12 is a non-volatile memory in which control programs, control data, etc. are stored in advance. The control programs and control data stored in the ROM 12 are pre-installed in accordance with the specifications of the movement determination device 10. The ROM 12 stores, for example, a program for controlling a circuit board of the movement determination device 10.

[0032] RAM 13 is a volatile memory. RAM 13 temporarily stores data being processed by processor 11. RAM 13 stores various application programs based on instructions from processor 11. RAM 13 may also store data necessary for executing application programs and execution results of application programs.

[0033] The NVM 14 is a nonvolatile memory to which data can be written and rewritten. The NVM 14 is configured, for example, by a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The NVM 14 stores control programs, applications, various data, and the like according to the operational use of the operation determination device 10.

[0034] Camera interface 15 is an interface for transmitting and receiving data to and from camera 3. For example, camera interface 15 transmits a signal instructing camera 3 to take a photograph under the control of processor 11. Camera interface 15 also acquires a photographed image from camera 3. For example, camera interface 15 may be one that supports a USB (Universal Serial Bus) connection, or one that supports a connection via Camera Link.

[0035] The distance sensor interface 16 is an interface for receiving a signal indicating the distance measured by the distance sensor 4 .

[0036] The input / output interface 17 is an interface for transmitting and receiving data to and from the input / output device 5. For example, the input / output interface 17 receives a signal indicating an operation accepted from an operator from the input / output device 5. The input / output interface 17 transmits the received signal to the processor 11. Furthermore, the input / output interface 17 transmits information indicating a screen to be displayed to the operator to the input / output device 5 under the control of the processor 11. For example, the input / output interface 17 may be one that supports a USB connection or one that supports a connection via a parallel interface.

[0037] The communication interface 18 is an interface for transmitting and receiving data to and from an external device. The communication interface 18 is connected to the external device via a network or the like. For example, the communication interface 18 supports a wired or wireless LAN (Local Area Network) connection.

[0038] The lighting interface 19 is an interface for supplying power to each of the lights 22 and 23 and for supplying a signal for controlling the on / off of the lights.

[0039] FIG. 3 is a block diagram showing an example of a configuration of functions realized by the processor 11 in FIG. 2 executing a program.

[0040] The control unit 30 shown in FIG. 3 is a function realized by the processor 11 executing a program, and includes an image acquisition unit 31, an image processing unit 32, and an operation determination unit 33.

[0041] The image acquisition unit 31 has a function of acquiring a captured image of an area including the object and its background from a direction different from the irradiation direction of the light from the first and second light sources, while the object is irradiated with two types of light having different main color components from different directions separately by the lighting 22 and 23.

[0042] The image processing unit 32 has a function of generating an image in which the background of the object (here, both hands of the user) is not reflected, based on the captured image acquired by the image acquisition unit 31. Specifically, the image processing unit 32 extracts, from the captured image, a first image having a first color component (here, "G (green)") as a main component and a second image having a second color component (here, "R (red)") as a main component, and generates an image in which the background of the object is not reflected by detecting a boundary (hereinafter, sometimes referred to as an "edge portion") between the object and its background from an image obtained by combining a part of an image obtained by binarizing the first image and a part of an image obtained by binarizing the second image. A more specific example of the processing will be described in detail later.

[0043] The action determination unit 33 has a function to determine whether the action of the object is appropriate based on the image generated by the image processing unit 32 (here, to determine whether the user's hand washing was performed appropriately in accordance with the specified procedure).

[0044] FIG. 4 is a conceptual diagram showing the range photographed by the camera 3. As shown in FIG.

[0045] As described above, camera 3 captures an image of an area including both hands 40 of the user from above. At this time, it is assumed that camera 3 is set so that both hands 40 of the user are within range 301 captured by camera 3. This makes it possible to capture the user's actions when washing both hands 40 with detergent or the like from the image captured by camera 3.

[0046] FIG. 5 is a conceptual diagram showing the relationship between photography by the camera 3 and illumination by the lights 22 and 23. In FIG.

[0047] Camera 3 captures an image of imaging range 35 including both hands 40 of the user, i.e., right hand 40R and left hand 40L, from above. In response, lighting devices 22 and 23 emit light obliquely from above toward both hands 40 of the user, positioned away from camera 3 in opposite directions. Light device 22 emits light 221 primarily composed of "G (green)," and lighting device 23 emits light 231 primarily composed of "R (red)." In this case, the image captured by camera 3 is characterized in that light 221 tends to create a particularly large difference in luminance between the back of the right hand 40R and the background (the back of the right hand 40R appears particularly bright against a dark background), and light 231 tends to create a particularly large difference in luminance between the back of the left hand 40L and the background (the back of the left hand 40L appears particularly bright against a dark background).

[0048] FIG. 6 is a conceptual diagram showing the relationship between an object, which is a simplified representation of both hands 40 of a user, and the photographing by the camera 3 and the illumination by the lights 22 and 23. In FIG.

[0049] FIG. 6 shows an object 401 that is a simplified representation of both hands 40 of a user.

[0050] A portion of the light 221 emitted by the lighting 22 hits the object 401 at an angle and forms a projection 222. Similarly, a portion of the light 231 emitted by the lighting 23 hits the object 401 and forms a projection 232. In this state, the camera 3 captures an image of the object 401 and its background within the range 301.

[0051] Next, an example of the operation of the image acquisition unit 31 and image processing unit 32 of the control unit 30 will be described with reference to the flowchart of Fig. 7. Here, images shown in Figs. 8 to 14 will also be referred to as appropriate.

[0052] In ACT1, the image acquisition unit 31 acquires a captured image (color image) of an area including both hands 40 and their background, with the camera 3, while the lights 221, 231 are irradiated onto both hands 40 of the user from the lights 22, 23.

[0053] An example of the acquired color image (Ic) is shown in Fig. 8. Note that a center line 50 is obtained on the color image (Ic) by processing described later, as shown in Fig. 8. The center line 50 will be described later.

[0054] In ACT2, the image processing unit 32 generates color separation images (Ig, Ir) corresponding to the illuminations 22 and 23, respectively, from the color image (Ic) in Fig. 8. That is, a color separation image (Ig) of the same color as the light of the illumination 22 and a color separation image (Ir) of the same color as the light of the illumination 23 are generated.

[0055] FIG. 9 shows an example of a color separation image (Ig) with a main component of "G (green)" corresponding to the illumination 22. Although no color can be seen in FIG. 9, the image obtained actually has a green tint. In image (Ig), there is a particularly large difference in brightness between the back of the right hand 40R and the background (the back of the right hand 40R is particularly bright compared to the background).

[0056] FIG. 10 shows an example of a color separation image (Ir) with a main component of "R (red)" corresponding to the illumination 23. Although no color can be seen in FIG. 10, a reddish image is actually obtained. In the image (Ir), there is a particularly large difference in brightness between the back of the left hand 40L and the background (the back of the left hand 40L is particularly bright compared to the background).

[0057] In ACT3, the image processing unit 32 binarizes the image (Ig) in FIG. 9 and the image (Ir) in FIG. 10, respectively, to extract luminance information (Ig2, Ir2).

[0058] Fig. 11 shows an example of luminance information (Ig2) obtained by binarizing the image (Ig) in Fig. 9. Fig. 12 shows an example of luminance information (Ir2) obtained by binarizing the image (Ir) in Fig. 10.

[0059] In ACT4, the image processing unit 32 adds up the luminance information (Ig2) of Fig. 11 and the luminance information (Ir2) of Fig. 12 according to the position of the image, and detects pixel portions showing a high luminance value (for example, the maximum value) equal to or higher than a certain level, thereby determining the center line 50. The center line 50 is placed on each of the image (Ig) of Fig. 9 and the image (Ir) of Fig. 10, similar to the way it is placed on the image (Ic) of Fig. 8 described above.

[0060] In ACT 5, the image processing unit 32 generates an image by combining the priority images on the left and right sides of the center line 50, i.e., the left half of the image (Ig) from the center line 50 in FIG. 9 and the right half of the image (Ir) from the center line 50 in FIG. 10. Next, in ACT 6, the image processing unit 32 determines a threshold value corresponding to the luminance change from the combined image. Furthermore, in ACT 7, the image processing unit 32 detects edge portions based on the determined threshold value. An edge portion is a portion where there is a luminance change (differential value) between consecutive pixels that is greater than or equal to the threshold value.

[0061] An example of an image in which the edge portions are represented in white is shown in Fig. 13. As shown in Fig. 13, the white edge portions make it possible to confirm the boundary between the back of the right hand 40R and the background, and the boundary between the back of the right hand 40R and the background.

[0062] In ACT8, the image processing unit 32 extracts the inside of the edge portion (the area of ​​the right hand 40R and the area of ​​the right hand 40R) from the background from the color image (Ic) in FIG. 8 described above, based on the detection result of the edge portion.

[0063] In ACT 9, the image processing unit 32 generates an input image to be used for motion determination based on the extracted region of the right hand 40R and the image showing the region of the right hand 40R. At this time, the region that had a background is displayed in black, for example.

[0064] An example of an image in which the area where there was a background is shown in black is shown in Figure 14. In this way, by adopting the extracted area of ​​right hand 40R and an image that correctly shows the area of ​​right hand 40R as the input image used for motion determination, the accuracy of motion determination can be improved.

[0065] Next, an example of the operation performed by the operation determining unit 33 of the control unit 30 will be described with reference to FIG.

[0066] In ACT11, the operation determination unit 33 starts a predetermined timer for measuring time.

[0067] In ACT12, the operation determination unit 33 reads or calculates a threshold value (pass threshold value) for determining pass / fail.

[0068] In ACT13, the operation determination unit 33 acquires the image generated by the operation of FIG. 7 described above.

[0069] In ACT 14, the action determination unit 33 estimates the action of both hands of the user based on the acquired image. For example, if there is any action that corresponds to the action of lathering well, the action of massaging the back of the hand, or the action of massaging between the fingers, it is estimated that the action has been performed.

[0070] In ACT15, the movement determination unit 33 adds 1 to the number of times of the estimated movement.

[0071] In ACT16, the operation determination unit 33 determines whether the value (T) indicated by the timer is equal to a predetermined value (T N ) is judged whether the value (T) indicated by the timer exceeds a predetermined value (T N If the value (T) indicated by the timer does not exceed the predetermined value (T N ) (Yes in ACT16), proceed to ACT17.

[0072] In ACT 17, the motion determination unit 33 performs motion determination based on the results of multiple motion estimations and records the determination results. For example, if it is estimated that the motions of lathering well, massaging the back of the hand, and massaging between the fingers were all performed appropriately in this order, it determines that the motion has passed, and if not, it determines that the motion has failed. The determination results are recorded in a specified storage medium.

[0073] (Summary of the embodiment) As described above in detail, according to the embodiment, an image in which the background of the object is not reflected can be easily generated. For example, even if a sink or other object reflected in the background of an object (such as a user's hand) in a captured image is made of a material that reflects light specularly like a mirror, has various distances and colors, or even if the color of hands varies from person to person, the image portion of the object and the image portion of the background can be accurately distinguished, and an image in which the background of the object is not reflected can be easily generated. Furthermore, by using the image generated in this way to create a model used to determine the object's motion using deep learning or to perform object motion determination, the accuracy of motion determination can be improved.

[0074] The above-mentioned program may be transferred in a state where it is stored in a device, or in a state where it is not stored in a device. In the latter case, the program may be transferred via a network, or in a state where it is recorded on a recording medium. The recording medium is a non-transitory tangible medium. The recording medium is a computer-readable medium. The recording medium may be in any form, such as a CD-ROM or a memory card, as long as it is capable of storing the program and is computer-readable.

[0075] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0076] 1...Motion determination system, 3...Camera (imaging device), 4...Distance sensor, 5...Input / output device, 10...Motion determination device (information processing device), 11...Processor, 12...ROM, 13...RAM, 14...NVM, 15...Camera interface, 16...Distance sensor interface, 17...Input / output interface, 18...Communication interface, 19...Lighting interface, 20...Sink, 21...Faucet, 22, 23...Lighting (light source), 30...Control unit, 31...Image acquisition unit, 32...Image processing unit, 33...Motion determination unit, 35...Imaging range, 40...Both hands, 40L...Left hand, 40R...Right hand

Claims

1. a first light source and a second light source that respectively irradiate an object with two types of light having different main color components from different directions; an imaging device that captures an image of an area including the object from a direction different from the irradiation direction of the light from the first and second light sources; an information processing device that generates an image in which the background of the object is not captured based on the captured image obtained from the imaging device; Equipped with The information processing device includes: a first image having a first color component as a main component and a second image having a second color component as a main component are extracted from the captured image, and a boundary between the object and its background is detected from an image obtained by combining a portion of an image obtained by binarizing the first image and a portion of an image obtained by binarizing the second image, thereby generating an image in which the background of the object is not captured; system.

2. The first color component and the second color component each correspond to one of red, green, and blue, which are the three primary colors of light. The system of claim 1 .

3. The object is a person's hands. The system of claim 1 .

4. The information processing device further comprises: determining whether the object's movement is appropriate based on the generated image; 4. A system according to any one of claims 1 to 3.

5. an image acquisition unit that acquires a captured image of an area including the object from a direction different from the irradiation directions of the light from a first light source and a second light source, while irradiating the object with two types of light having different main color components from different directions, using an imaging device; an image processing unit that generates an image that does not show a background of the object based on the photographed image acquired by the image acquisition unit; Equipped with The image processing unit a first image having a first color component as a main component and a second image having a second color component as a main component are extracted from the captured image, and a boundary between the object and its background is detected from an image obtained by combining a portion of an image obtained by binarizing the first image and a portion of an image obtained by binarizing the second image, thereby generating an image in which the background of the object is not captured; Information processing device.

6. On the computer, an image acquisition function that acquires a captured image of an area including the object from a direction different from the irradiation direction of the light from the first and second light sources, while irradiating the object with two types of light having different main color components from different directions, using an imaging device; an image processing function for generating an image in which the background of the object is not captured based on the photographed image acquired by the image acquisition function; A program for realizing the above, The image processing function includes: a first image having a first color component as a main component and a second image having a second color component as a main component are extracted from the captured image, and a boundary between the object and its background is detected from an image obtained by combining a portion of an image obtained by binarizing the first image and a portion of an image obtained by binarizing the second image, thereby generating an image in which the background of the object is not captured; program.

Images