Object identification system and object identification method

Abstract

To appropriately identify a target object on the basis of an image captured by an imaging unit worn by a worker.SOLUTION: An object identification system 10 that identifies an object pointed at by a worker on the basis of an image captured by a camera 11 worn by a worker includes a calculation unit 21 that calculates a positional relationship between the worker's eyes and the camera on the basis of a reference image captured by the camera while the worker is directly facing a reference object provided at a predetermined distance, a creation unit 22 that creates an environmental map of the space captured by the camera on the basis of the spatial image captured by the camera, a setting unit 23 that sets a line of sight plane that includes the worker's eyes and fingertips and the target object on the environmental map on the basis of the positional relationship, an estimation unit 24 that estimates the worker's line of sight on the line of sight plane on the basis of the coordinate difference between the center coordinates on the captured image Pi of the fingertip and the object captured by the camera and the coordinates of the fingertip with the worker pointing at the object, and an identification unit 25 that identifies the object pointed at by the worker on the basis of the line of sight.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an object identification system and an object identification method.

Background Art

[0002] In maintenance and inspection work of equipment such as servers, an operator may point at and name an object. In pointing and naming, the operator extends an arm forward, points a fingertip at the object, and shouts.

[0003] Patent Document 1 discloses an information processing apparatus that captures an object and a fingertip of an operator who is away from the object and points at the object with an imaging unit worn by the operator, and specifies an instruction position intended by the instructor on the captured image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In maintenance and inspection work, when a camera is attached to the operator's body, based on the captured image captured by the camera, it is possible to grasp whether the operator is performing pointing and naming and which object the operator is pointing at. However, a parallax occurs between the operator's line of sight and the direction of the camera attached to the operator. Therefore, even when the fingertip and the object overlap in the viewing angle (field of view) of the operator who is pointing and naming, a deviation occurs in the image captured by the camera attached to the operator. Thus, it is not easy to identify the object pointed at by the operator from the image captured by the camera.

[0006] Therefore, the present invention has been made in view of the above problems, and an object thereof is to provide a technique for appropriately identifying an object based on an image captured by an imaging unit worn by an operator.

Means for Solving the Problems

[0007] To solve the above object, the present invention is an object identification system that identifies an object pointed at by the operator based on an image captured by an imaging unit worn by the operator, the system including: a calculation unit that calculates the positional relationship between the eyes of the operator and the imaging unit based on a reference image captured by the imaging unit in a state where the operator is facing a reference object provided at a predetermined distance; a creation unit that creates an environmental map of the space captured by the imaging unit based on the space image captured by the imaging unit; a setting unit that sets a line-of-sight plane including the eyes and fingertips of the operator and the object on the environmental map based on the positional relationship; an estimation unit that estimates a line of sight on a predetermined image captured by the imaging unit of the fingertips and the object in a state where the operator is pointing at the object; a correction unit that corrects the predetermined image based on the line of sight; and an identification unit that identifies the object pointed at by the operator based on the corrected image corrected by the correction unit. In addition, "wearing" in the present invention includes a state of being attached via a helmet or the like worn by the operator.

Effects of the Invention

[0008] According to the present invention, an object can be appropriately identified based on an image captured by an imaging unit worn by an operator.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0010] Hereinafter, a specific example of the object identification system according to the embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited by the examples, but is defined by the claims.

Example

[0011] FIG. 1 is a schematic diagram of the object identification system at the work site according to Example 1.

[0012] The object identification system 10 is a system that identifies the object T pointed at by the operator W. The object T is, for example, a device or facility such as a server. The operator W makes a pointing call to the object T during the maintenance and inspection of the object T. In the pointing call, the operator W extends the arm forward and points the fingertip F at the object T while shouting.

[0013] Operator W is wearing a helmet H on his head. A monocular camera 11, which is an example of an "imaging unit" that constitutes the object identification system 10, is attached to the upper front of the helmet H. Note that the camera 11 may be attached to other parts such as the chest, in addition to the head of operator W. Furthermore, the object identification system 10 other than the camera 11 may be provided integrally with the camera 11, or may be provided in the pocket of operator W or other places in a state where it can communicate with the camera 11 wirelessly or by wire.

[0014] Here, the camera 11 is attached to the helmet H worn by operator W and is located above the line of sight L1 of operator W. The direction L2 of the camera 11 is fixed obliquely downward. Therefore, a parallax θ occurs between the line of sight L1 of operator W and the direction L2 of the camera.

[0015] Figure 2 is a diagram comparing the field of view of the operator according to Example 1 with the camera image.

[0016] In the field of view angle (field of view) V of operator W who points at and names the object T, the fingertip F and the object T overlap at the center of the field of view angle V. On the other hand, in the captured image i1, which is an example of a "predetermined image" obtained by the camera 11 capturing operator W who points at and names the object T, the fingertip F is displayed at the center of the captured image i1, and the object T is displayed at the upper part of the captured image i1. The difference (shift) in the positions of the fingertip F and the object T displayed in the captured image i1 is proportional to the magnitude of the parallax θ between the line of sight of operator W and the camera 11 and the distance from the camera 11 to the object T. In this example, since the camera 11 is above the line of sight of operator W, the object T is displayed at the upper part of the captured image i1.

[0017] Figure 3 is a diagram for explaining the position of the fingertip of the operator who points at and names the object according to Example 1. (a) shows a state where the fingertip of the operator is located on the line of sight, and (b) shows a state where the fingertip of the operator is located to the left of the line of sight.

[0018] Generally, when the operator W points and names, the fingertip F of the operator W is located near the upper and lower centers of the viewing angle V of the operator W. That is, even if the operator W or the movement of the operator W (the angle of bending the arm) is different for each pointing and naming, the fingertip F of the operator W who points and names the object T is located near either the upper or lower center of the viewing angle V of the operator W.

[0019] FIG. 4 is a functional block diagram of the object identification system according to the first embodiment, and FIG. 5 is a diagram for explaining the object identification process according to the first embodiment.

[0020] The object identification system 10 executes an object identification process for identifying the object T pointed by the operator W. The object identification system 10 includes a monocular camera 11, a memory 12 as an example of a "storage unit", a processor 13, a warning unit 14, and a display unit 15.

[0021] The camera 11 is attached to the helmet H worn by the operator W and captures an image of the object T. The camera 11 transmits the captured image i1 of the object T to the processor 13.

[0022] The memory 12 has a semiconductor storage medium such as a RAM (Random Access Memory), a ROM (Read Only Memory), an SSD (Solid State Drive), etc. The memory 12 may store the individual parallax for each operator W, the vertical parallax of the operator W for each of a plurality of heights with respect to the operator W, and a predetermined range from the object T for each of the plurality of objects T. The individual parallax is the individual difference in the position of the fingertip F of each operator W when pointing and naming. The vertical parallax is the tendency of the position of the fingertip F of the operator W when the operator W points and names objects T with different heights. The predetermined range is the range in which the operator W is allowed to approach the object T.

[0023] The processor 13 is, for example, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), ASIC (Application Specific Integrated Circuit), or the like. The processor 13 includes a calculation unit 21, a creation unit 22, a setting unit 23, an estimation unit 24, and an identification unit 25.

[0024] The calculation unit 21 calculates the positional relationship between the eyes of the operator W and the camera 11 by means of a calibration operation described below that is performed in advance before the operator W starts the maintenance inspection operation. The calculation unit 21 may calculate the distance between the fingertip F and the object T based on an environmental map of the space imaged by the camera 11, which will be described later. Further, the calculation unit 21 may calculate the distance between the eyes of the operator W and the fingertip F based on the positional relationship between the eyes of the operator W and the camera 11.

[0025] The creation unit 22 creates an environmental map of the space imaged by the camera 11 by means of SLAM (Simultaneous Localization and Mapping) or the like based on the space image imaged by the camera 11.

[0026] The setting unit 23 sets a line-of-sight plane P (Fig. 5(b)) including the eyes and fingertip F of the operator W and the object T on the environmental map based on the positional relationship calculated by the calculation unit 21. Note that the setting unit 23 may set the line-of-sight plane P based on the positional relationship and individual parallax. Further, the setting unit 23 may set the line-of-sight plane P based on the positional relationship and vertical parallax. Further, the setting unit 23 may set the line-of-sight plane P based on the positional relationship, individual parallax, and vertical parallax.

[0027] The estimation unit 24 estimates the line of sight L1 of the operator W on the line-of-sight plane P based on the coordinate difference between the central coordinate image on the captured image i1 and the coordinates of the fingertip F (see Fig. 5(c)).

[0028] The specific unit 25 identifies the object T pointed at by the operator W based on the line of sight L1 of the operator W. At this time, on the line of sight L1 of the operator W, the eyes and fingertips F of the operator W and the object T to be dealt with are arranged in this order.

[0029] FIG. 6 is a schematic diagram for explaining the approach distance according to the first embodiment.

[0030] When the distance between the fingertip F of the operator W and the object T approaches a predetermined range, the warning unit 14 generates an alert Al. Further, when the distance between the eyes of the operator and the fingertip F is less than a first distance (for example, 15 cm) or greater than a second distance (for example, more than 1.5 times farther than the average length of an adult's arm) from the first distance, the warning unit 14 generates an alert Al. Further, when the position of the fingertip F on the environmental map is separated from the position of the past fingertip F by a third distance or more, the warning unit 14 generates an alert Al.

[0031] FIG. 7 is a diagram showing the screen of the display device according to the first embodiment.

[0032] The display unit 15 may be a display device such as a display. The display unit 15 displays the object T and a processed image i2 in which the visual line L1 of the operator W is emphasized on the captured image i1. Note that the display unit 15 may emphasize and display either one of the object T and the visual line of the operator W in the processed image i2.

[0033] Next, a calibration operation for calculating the parallax θ between the line of sight L1 of the operator W and the direction L2 of the camera 11 will be described.

[0034] FIG. 8 is a schematic diagram for explaining the calibration operation according to the first embodiment, and FIG. 9 is a diagram for explaining the parallax calculated in the calibration operation according to the first embodiment.

[0035] As shown in FIG. 8, the camera 11 is attached to the body (e.g., the forehead) of the operator W. At a position separated from the operator W by a predetermined distance, a marker M (e.g., an AR marker), which is an example of a "reference object", is provided. The marker M is provided vertically at a predetermined height. This enables the operator W to easily perform calibration. The operator W stands upright facing the marker M from a position separated from the marker M by a predetermined distance, and aligns the line of sight L1 with the marker M.

[0036] The camera 11 images the marker M. The calculation unit 21 recognizes the marker M in the captured image i1 captured by the camera 11, and calculates the two-dimensional coordinates pM of the marker M in the coordinates of the captured image i1, the rotation vector R indicating the positional relationship between the marker M and the camera 11, and the translation vector t = (tx, ty, tz). At this time, when the marker M on the line-of-sight plane P (line of sight L1) is t (m) away from the camera 11, it is imaged at the coordinates pM in the coordinate system (captured image i1) of the camera 11.

[0037] The calculation unit 21 calculates the parallax θ between the line of sight L1 and the direction L2 of the camera 11 based on the rotation vector R. The calculation unit 21 can convert from the image coordinate system to the line-of-sight plane P based on the coordinates pM and the parallax θ, and can convert the fingertip coordinates in the captured image i1 into three-dimensional coordinates. Since the eyes of the operator W are directly below the camera 11, the coordinates of the eyes are (0, 0, tz).

[0038] FIG. 10 is a flowchart showing the object identification process according to the first embodiment.

[0039] First, before the operator W starts the maintenance and inspection work, the calculation unit 21 calculates the positional relationship between the eyes of the operator W and the camera 11 through a calibration operation (S1).

[0040] Next, the creation unit 22 creates a map of the environment of the space imaged by the camera 11 by SLAM or the like based on the space image imaged by the camera 11 (S2). Next, the setting unit 23 sets a line-of-sight plane P including the eyes and fingertips F of the worker W and the object T on the environmental map based on the positional relationship calculated by the calculation unit 21 (S3).

[0041] Next, the estimation unit 24 estimates the line of sight L1 of the worker W on the line-of-sight plane P based on the coordinate difference between the central coordinate image on the captured image i1 and the coordinates of the fingertip F (S4). Next, the specification unit 25 specifies the object T pointed at by the worker W based on the line of sight L1 of the worker W (S5).

[0042] According to this configuration, the object identification system 10 that identifies the object T pointed at by the worker W based on the image captured by the camera 11 worn by the worker W includes a calculation unit 21, a creation unit 22, a setting unit 23, an estimation unit 24, and a specification unit 25. The calculation unit 21 calculates the positional relationship between the eyes of the worker W and the camera 11 based on the marker M imaged by the camera 11 in a state where the worker W is facing the marker M provided at a predetermined distance. The creation unit 22 creates a map of the environment of the space imaged by the camera 11 based on the space image imaged by the camera 11. The setting unit 23 sets a line-of-sight plane P including the eyes and fingertips F of the worker W and the object T on the environmental map based on the positional relationship. The estimation unit 24 estimates the line of sight L1 of the worker W on the line-of-sight plane P based on the coordinate difference between the central coordinates on the captured image i1 of the fingertip F and the object T and the coordinates of the fingertip when the worker W points at the object T. The specification unit 25 specifies the object T pointed at by the worker W based on the line of sight L1. Thereby, based on the image captured by the camera 11 worn by the worker W, the object T pointed at by the worker W can be appropriately specified.

[0043] Furthermore, a memory 12 for recording individual parallax is provided for each of the plurality of workers W, and the setting unit 23 sets the line-of-sight plane P based on the positional relationship between the eyes of the worker W and the camera 11 and the individual parallax. Thereby, even when there are individual differences among the workers W, the line-of-sight plane P can be appropriately set.

[0044] Furthermore, a memory 11 is provided for recording the vertical parallax of the operator W for each of a plurality of heights with respect to the operator W, and the setting unit 23 sets the line-of-sight plane based on the positional relationship and the vertical parallax between the eyes of the operator W and the camera 11. Thereby, even at a work site where the operator W looks up or down at the object T, the line-of-sight plane can be appropriately set.

[0045] Furthermore, a memory 12 for recording a predetermined range from each object T for each of a plurality of objects T and a warning unit 14 for generating an alert Al are provided. The calculation unit 21 calculates the distance between the fingertip F and the object T based on the environmental map, and the warning unit 14 generates an alert Al when the distance between the fingertip F and the object T approaches the predetermined range. Thereby, it is possible to prevent the operator W from approaching the energized object T more than necessary.

[0046] Furthermore, a warning unit 14 for generating an alert Al is provided. The calculation unit 21 calculates the distance between the eyes of the operator W and the fingertip F based on the positional relationship between the eyes of the operator W and the camera 11. The warning unit 14 generates an alert Al when the distance between the eyes of the operator W and the fingertip F is less than a first distance or equal to or greater than a second distance greater than the first distance, or when the position of the fingertip F on the environmental map is separated from the position of the past fingertip F by a third distance or more. Thereby, the operator W can confirm that the camera 11 has collided with another object or the fixture of the camera 11 has become loose, and the position and orientation of the camera 11 have been changed.

Example

[0047] The object identification system 30 according to Embodiment 2 of the present invention will be described. The object identification system 30 according to the second embodiment is different from the object identification system 10 according to the first embodiment only in the configuration of the processor 33, and the other configurations are the same as those of the object identification system 10 according to the first embodiment. Therefore, the description will focus on the differences from the first embodiment.

[0048] FIG. 11 is a diagram of an object identification system at a work site according to Example 2. FIG. 12 is a flowchart showing the object identification process according to Example 2.

[0049] The processor 33 of the object identification system 30 includes a calculation unit 21, a creation unit 22, an estimation unit 44, and an identification unit 45.

[0050] In Example 1, when the worker W pointed and named, it was assumed that the fingertip F of the worker W was located near the upper and lower centers of the viewing angle V of the worker W. In Example 2, by assuming that the fingertip F is located near the upper, lower, left, and right centers of the viewing angle V of the worker W, it becomes possible to estimate the line of sight L1 without setting the line-of-sight plane P. Therefore, the estimation unit 44 estimates the line of sight L1 from the above assumption and the positional relationship among the eyes of the worker W, the camera 11. The identification unit 45 identifies the object T that the worker W is gazing at near the upper, lower, left, and right centers of the field of view based on the line of sight L1. Further, when the above assumption is made, since the information of the fingertip F is not required for the estimation of the line of sight L1, it is not necessary to perform pointing and naming in this embodiment.

[0051] According to this configuration, based on the image captured by the camera 11 worn by the worker W, the object T that the worker W is gazing at can be easily and appropriately identified.

Example

[0052] The object identification system 50 according to Example 3 of the present invention will be described. Note that the object identification system 50 according to the third embodiment is different from the object identification system 10 according to the first example only in the mounting position on the worker W, and the other configurations are the same as those of the object identification system 10 according to Example 1. Therefore, the description will focus on the differences from Example 1.

[0053] FIG. 13 is a diagram of an object identification system at a work site according to Example 3..

[0054] The camera 11 of the object identification system 50 is attached to the right side of the helmet H worn by the operator W on the head. That is, the camera 11 of the object identification system 50 is disposed to the right side of the eyes of the operator W.

[0055] According to this configuration, when the fingertip F of the operator W who points at and names the object T is located near the left and right center of the viewing angle V of the operator W, the object T pointed at by the operator W can be appropriately identified based on the image captured by the camera 11 worn by the operator W.

[0056] Note that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Also, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment. Also, the configuration of another embodiment can be added to the configuration of one embodiment. Also, for a part of the configuration of another embodiment, other configurations can be added, deleted, or replaced.

Explanation of Reference Numerals

[0057] 10... Object identification system, 11... Camera, 12... Memory, 14... Warning unit, 15... Display unit, 21... Calculation unit, 22... Creation unit, 23... Setting unit, 24... Estimation unit, 25... Identification unit, 30... Object identification system, 44... Estimation unit, 45... Identification unit, 50... Object identification system, Al... Alert, F... Fingertip, i1... Captured image, i2... Processed image, L1... Line of sight, P... Line-of-sight plane, T... Object, W... Operator, θ... Parallax

Claims

1. An object identification system for identifying an object pointed at by an operator based on an image captured by a monocular imaging unit worn by the operator, comprising: a calculation unit that calculates the positional relationship between the operator's eye and the imaging unit based on a reference image captured by the imaging unit in a state where the operator is facing a reference object provided at a predetermined distance; a creation unit that creates an environmental map of the space captured by the imaging unit based on the space image captured by the imaging unit; a setting unit that sets a line-of-sight plane including the operator's eye and fingertip and the object on the environmental map based on the positional relationship; an estimation unit that estimates the operator's line of sight on the line-of-sight plane based on the coordinate difference between the central coordinates on a predetermined image captured by the imaging unit of the fingertip and the object in a state where the operator points at the object; an identification unit that identifies the object pointed at by the operator based on the line of sight, the object identification system comprising.

2. An object identification system for identifying an object being gazed at by an operator based on an image captured by a monocular imaging unit worn by the operator, comprising: a calculation unit that calculates the positional relationship between the operator's eye and the imaging unit based on a reference image captured by the imaging unit in a state where the operator is facing a reference object provided at a predetermined distance; a creation unit that creates an environmental map of the space captured by the imaging unit based on the space image captured by the imaging unit; an estimation unit that estimates the operator's line of sight on the environmental map based on the positional relationship; an identification unit that identifies the object being gazed at by the operator based on the line of sight, the object identification system comprising.

3. The reference object is a marker provided vertically at a predetermined height, The object identification system according to claim 1 or 2.

4. A storage unit that records individual parallax for each of a plurality of the operators, The setting unit sets the line-of-sight plane based on the positional relationship and the individual parallax, The object identification system according to claim 1.

5. A storage unit that records the vertical parallax of the operator for each of a plurality of heights of the operator, The setting unit sets the line-of-sight plane based on the positional relationship and the vertical parallax, The object identification system according to claim 1.

6. a storage unit that records a predetermined range from each of the plurality of objects; a warning unit that generates an alert; comprising Based on the environmental map, the calculation unit calculates the distance between the fingertip and the object. When the distance between the fingertip and the object approaches the predetermined range, the warning unit generates an alert. The object identification system according to claim 1.

7. Comprising a warning unit that generates an alert. Based on the positional relationship between the eyes of the operator and the imaging unit, the calculation unit calculates the distance between the eyes of the operator and the fingertip. When the distance between the eyes of the operator and the fingertip is less than a first distance or greater than or equal to a second distance greater than the first distance, or when the position of the fingertip on the environmental map is separated from the past position of the fingertip by a third distance or more, the warning unit generates an alert. The object identification system according to claim 1.

8. Comprising a display unit that displays a processed image in which at least one of the object and the line of sight of the operator is emphasized on the predetermined image. The object identification system according to claim 1.

9. An object identification method by an object identification system that identifies an object pointed at by an operator based on an image captured by a monocular imaging unit worn by the operator, Based on the coordinate difference between the central coordinates on the reference image captured by the imaging unit when the operator is facing a reference object provided at a predetermined distance and the coordinates of the object, the positional relationship between the eyes of the operator and the imaging unit is calculated. Based on the spatial image captured by the imaging unit, an environmental map of the space captured by the imaging unit is created. Based on the positional relationship, a line-of-sight plane including the eyes and fingertip of the operator and the object on the environmental map is set. Based on the coordinate difference between the central coordinates on the predetermined image captured by the imaging unit of the fingertip and the object in a state where the operator points at the object and the coordinates of the fingertip, the line of sight of the operator on the line-of-sight plane is estimated. An object identification method for identifying an object pointed at by an operator based on the line of sight.

10. An object identification method by an object identification system that identifies an object being gazed at by an operator based on an image captured by a monocular imaging unit worn by the operator, Based on the coordinate difference between the central coordinates on the reference image captured by the imaging unit when the operator is facing a reference object provided at a predetermined distance and the coordinates of the object, the positional relationship between the eyes of the operator and the imaging unit is calculated. Based on the spatial image captured by the imaging unit, an environmental map of the space captured by the imaging unit is created. Based on the above positional relationship, estimate the line of sight of the operator on the environmental map, An object identification method for identifying an object being stared at by the operator based on the line of sight.

Images