Real world pointing system and real world pointing method

Abstract

To realize real world pointing which is easy to install and operate.SOLUTION: In a pointing system 100, a controller 30 analyzes a pick-up image of a visual marker 23 attached to the non-movable portion of a pan tilt actuator 20, obtains a homogeneous transformation matrix 0TSini indicating the position and the posture of the controller in a world coordinate system Σ0 in a first state where the camera faces the pan tilt actuator, detects the change of the controller's posture with a motion sensor during a change from the first state to a second state where the camera faces a target 40, obtains a homogeneous transformation matrix SiniTS related to the change from the first state to the second state, and obtains the spatial coordinate Sr of the target in the controller's coordinate system ΣS based on the distance to the target in the second state. As the result, the pan tilt actuator 20 operates at a pan angle θP and a tilt angle θT obtained from inverse kinematics (θP,θT,lb)T=IK(0r).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a real world pointing system and a real world pointing method for pointing to a target that exists in the real world, which is a three-dimensional space, with a laser beam. [Background technology]

[0002] One of the practical tasks that care recipients want a robot to perform is to grasp and transport everyday items in a room. However, because our daily living space is extremely diverse, it is very difficult to fully automate tasks involving handling unregistered and non-standard objects. Therefore, the inventor has been working on a manual (semi-automatic) approach that makes the most of the remaining excellent intelligence and skills of care recipients, in addition to the high-precision measurement and control technology of the robot, rather than fully automating the robot, aiming for the early practical application of assistive robots. In this case, the most important thing is to research and develop an intuitive instruction interface, that is, how to instruct the robot to operate simply and reliably. From this perspective, the inventor has developed an intuitive interface (real-world clicker) that can generate various commands related to object manipulation by operating a high-precision TOF laser distance sensor mounted on a pan-tilt actuator with a PC mouse and clicking on a real object while visually observing it (see, for example, Non-Patent Document 1). Here, real-world clicking means measuring the 3D position of a laser spot in the world coordinate system from the pan angle, tilt angle, and laser beam length, and the accuracy has a resolution of about 5 mm at a distance of about 5 m. The user clicks on the target object's grasp point in the real world and sends the coordinates to the robot, which then automatically moves to the vicinity of the object, grasps it with its arm and hand, and carries it to the user. In addition, since the real-world clicker records the location where the object was, the user can automatically return the object to its original location.

[0003] The inventors have further developed a technology that uses a smartphone, a familiar device that is commonly used on a daily basis, as a controller for a real-world clicker, and links the movement of a pan-tilt actuator to the tilt of the smartphone, thereby allowing the user to point to the position of a target object that exists in the real world in a three-dimensional space with an intuitive operating feel (see, for example, Patent Document 1). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Proposal of a Real-World Clicking Method Using a TOF Laser Sensor and a Pan-Tilt Actuator and Its Application to Teaching Motion to a Life-Support Robot, Yuki Abiko, Yuta Hidaka, Kenjiro Sato, Satoshi Iwaki, Tetsushi Ikeda, Transactions of the Society of Instrument and Control Engineers, Vol. 52, No. 11, pp. 614-624, 2016 [Patent documents]

[0005] [Patent Document 1] JP 2023-77952 A Summary of the Invention [Problem to be solved by the invention]

[0006] The above real-world pointing has the following constraints: 1) the pan-tilt actuator is installed horizontally, 2) the position of the pan-tilt actuator and the user's head are relatively close enough compared to the distance between the pan-tilt actuator and the target, and 3) the user's head is always facing vertically upward. This causes practical problems such as limitations on the installation location and orientation of the pan-tilt actuator and the inability of the user to operate it in any posture.

[0007] In view of the above problems, an object of the present invention is to provide a real-world pointing technology that is easy to install and operate without any constraints. [Means for solving the problem]

[0008] According to one aspect of the present invention, there is provided a system in which a controller operates a pan-tilt actuator at a remote location and points to a target in the real world with a laser beam emitted from a laser pointer mounted on the pan-tilt actuator, the system comprising: a controller having a camera and a motion sensor capable of capturing optical images and measuring depth; and a homogeneous transformation matrix (H0) representing a position and orientation of the controller in a world coordinate system Σ0 in a first state in which the camera is facing the pan-tilt actuator, the ... which the camera is facing the pan-tilt actuator, the homogeneous transformation matrix (H0) representing a position and orientation of the controller in a world coordinate system Σ0 in which the camera is facing the pan-tilt actuator, the homogeneous transformation matrix (H0) representing a position and orientation of the controller in a world coordinate system Σ0 in which the camera is facing the pan-tilt actuator, the homogeneous transformation matrix (H0) representing a position and orientation of the controller in a world coordinate system Σ0 in which the camera is facing the pan-tilt actuator, the homogeneous transformation matrix (H0) representing a position and orientation of the controller 0 T Sini a homogeneous transformation matrix relating to the change from the first state to the second state, the homogeneous transformation matrix relating to the change from the first state to the second state being detected by the motion sensor. Sini T S and calculating the coordinate system Σ of the controller from the distance to the target measured by the camera in the second state. S The spatial coordinates of the target in S r is calculated, and the pan-tilt actuator is calculated by the inverse kinematics (θ P ,θ T ,l b ) T =IK( 0 r) Pan angle θ P and tilt angle θ T (However, l b is the beam length of the laser beam, 0 r is the spatial coordinate of the laser spot of the laser beam in the world coordinate system Σ0. 0 r= 0 T Sini Sini T S S r.) A real-world pointing system is provided.

[0009] According to one aspect of the present invention, there is provided a method for operating a pan-tilt actuator at a remote location using a controller having a camera and a motion sensor capable of capturing optical images and measuring depth, and pointing to a target in the real world with a laser beam emitted from a laser pointer mounted on the pan-tilt actuator, the method comprising the steps of: analyzing an image captured by the camera of a visual marker attached to a non-movable portion of the pan-tilt actuator; and calculating a homogeneous transformation matrix representing a position and orientation of the controller in a world coordinate system Σ0 in a first state in which the camera is facing the pan-tilt actuator. 0 T Sini a homogeneous transformation matrix relating to the change from the first state to the second state, the homogeneous transformation matrix relating to the change from the first state to the second state being detected by the motion sensor when the controller changes from the first state to a second state in which the camera is facing the target. Sini T S and the controller calculates a coordinate system Σ of the controller from the distance to the target measured by the camera in the second state. S The spatial coordinates of the target in S r is calculated, and the pan-tilt actuator is calculated by the inverse kinematics (θ P ,θ T ,l b ) T =IK( 0 r) Pan angle θ P and tilt angle θ T (However, l b is the beam length of the laser beam, 0 r is the spatial coordinate of the laser spot of the laser beam in the world coordinate system Σ0. 0 r= 0 T Sini Sini T S S r.) A real-world pointing method is provided. Effect of the Invention

[0010] According to the present invention, the pan-tilt actuator can be installed at any location and in any orientation, and the user can operate the pan-tilt actuator with a controller in any posture to point a laser beam at a target in the real world. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic diagram of a real world pointing system according to an embodiment of the present invention; [Diagram 2] FIG. 11 is a control flow diagram of the pan-tilt actuator in the coarse adjustment mode. [Diagram 3] 11A to 11C are diagrams illustrating the operation and posture change of the smartphone in the coarse adjustment mode. [Figure 4] FIG. 1 illustrates the forward kinematics of a pan-tilt actuator. [Diagram 5] FIG. 11 is a control flow diagram of the pan-tilt actuator in the fine adjustment mode. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed explanations than necessary may be omitted. For example, detailed explanations of already well-known matters or duplicate explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the inventor provides the accompanying drawings and the following explanation so that those skilled in the art can fully understand the present invention, and does not intend to limit the subject matter described in the claims by them. In addition, the dimensions of each member depicted in the drawings, detailed shapes of details, etc. may differ from the actual ones.

[0013] <Example of real-world pointing system configuration> 1 is a schematic diagram of a real world pointing system according to an embodiment of the present invention. The real world pointing system 100 according to this embodiment includes a laser pointer 10, a pan-tilt actuator 20 that changes the direction of the laser pointer 10, and a smartphone 30 as a controller that operates the pan-tilt actuator 20.

[0014] The laser pointer 10 is a device having a function of emitting a laser beam 11 and a function of measuring the distance to a target 40. Specifically, the laser pointer 10 has a TOF (Time of Flight) type laser sensor function that projects a pulse of the laser beam 11 toward the target 40 and measures the distance to the target 40 by measuring the time from emitting the laser beam 11 to receiving the reflected light from the target 40.

[0015] The pan-tilt actuator 20 is an electric device in which a main body 21 is installed on a fixed object such as a ceiling and drives a camera platform 22 in the tilt direction indicated by the arrow a and the pan direction indicated by the arrow b in the figure. A laser pointer 10 is mounted and fixed on the camera platform 22 of the pan-tilt actuator 20. This allows the attitude of the laser pointer 10 to be changed by instructing the pan angle and tilt angle to the pan-tilt actuator 20, thereby changing the direction of the laser beam 11 irradiated from the laser pointer 10. When the laser beam 11 is irradiated onto an object, a laser spot 12 that is visible to the user is formed on the surface of the object. The position of this laser spot 12 is the pointing position in the real world of a three-dimensional space.

[0016] In the example of Fig. 1, the pan-tilt actuator 20 is installed on the ceiling. Furthermore, a visual marker 23 is attached to a non-movable portion of the pan-tilt actuator 20, for example, immediately next to the main body 21. The visual marker 23 is used in AR (Augmented Reality) technology and the like. By photographing the visual marker 23 and analyzing the captured image, the position of the object to which it is attached can be spatially recognized.

[0017] Needless to say, the smartphone 30 is a mobile terminal that almost everyone owns and uses on a daily basis. Recent smartphones 30 are equipped with a motion sensor that detects the attitude and movement of the case, in addition to an optical camera and wireless communication functions. More specifically, the motion sensor is a 9-axis sensor in which an acceleration sensor, a gyro sensor, and a geomagnetic sensor are integrated into an integrated module. The acceleration sensor is a sensor that detects acceleration in three orthogonal axial directions of the case, the gyro sensor is a sensor that detects angular velocity around three orthogonal axes, and the geomagnetic sensor is a sensor that detects geomagnetism in three orthogonal axial directions of the case.

[0018] Some smartphones 30 further include a depth camera, such as LiDAR. Such a depth camera is integrated with an optical camera and disposed on the back of the smartphone 30. In the case of a smartphone 30 equipped with a depth camera, a user can "photograph" a target 40 in the same manner as photographing a subject with an optical camera, and accurately measure the distance to the target 40.

[0019] In the real world pointing system 100 configured as above, the laser pointer 10 and the pan-tilt actuator 20 are capable of various short-range wireless communication 31 such as Wi-Fi (registered trademark) and Bluetooth (registered trademark) with the smartphone 30 via a wireless communication interface (not shown). A dedicated application, which is a computer program for controlling the laser pointer 10 and the pan-tilt actuator 20, is installed in the smartphone 30 as described below, and the smartphone 30 functions as a controller of the real world pointing system 100 by starting the dedicated application. Specifically, in a coarse adjustment mode described later, the pan-tilt actuator 20 is controlled by photographing the visual marker 23 and the target 40, and the laser spot 12 of the laser beam 11 irradiated from the laser pointer 10 is guided to the desired position on the target 40. Then, in a fine adjustment mode described later, the position of the laser spot 12 can be finely adjusted by operating a UI displayed on the touch screen of the smartphone 30.

[0020] <Coarse adjustment mode> FIG. 2 is a control flow diagram of the pan-tilt actuator 20 in the coarse adjustment mode. FIG. 3 is a diagram for explaining the operation and attitude change of the smartphone in the coarse adjustment mode. The real world pointing system 100 starts in the coarse adjustment mode. When the application of the smartphone 30 is started (S11), the application first requests the user to photograph the visual marker 23. In response, the user points the camera on the back of the smartphone 30 toward the pan-tilt actuator 20 to photograph the visual marker 23 (S12). The position and attitude of the user and the smartphone 30 at this time are arbitrary. For example, the user can lie on his / her back on a bed, hold the smartphone 30, and point the camera on the back toward the pan-tilt actuator 20 installed on the ceiling to photograph the visual marker 23.

[0021] When the visual marker 23 is photographed, the app on the smartphone 30 analyzes the captured image of the visual marker 23 and calculates a homogeneous transformation matrix (HM) that represents the position and orientation of the smartphone 30 as viewed from the world coordinate system Σ0 in the first state in which the camera is facing the pan-tilt actuator 20. 0 T Sini (S13).

number

[0022] The world coordinate system Σ0 is a coordinate system fixed to the real world, such as the ceiling on which the pan-tilt actuator 20 is installed. Regardless of the orientation of the pan-tilt actuator 20, the Z axis (Z0) of the coordinate system Σ0 is set to be vertically upward, and the X axis (X0) and Y axis (Y0) are set to be two orthogonal axes on a horizontal plane perpendicular to Z0. Rotation matrix 0 R Sini is the coordinate system Σ of the smartphone 30 when the camera is facing the pan-tilt actuator 20 from the world coordinate system Σ Sini is the coordinate transformation matrix to the coordinate system Σ Sini In the Z axis (Z Sini ) and the X-axis (X Sini ) is the longitudinal direction of the smartphone 30 facing upward and the transverse direction facing right when the camera is facing the pan-tilt actuator 20, and the Y-axis (Y Sini ) is the extension direction of the camera in that state. 0 p Sini is the position of the smartphone 30 in the world coordinate system Σ0. 0 T Sini Rotation matrix in 0 R Sini and the translation vector 0 p Sini can be obtained by calculating the relative positional relationship between the visual marker 23 and the smartphone 30 from the captured image of the visual marker 23 without using a motion sensor.

[0023] Next, the smartphone 30 app asks the user to "photograph" the target 40. Photographing here means measuring the distance using the depth camera. Accordingly, the user points the camera of the smartphone 30 at the target 40 and photographs the target 40. During this process, the smartphone 30 app calculates, from the signal of the motion sensor, a homogeneous transformation matrix Sini T S (S14).

number

[0024] Rotation matrix Sini R S is the coordinate system Σ of the smartphone 30 when the camera of the smartphone 30 is facing the pan-tilt actuator 20. Sini Then, the coordinate system Σ of the smartphone 30 when the camera is facing the target 40 S This can be obtained from the signal of the motion sensor. On the other hand, it is difficult to measure the translation of the smartphone 30 caused by the change in the posture by the motion sensor, and the position of the smartphone 30 hardly changes when the target 40 is photographed. Sini T S The translation vector is considered to be zero.

[0025] Specifically, the distance to the target 40 is measured by the user pressing the shutter when the target 40 enters the distance measurement point while viewing the image displayed on the screen of the smartphone 30. When the distance to the target 40 is measured, the application of the smartphone 30 calculates the spatial coordinates of the distance measurement point on the target 40. S r is calculated (S15). For example, the distance to the target 40 obtained by "photographing" the target 40 is denoted as ls. Here, the coordinate system Σ point is the coordinate system Σ of the smartphone 30 when "photographing" the target 40 SIf we consider it to be the same as S r=(0,l s ,0) T It is expressed as:

[0026] S r is expressed using the world coordinate system Σ0 0 r= 0 T Sini Sini T S S The smartphone app 30 calculates the inverse kinematics (θ P ,θ T ,l b ) T =IK( 0 r) the pan angle θ of the pan-tilt actuator 20 P and tilt angle θ T The pan angle θ of the pan-tilt actuator 20 is calculated (S16). P and tilt angle θ T When is calculated, the application of the smartphone 30 transmits a rotation instruction to the pan-tilt actuator 20 using the calculated value as a control signal (S17). Meanwhile, the pan-tilt actuator 20 is in a state of waiting for a control instruction from the smartphone 30 (NO in S21), and upon receiving a rotation instruction from the smartphone 30 (YES in S21), the application of the smartphone 30 transmits the instructed pan angle θ P and tilt angle θ T Then, the laser pointer 10 is directed toward the target 40, and the laser beam 11 is emitted from the laser pointer 10 toward the target 40 (S23).

[0027] As described above, in the coarse adjustment mode, the pan-tilt actuator 20 can be operated to guide the laser spot 12 of the laser beam 11 to the target 40 by sequentially photographing the visual marker 23 attached to the non-movable part of the pan-tilt actuator 20 and the target 40 with the smartphone 30.

[0028] <Fine-tuning mode> In the above-described coarse adjustment mode, the translation vector of the smartphone 30 is considered to be zero when the camera of the smartphone 30 is directed toward the target 40, so that the laser spot 12 may deviate from the desired position immediately after the coarse adjustment. Such deviation can be corrected in the fine adjustment mode.

[0029] Before describing the fine adjustment mode, the forward kinematics of the pan-tilt actuator 20 will be described. FIG. 4 is a diagram illustrating the forward kinematics of the pan-tilt actuator 20. θ P is the pan angle, θ T is the tilt angle, l b is the beam length of the laser beam 11, δ Z is the offset of the laser pointer 10 in the Z-axis direction of the rotor coordinate system. FIG. 4 shows an example in which the pan-tilt actuator 20 is placed on a table, but the same can be considered even if it is installed on the ceiling. δ Z Since is small enough, it can be ignored and the forward kinematics of the pan-tilt actuator 20 can be calculated as shown in equation (1). 0 r is the spatial coordinate of the laser spot 12 of the laser beam 11 in the world coordinate system Σ0.

number

[0030] In the fine adjustment mode, the position of the laser spot 12 is set as the origin, and the Y axis (Y point ) is the traveling direction of the laser beam 11, and the X-axis (X point ) is the horizontal direction, and the Z axis (Z point ) is X point and Y point The pointing coordinate system Σ point For the spot micro-motion operation near the origin of the coordinate system, we attempt to optimize the manipulability of a multi-link manipulator that regards the laser beam 11 as a passive translation link. Here, the index of optimality is as follows: (1) Manipulability is the same for any q (2) For any q, the manipulability ellipsoid is a sphere. where q=(l b ,θT ,θ P ) T It is.

[0031] By differentiating equation (1), 0 The linear relationship between the r and q fractions is given by equation (2), which determines the motion of the pan-tilt actuator 20.

number

number

number

number

[0032] 0 r in the coordinate system Σ point can be expanded as follows: point r= point T0 0 r= point R0 0 r+ point p0(5)

[0033] By differentiating equation (5), point The linear relationship between the r and q fractions is given by equation (6), which determines the motion of the pan-tilt actuator 20. point dr= point R0 0 dr= point R0J(q)dq= point R0UΣV T dq (6)

[0034] Here, X of the laser spot 12 point The indicated amount of micro-movement is dx, Z point The instruction amount of the small movement is dz, and the new operation input is (dx, 0, dz) T Introducing Equation (7) can be obtained from Equation (6). dq=J -1 (q) 0 R point (dx,0,dz) T (7)

[0035] In equation (7), 0 R point = 0 R Sini Sini R S S R point In the coordinate system Σ S and Σ point is the same S R point is a unit matrix. Therefore, equation (7) is transformed into equation (8). dq=J -1 (q) 0 R Sini Sini R S (dx,0,dz) T (8)

[0036] When equation (8) is input into equation (5), ( point dx, point dz) T =(dx,dz) T That is, the instruction amount (dx, dz) of the minute movement of the laser spot 12 is obtained. T In contrast, the laser spot 12 ( point dx, point dz) T It can be seen that only

[0037] In the formula (6), point Since R0 and U are both orthogonal matrices, point R0U is also an orthogonal matrix. Therefore, in equation (6), point R0UΣV T Since is expressed as a singular value decomposition, we use Σ as a pre-matrix.-1 Using this, the manipulability ellipsoid becomes a circle rather than a sphere.

[0038] 5 is a control flow diagram of the pan-tilt actuator 20 in the fine adjustment mode. The user can switch the real world pointing system 100 from the coarse adjustment mode to the fine adjustment mode at any time. The user can also switch the real world pointing system 100 from the fine adjustment mode to the coarse adjustment mode at any time.

[0039] When the smartphone 30 transitions to the fine adjustment mode, the smartphone 30 application first acquires the distance to the target 40 and the operation angle of the pan-tilt actuator 20 at the time of transition to the fine adjustment mode (S31). The distance to the target 40 is specifically the laser beam length l b and can be obtained from the distance measurement information of the laser pointer 10. The operating angle of the pan-tilt actuator 20 includes at least the tilt angle θ T In order to specify the spatial position of the laser spot 12 after fine adjustment, the pan angle θ P may be included.

[0040] The application of the smartphone 30 displays a fine adjustment mode UI on the touch screen and receives an instruction from the user to slightly move the laser spot 12 (S32). The user can move the laser spot 12 slightly in units of several mm to several cm, that is, perform an inching operation through the UI. For example, the fine adjustment mode UI can be configured with four buttons representing the up, down, left, and right directions. With such a UI, when the user wants to move the laser spot 12 by a desired inching amount in a desired direction, he or she can click the button for that direction a desired number of times. Alternatively, if the fine adjustment mode UI is set to a reticle (sight), the user can issue an instruction equivalent to multiple inching operations to move the laser pointer 12 to any position within the reticle with one click, without clicking a button multiple times.

[0041] When the smartphone 30 application receives a micro-movement instruction for the laser spot 12 from the user through the fine adjustment mode UI (YES in S33), the smartphone 30 application converts the instructed micro-movement amount into the rotation angles in the pan direction and tilt direction of the pan-tilt actuator 20 (S34). T According to equation (8), the laser beam length l of the laser beam 11 is b and the pan angle θ of the pan-tilt actuator 20 P and tilt angle θ T The inverse Jacobian matrix J, expressed as -1 (q), and the homogeneous transformation matrix 0 T Sini and Sini T S The rotation matrix contained in 0 R Sini and Sini R S Multiplying by this gives the offset amount dθ for the pan and tilt angles. P and dθ T Calculate.

[0042] The offset amount dθ of the pan angle and tilt angle of the pan-tilt actuator 20 P and dθ T When is calculated, the application of the smartphone 30 transmits a rotation instruction to the pan-tilt actuator 20 using the calculated value as a control signal (S35), and then returns to waiting for an instruction to slightly move the laser spot 12. Meanwhile, the pan-tilt actuator 20 is in a state of waiting for a control instruction from the smartphone 30 (NO in S41), and when it receives a rotation instruction from the smartphone 30 (YES in S41), it transmits the instructed offset amount dθ P and dθ T The camera 10 controls its own pan angle and tilt angle in accordance with the above (S42), and then returns to a state of waiting for a control instruction from the smartphone 30 again.

[0043] Thus, in fine-tuning mode, the inverse singular values ​​of the Jacobian matrix and the rotation matrix 0 R Sini and Sini R SThe offset amounts of the pan and tilt angles of the pan-tilt actuator 20 are calculated using the above, and the pan-tilt actuator 20 is controlled by the offset amounts. This allows fine adjustment of pointing to a target 40 in a three-dimensional space with the same inching amount, i.e., the same operational feeling, regardless of the distance between the pan-tilt actuator 20 and the target 40. Note that the spatial position of the laser spot 12 after fine adjustment is determined by the laser beam length l obtained in step S31. b and the pan angle θ of the pan-tilt actuator 20 P and tilt angle θ T offset amount dθ P and dθ T The new pan angle θ P and tilt angle θ T can be determined by substituting into equation (1).

[0044] <<Variations>> Instead of the smartphone 30, a tablet terminal or the like may be used as a controller for the pan-tilt actuator 20.

[0045] The laser pointer 10 does not need to have both the function of emitting the laser beam 11 and the function of measuring the distance to the target 40. For example, the laser pointer 10 may have only the function of emitting the laser beam 11, and may be provided with another device (e.g., a TOF camera) having a function of measuring the distance to the target 40.

[0046] In the fine adjustment mode illustrated in FIG. 5, the operation of the pan-tilt actuator 20 is controlled by the offset amount of the pan angle and tilt angle, but the smartphone 30 may calculate the absolute amount of the pan angle and tilt angle reflecting the offset amount and control the operation of the pan-tilt actuator 20 by the calculated value.

[0047] As described above, the embodiment has been described as an example of the technology of the present invention. For this purpose, the attached drawings and detailed description have been provided. Therefore, among the components described in the attached drawings and detailed description, not only components essential for solving the problem but also components that are not essential for solving the problem in order to illustrate the above technology may be included. Therefore, just because those non-essential components are described in the attached drawings or detailed description, it should not be immediately recognized that those non-essential components are essential. In addition, since the above-mentioned embodiment is intended to illustrate the technology of the present invention, various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents. [Industrial Applicability]

[0048] The real-world pointing system of the present invention is easy to install and operate, and can therefore be used to operate robots in the field of nursing care, for example, by allowing a person in need of care to operate a smartphone while in bed to point to everyday items in the room and have an assistance robot go and retrieve them. [Explanation of symbols]

[0049] 100 Real World Pointing Systems 10 Laser Pointer 11 Laser Beam 12 Laser spot 20 Pan-tilt actuator 23 Visual Markers 30 Smartphone (controller) 40 Target

Claims

1. A system for pointing to a target in the real world with a laser beam emitted from a laser pointer mounted on a pan-tilt actuator by operating a pan-tilt actuator at a remote location using a controller, comprising: The controller has a camera and a motion sensor capable of capturing an optical image and measuring depth, and analyzes an image captured by the camera of a visual marker attached to a non-movable portion of the pan-tilt actuator to obtain a world coordinate system Σ in a first state in which the camera is facing the pan-tilt actuator. ０ A homogeneous transformation matrix representing the position and orientation of the controller in ０ T Ｓｉｎｉ a homogeneous transformation matrix relating to the change from the first state to the second state, Ｓｉｎｉ T Ｓ and calculating the coordinate system Σ of the controller from the distance to the target measured by the camera in the second state. Ｓ The spatial coordinates of the target in Ｓ Find r, The pan-tilt actuator is based on the inverse kinematics (θ Ｐ , θ Ｔ , l ｂ ) Ｔ =IK( ０ r) Pan angle θ Ｐ and tilt angle θ Ｔ (However, ｂ is the beam length of the laser beam, ０ r is the world coordinate system Σ ０ The spatial coordinates of the laser spot of the laser beam in ０ r = ０ T Ｓｉｎｉ Ｓｉｎｉ T Ｓ Ｓ r.) A real-world pointing system characterized by:

2. The controller calculates a homogeneous transformation matrix by regarding a translation vector of the controller as zero when the controller changes from the first state to the second state. ０ T Ｉｎｉ Ask for 2. The real world pointing system of claim 1.

3. The controller receives an instruction for a micro-movement of the laser spot from a user, and calculates a coefficient and a homogeneous transformation matrix based on the beam length of the laser beam and the operation angle of the pan-tilt actuator in response to the instruction for the micro-movement of the laser spot. ０ T Ｓｉｎｉ and Ｓｉｎｉ T Ｓ and calculating offset amounts of the pan angle and tilt angle of the pan-tilt actuator by multiplying the offset amount by a rotation matrix included in the offset amount, and instructing the pan-tilt actuator via wireless communication the absolute amounts of the pan angle and tilt angle that reflect the calculated offset amount or the offset amount; The pan-tilt actuator is configured to finely adjust its own pan and tilt angles by an offset amount or an absolute amount instructed by the controller.

3. The real world pointing system according to claim 1, wherein the real world pointing system is a pointing system for displaying a real world image.

4. The instruction for the minute movement of the laser spot is an inching operation in any direction of the up, down, left, or right of the laser spot.

4. The real world pointing system of claim 3.

5. A method for pointing to a target in the real world with a laser beam emitted from a laser pointer mounted on a pan-tilt actuator by operating a pan-tilt actuator at a remote location using a controller having a camera and a motion sensor capable of capturing an optical image and measuring depth, comprising the steps of: The controller analyzes a captured image of a visual marker attached to a non-movable portion of the pan-tilt actuator captured by the camera, and calculates a world coordinate system Σ ０ A homogeneous transformation matrix representing the position and orientation of the controller in ０ T Ｓｉｎｉ Seeking A change in the attitude of the controller when the controller changes from the first state to a second state in which the camera is facing the target is detected by the motion sensor, and a homogeneous transformation matrix relating to the change from the first state to the second state is detected. Ｓｉｎｉ T Ｓ Seeking The controller calculates a coordinate system Σ of the controller from the distance to the target measured by the camera in the second state. Ｓ The spatial coordinates of the target in Ｓ Find r, The pan-tilt actuator is based on the inverse kinematics (θ Ｐ , θ Ｔ , l ｂ ) Ｔ =IK( ０ r) Pan angle θ Ｐ and tilt angle θ Ｔ (However, ｂ is the beam length of the laser beam, ０ r is the world coordinate system Σ ０ The spatial coordinates of the laser spot of the laser beam in ０ r = ０ T Ｓｉｎｉ Ｓｉｎｉ T Ｓ Ｓ r.) A real world pointing method comprising:

6. The controller calculates a homogeneous transformation matrix by regarding a translation vector of the controller as zero when the controller changes from the first state to the second state. ０ T Ｉｎｉ Ask for 6. The real world pointing method according to claim 5,

7. The controller receives an instruction for a micro-movement of the laser spot from a user, and calculates a coefficient and a homogeneous transformation matrix based on the beam length of the laser beam and the operation angle of the pan-tilt actuator in response to the instruction for the micro-movement of the laser spot. ０ T Ｓｉｎｉ and Ｓｉｎｉ T Ｓ and calculating offset amounts of the pan angle and tilt angle of the pan-tilt actuator by multiplying the offset amount by a rotation matrix included in the offset amount, and instructing the pan-tilt actuator via wireless communication the absolute amounts of the pan angle and tilt angle that reflect the calculated offset amount or the offset amount; The pan-tilt actuator fine-tunes its own pan and tilt angles by an offset amount or an absolute amount instructed by the controller.

7. The real world pointing method according to claim 5, wherein the real world pointing method further comprises:

8. The instruction for the minute movement of the laser spot is an inching operation in any direction of the up, down, left, or right of the laser spot.

8. The real world pointing method according to claim 7,

9. A smartphone application that causes a smartphone having a camera and a motion sensor capable of capturing optical images and measuring depth to function as a controller in a system that points to a target in the real world with a laser beam emitted from a laser pointer whose direction can be changed by a pan-tilt actuator, A captured image of a visual marker attached to a non-movable portion of the pan-tilt actuator is analyzed to obtain a world coordinate system Σ in a first state in which the camera is facing the pan-tilt actuator. ０ A homogeneous transformation matrix representing the position and orientation of the controller in ０ T Ｓｉｎｉ The means to seek A change in the attitude of the controller when the state changes from the first state to a second state in which the camera is facing the target is detected by the motion sensor, and a homogeneous transformation matrix relating to the change from the first state to the second state is calculated. Ｓｉｎｉ T Ｓ The means to seek In the second state, the distance to the target measured by the camera is converted into the coordinate system Σ of the controller. Ｓ The spatial coordinates of the target in Ｓ A means for determining r; and Inverse kinematics (θ Ｐ , θ Ｔ , l ｂ ) Ｔ =IK( ０ r) to pan angle θ Ｐ and tilt angle θ Ｔ and instruct the pan-tilt actuator (where l ｂ is the beam length of the laser beam, ０ r is the world coordinate system Σ ０ The spatial coordinates of the laser spot of the laser beam in ０ r = ０ T Ｓｉｎｉ Ｓｉｎｉ T Ｓ Ｓ r.) As a means to make smartphones function A smartphone app that features

Images