Positioning method and positioning system

Abstract

To provide guidance of an accurate loading position of a loading article relative to an object to be loaded by using a projector, without depending on a position and a posture of the projector.SOLUTION: A control device 30 for controlling a projector includes a virtual space management section 311, a calculation section 312, and a projection control section 313. The virtual space management section 311 disposes virtual objects corresponding respectively to a projector, a loaded object, and a loading article in a virtual space, while maintaining the relative positional relationships in a real space between the projector, the loaded object, and the loading article. The calculation section 312 sets an axis being parallel to a vertical axis in a virtual space, on a guide axis at the time of projecting a guide image for providing guidance of a loading position of a loading article by using orthographic projection, in order to calculate an orthographic projection guide image corresponding to the guide image projected using orthographic projection on the virtual object corresponding to the loading article in the virtual space, on the basis of the relative positional relationships of the virtual objects. The projection control section 313 controls the projector such that the orthographic projection guide image is projected on the loading article.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a positioning method and a positioning system.

Background Art

[0002] In the process of assembling a product or a building, when positioning a load on a load-carrying object, various techniques have been proposed to assist in positioning by displaying a marker for guiding the position on the upper surface of the load-carrying object using a laser light source or a projector. For example, Patent Document 1 discloses a method for positioning a load that projects a marker image by a projector in order to position an insert on the inner surface of a lower mold at a predetermined position. This marker image includes a positioning line having a predetermined width. The insert is an example of a load, and the inner surface of the lower mold is an example of a load-carrying portion.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technology disclosed in Patent Document 1, unless the placement position of the projector is vertically above the loaded object and the loaded item, and the optical axis of the projection light from the projector is parallel to the vertical line of the loaded object and the loaded item, the projection is from an oblique angle, so the marker is projected distorted and is not displayed at the correct position. For example, when the loaded object is introduced in a tilted state from the horizontal, the marker is not displayed at the desired position. Also, when there are irregularities on the upper surface of the loaded object, the marker is projected distorted and the correct position cannot be guided. Furthermore, since the projected image of the projector is projected by perspective projection, unless the loaded item is directly placed on the loaded object, due to the influence of the projection, the marker is displayed at a position shifted from the desired position and the correct position cannot be guided.

Means for Solving the Problems

[0005] One aspect of the positioning method of the present disclosure is a method for positioning a second object loaded on a first object in a first direction in real space, including a first virtual object corresponding to the first object, a virtual projection device corresponding to a projection device that projects a guide image for guiding the loading position of the second object with respect to the first object onto at least one of the first object and the second object, and a virtual detection device corresponding to a detection device that detects at least the second object in the real space. The first virtual object, the virtual projection device, and the virtual detection device are arranged in a virtual space while maintaining their relative positional relationships in the real space, and a second virtual object corresponding to the second object is arranged in the virtual space according to the detection result by the detection device. Among the three axes constituting the three-dimensional coordinate system associated with the first virtual object, the axis parallel to the first direction is set as the guide axis when orthographically projecting the guide image. A orthographic guide image corresponding to the guide image orthographically projected from the virtual projection device along the guide axis onto the second virtual object is generated based on the relative positional relationship between the second virtual object arranged in the virtual space and the virtual projection device according to the detection result by the detection device. The projection device is controlled so that the orthographic guide image is projected onto the second object.

[0006] Also, one aspect of the positioning system of the present disclosure is a positioning system for determining the position of a second object mounted on a first object in a first direction in the real space, the projection device projecting a guide image for guiding the mounting position of the second object with respect to the first object onto at least one of the first object and the second object, the detection device detecting at least the second object in the real space, and a processing device, wherein the processing device arranges a first virtual object corresponding to the first object, a virtual projection device corresponding to the projection device, and a virtual detection device corresponding to the detection device in a virtual space while maintaining their relative positional relationships in the real space, arranges a second virtual object corresponding to the second object in the virtual space according to the detection result by the detection device, sets an axis parallel to the first direction among three axes constituting a three-dimensional coordinate system associated with the first virtual object as a guide axis when orthographically projecting the guide image, generates an orthographic guide image corresponding to the guide image orthographically projected from the virtual projection device along the guide axis onto the second virtual object arranged in the virtual space based on the relative positional relationship between the second virtual object and the virtual projection device arranged in the virtual space according to the detection result by the detection device, and controls the projection device so that the orthographic guide image is projected onto the second object.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Mode for Carrying Out the Invention

[0008] The embodiments described below are subject to various technically preferable limitations. However, the embodiments of the present disclosure are not limited to the forms described below. 1. First Embodiment FIG. 1 is a diagram showing a configuration example of a positioning system 1 according to the first embodiment of the present disclosure. The positioning system 1 is a system for assisting in arranging the load OB2 at the correct position with respect to the object to be loaded OB1 in the process of assembling a product by loading the load OB2 vertically with respect to the object to be loaded OB1. The vertical direction is an example of the loading direction for loading the load OB2 with respect to the object to be loaded OB1, and is, for example, the direction of gravity acting on the object to be loaded OB1 and the load OB2. The object to be loaded OB1 is the lower part of the product manufactured in the above assembly process, and the load OB2 is the upper part of the product. The space in which this assembly process is executed is an example of the actual space in the present disclosure, and the vertical direction in the actual space is an example of the first direction in the present disclosure. Further, the object to be loaded OB1 is an example of the first object in the present disclosure, and the load OB2 is an example of the second object in the present disclosure.

[0009] As shown in FIG. 1, the object OB1 to be loaded in this embodiment is a component formed in a rectangular shape when viewed from above, and has protrusions P1 to P4 protruding from the upper surface. In the above assembly process in this embodiment, the object OB1 to be loaded is fixed in a posture with the upper surface horizontal on, for example, a workbench or the like.

[0010] As shown in FIG. 1, the loaded object OB2 is a component formed in a rectangular shape when viewed from above, similar to the object OB1 to be loaded, and has four protrusions on the upper surface corresponding to the protrusions P1 to P4 respectively. Further, on the bottom surface of the loaded object OB2, four recesses corresponding to the protrusions P1 to P4 one by one are provided. In FIG. 1, the illustration of these four recesses is omitted. The operator in the above assembly process adjusts the position and posture of the loaded object OB2 with respect to the object OB1 to be loaded so that each of the protrusions P1 to P4 and the above four recesses correspond to each other, and lowers the loaded object OB2 in the vertical direction with respect to the object OB1 to be loaded. That is, the protrusions P1 to P4 serve as a plurality of reference points during positioning. In this embodiment, the number of reference points is 4, but the number of reference points may be 2 or 3, or 5 or more.

[0011] Also, in this embodiment, as shown in FIG. 1, AR (Augmented Reality) markers AM1 to AM4 are pre-applied on the upper surface of the loaded object OB2. AR is a technology for displaying various contents superimposed on the real space. Generally, an AR marker is used as information indicating the display position of the content in the real space. Although details will be described later, the AR markers AM1 to AM4 are used for detecting the position and posture of the loaded object OB2 that change moment by moment in the above assembly process.

[0012] Conventionally, when loading the load OB2 onto the object to be loaded OB1, the operator had to position the load OB2 while checking the current position of the load OB2 relative to the object to be loaded OB1 each time, such as by looking into the space between the load OB2 and the object to be loaded OB1. Therefore, when the load OB2 is large or heavy, there are inconveniences such as the need for multiple operators to hold the load OB2 and coordinate their actions while working.

[0013] If the positions of the plurality of reference points described above can be continuously displayed on the load OB2, positioning becomes easier. For example, if a plurality of laser pointers are arranged vertically above the plurality of reference points, the positions of the plurality of reference points can be displayed on the load OB2, but it is not possible to respond opportunely when any of the plurality of reference points is changed. The positioning system 1 is a system that guides a plurality of preset reference points to the operator using the projector 10. Although details will be described later, according to the present embodiment, the accurate positions of each of the plurality of reference points can be guided to the operator regardless of the arrangement position and the direction of the optical axis of the projector 10. As shown in FIG. 1, the positioning system 1 includes, in addition to the projector 10, an imaging device 20 and a control device 30. Each of the projector 10 and the imaging device 20 is connected to the control device 30 by wire or wirelessly.

[0014] The imaging device 20 is, for example, a video camera. The imaging device 20 includes an imaging lens (not shown) and an image sensor (not shown) on which light from the imaging lens forms an image. The image sensor is, for example, a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, but is not particularly limited. The imaging device 20 is arranged in the real space at a position and posture such that the entire upper surface of the object to be loaded OB1 is within the imaging field of view. In FIG. 1, the imaging range of the imaging device 20 is indicated by a two-dot chain line. The installation position of the imaging device 20 in the real space is not limited to directly above the object to be loaded OB1, and the optical axis of the imaging lens of the imaging device 20 does not have to be parallel to the vertical axis. The imaging device 20 captures an image within the imaging field of view at any time and transmits image data representing the captured image to the control device 30. When the object to be loaded OB2 is introduced into the imaging field of view of the imaging device 20, the position and posture of the object to be loaded OB2 in the real space are estimated based on the markers AM1 to AM4 shown in the captured image of the imaging device 20. That is, the imaging device 20 serves as a detection device for detecting the object to be loaded OB2.

[0015] The projector 10 includes a display panel (not shown) and a projection lens for enlarging the display image displayed on the display panel and projecting it at least onto the object to be loaded OB1. The display panel is, for example, a liquid crystal panel having a plurality of pixels for forming a display image, but is not particularly limited. The projector 10 is arranged in the real space at a position and posture such that the entire upper surface of the object to be loaded OB1 is within the projection range. In FIG. 1, the projection range of the projector 10 is indicated by a one-dot chain line. The installation position of the projector 10 in the real space is not limited to directly above the object to be loaded OB1, and the optical axis of the imaging lens of the projector 10 does not have to be parallel to the vertical axis. The projector 10 projects a guide image G1 for guiding an operator to the loading position of the object to be loaded OB2 with respect to the object to be loaded OB1 under the control of the control device 30. The projector 10 is an example of the projection device in the present disclosure.

[0016] FIG. 2 is a diagram showing an example of the guide image G1. FIG. 2 illustrates a state in which the guide image G1 is projected onto the upper surface of the object to be loaded OB1. However, when the operator introduces the load OB2 above the object to be loaded OB1, the guide image G1 is reflected on the load OB2. As shown in FIG. 2, the guide image G1 includes markers M1 to M4 and guide lines L1 and L2. Each of the markers M1 to M4 is arranged at a position corresponding to the protrusions P1 to P4 when the upper surface of the object to be loaded OB1 is viewed from directly above, and is a rectangle that is slightly larger than the upper surface of each of the protrusions P1 to P4. Each of the markers M1 to M4 is for guiding the operator to the position of the reference point when correctly loading the load OB2 onto the object to be loaded OB1. The guide line L1 is a line segment along the long side of the aforementioned rectangle, and the guide line L2 is a line segment along the long side of the aforementioned rectangle. The intersection of the guide line L1 and the guide line L2 corresponds to the center of the upper surface when the upper surface of the object to be loaded OB1 is viewed from directly above.

[0017] In this embodiment, the parameters around the lenses of each of the projector 10 and the imaging device 20 (so-called, internal parameters), and the parameters representing the relative position and orientation of the projector 10 and the imaging device 20 (so-called, external parameters) are calibrated in advance prior to the start of the assembly process and are known. These internal parameters and external parameters are stored in advance in the control device 30. The external parameters include parameters for associating the projector coordinate system with the real space coordinate system and parameters for associating the camera coordinate system with the real space coordinate system. The details of each of the real space coordinate system, the projector coordinate system, and the camera coordinate system are as follows.

[0018] FIG. 3 is a diagram for explaining a real space coordinate system, a projector coordinate system, and a camera coordinate system. The real space coordinate system is a three-dimensional coordinate system AX1 with the center OP of the upper surface of the object OB1 to be loaded as the origin, the vertical direction as the z-axis, the long side direction of the upper surface of the object OB1 to be loaded as the x-axis, and the short side direction of the upper surface of the object OB1 to be loaded as the y-axis. The upper surface of the object OB1 to be loaded is the loading surface on which the object OB2 is loaded. The real space coordinate system may be the local coordinate system of the object OB1 to be loaded. The projector coordinate system is a three-dimensional coordinate system AX2 with the principal point PP of the projection lens of the projector 10 as the origin, the optical axis of the projection lens as the z-axis, the horizontal scanning direction in the projection image of the projector 10 as the x-axis, and the vertical scanning direction as the y-axis. The projector coordinate system may be the local coordinate system of the projector 10. The camera coordinate system is a three-dimensional coordinate system AX3 with the principal point MP of the imaging lens of the imaging device 20 as the origin, the optical axis of the imaging lens as the z-axis, the horizontal scanning direction in the projection image of the imaging device 20 as the x-axis, and the vertical scanning direction as the y-axis. The camera coordinate system may be the local coordinate system of the imaging device 20.

[0019] As an example of the calibration method of the external parameters, a method based on the principle of PnP (Perspective n-Points) can be mentioned. For example, the calibration method of the parameters for associating the camera coordinate system and the real space coordinate system is as follows. The user who executes this calibration method or the control device 30 described later first projects a measurement image such as a phase shift image onto the upper surface of the object OB1 to be loaded by the projector 10. Then, the user or the control device 30 described later causes the imaging device 20 to image the object OB1 to be loaded in a state where the measurement image is projected.

[0020] Next, the user or the control device 30 described later sets a plurality of measurement points on the upper surface of the object OB1 on which the measurement image is projected, and obtains the coordinates (X, Y, Z) of each measurement point in the real space coordinate system and the coordinates (mx, my) in the captured image by the imaging device 20. The coordinates (X, Y, Z) are three-dimensional coordinates, and the coordinates (mx, my) in the captured image are two-dimensional coordinates. The coordinates (mx, my) in the captured image are the coordinates of the pixels indicating the measurement points included in the captured image. This captured image corresponds to the front clipping plane F1 shown in FIG. 4. More specifically, the tip is located at the principal point MP of the imaging lens of the imaging device 20, and the front clipping plane F1 in the frustum of a cone S1 including the object OB1 is reflected on the imaging surface of the imaging device 20 through the imaging lens, thereby obtaining a captured image. The imaging surface is, for example, a surface on which a plurality of pixels of a CMOS sensor are arranged. The three-dimensional position of the measurement point in the camera coordinate system is represented as (mx, my, f) using the coordinates (mx, my) in the captured image and the distance f from the principal point MP to the front clipping plane F1.

[0021] The position (X, Y, Z) of the measurement point in the real space coordinate system and the position (mx, my) of the measurement point in the camera coordinate system are associated with each other using the internal parameter matrix F shown in Equation (2) and the matrix RT shown in Equation (3) as shown in Equation (1) below. Note that s on the left side of Equation (1) is a scaling factor for making the third row on the left side of Equation (1) equal to 1. In the internal parameter matrix F, fx represents the focal length in the horizontal scanning direction among the focal lengths of the imaging lens of the imaging device 20, and fy represents the focal length in the vertical scanning direction among the focal lengths of the imaging lens. Also, cx and cy are the x-coordinate and y-coordinate on the captured image of the intersection point PX of the straight line passing through the principal point MP and along the z-axis and the front clipping plane F1.

Equation

Equation

Equation

[0022] The matrix RT includes a 3x3 rotation matrix composed of components r11 to r33 and a 3x1 translation matrix composed of components t1 to t3. In this embodiment, this rotation matrix and translation matrix are external parameters that associate the camera coordinate system with the real space coordinate system. The rotation matrix indicates the inclination, i.e., the attitude, of the object OB1 with respect to the imaging device 20, and the translation matrix indicates the position of the object OB1 with respect to the imaging device 20. The user or the control device 30 described later can calculate the matrix RT, for example, by solving a system of simultaneous equations obtained by substituting the coordinates in the camera coordinate system and the coordinates in the real space coordinate system for six measurement points into Equation (1). Similarly, for the external parameters that associate the projector coordinate system with the real space coordinate system, the user or the control device 30 described later can calculate them using the position (X, Y, Z) of the measurement point and the position (mx, my) of the measurement point in the projector coordinate system. In this case, the position (mx, my) is the position corresponding to the measurement point on the display panel of the projector 10. The position (mx, my) of the measurement point in the projector coordinate system is the coordinates of the pixels of the display panel.

[0023] When calculating the external parameters that associate the projector coordinate system with the real space coordinate system, the user or the control device 30 described later may use the position (X, Y, Z) of the measurement point, the external parameters that associate the camera coordinate system with the real space coordinate system, and the external parameters that associate the camera coordinate system with the projector coordinate system. That is, the user or the control device 30 described later may convert the coordinates (mx, my) in the camera coordinate system to the coordinates (mx, my) in the projector coordinate system according to the correspondence relationship that associates the camera coordinate system with the projector coordinate system in order to calculate the position (mx, my) of the measurement point in the projector coordinate system. The correspondence relationship that associates the camera coordinate system with the projector coordinate system is a matrix that associates the pixels of the captured image with the pixels of the display panel.

[0024] The control device 30 is a device that controls the projection of the guide image G1 by the projector 10. The control device 30 is, for example, a personal computer. FIG. 5 is a diagram showing a configuration example of the control device 30. As shown in FIG. 5, the control device 30 includes a processing device 310, a communication device 320, a display device 330, an input device 340, and a storage device 350. The processing device 310 is one or more processors. The processing device 310 is, for example, a CPU (Central Processing Unit). The processing device 310 functions as the control center of the control device 30 by operating according to the program PRA stored in the storage device 350.

[0025] The communication device 320 is a device that performs wireless communication or wired communication with other devices, and includes, for example, an interface circuit. Specific examples of other devices that communicate with the communication device 320 include the projector 10 and the imaging device 20.

[0026] The display device 330 includes, for example, a panel display such as a liquid crystal display, a plasma display, or an organic EL display, and its driving circuit. The display device 330 displays the image of the aforementioned virtual space under the control of the processing device 310. The input device 340 includes a pointing device such as a mouse and a keyboard having a plurality of operators such as numeric keys. The input device 340 receives a user's operation on the pointing device or the keyboard, and outputs operation content data indicating the received operation to the processing device 310. Thereby, the user's operation on the input device 340 is transmitted to the processing device 310.

[0027] The storage device 350 is a recording medium readable by the processing device 310. The storage device 350 includes, for example, a non-volatile memory and a volatile memory. The non-volatile memory is, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, a RAM (Radom Access Memory). In addition to the aforementioned internal parameters and external parameters, the non-volatile memory of the storage device 350 stores first object data, second object data, and image data representing the guide image G1. Hereinafter, the image data representing the guide image G1 is referred to as guide image data.

[0028] The first object data is data representing the shape and size of the object to be loaded OB1. The second virtual object data represents the shape and size of the load OB2, and also indicates the relative position of each of the AR markers AM1 to AM4 as viewed from the center of the upper surface of the load OB2. If the position of each of the AR markers AM1 to AM4 in the real space coordinate system is known, based on the second object data, the position and orientation of the load OB2 in the real space coordinate system can be known.

[0029] Also, various programs are stored in the non-volatile memory. Examples of the various programs stored in the non-volatile memory include a kernel program and a program PRA. In FIG. 5, the illustration of the kernel program is omitted. The kernel program is a program for causing the processing device 310 to implement an OS (Operating System). When the power of the control device 30 is turned on, the processing device 310 reads the kernel program from the non-volatile memory into the volatile memory and starts executing the read kernel program. The processing device 310 operating according to the kernel program starts executing another program when instructed to start executing the other program by an operation on the input device 340.

[0030] For example, when the start of the execution of program PRA is instructed by an operation on the input device 340, the processing device 310 reads program PRA from the non-volatile memory into the volatile memory and starts the execution of the program PRA read into the volatile memory. The processing device 310 operating according to program P functions as a virtual space management unit 311, a calculation unit 312, a projection control unit 313, and a display control unit 314 shown in FIG. 5. That is, each of the virtual space management unit 311, the calculation unit 312, the projection control unit 313, and the display control unit 314 shown in FIG. 5 is a software module realized by operating the processing device 310 according to program PRA. The roles of each of the virtual space management unit 311, the calculation unit 312, the projection control unit 313, and the display control unit 314 shown in FIG. 5 are as follows.

[0031] The virtual space management unit 311 arranges a first virtual object corresponding to the object to be loaded OB1 and a virtual projection device corresponding to the projector 10 in a three-dimensional virtual space while maintaining the relative positional relationship in the real space. A specific example of the three-dimensional virtual space is a CAD (Computer-Aided Design) space. In the present embodiment, the virtual space management unit 311 arranges the first virtual object, the virtual projection device, and the virtual imaging device in the virtual space such that the relative positional relationship (position and orientation) of the first virtual object, the virtual projection device, and the virtual imaging device in the virtual space is equal to the relative positional relationship of the object to be loaded OB1, the projector 10, and the imaging device 20 in the real space based on the above-described external parameters.

[0032] Hereinafter, a three-dimensional coordinate system with the center of the first virtual object as the origin, the vertical direction as the z-axis, the long side direction of the upper surface of the first virtual object as the x-axis, and the short side direction of the upper surface of the first virtual object as the y-axis is referred to as a virtual space coordinate system. The virtual space coordinate system is an example of a three-dimensional coordinate system associated with the first virtual object. Also, a three-dimensional coordinate system with the principal point of the projection lens of the virtual projection device as the origin, the optical axis of the virtual projection device as the z-axis, the horizontal scanning direction in the projection image of the virtual projection device as the x-axis, and the vertical scanning direction as the y-axis is referred to as a virtual projector coordinate system. Further, a three-dimensional coordinate system with the principal point of the projection lens of the virtual imaging device as the origin, the optical axis of the virtual imaging device as the z-axis, the horizontal scanning direction in the captured image of the virtual imaging device as the x-axis, and the vertical scanning direction as the y-axis is referred to as a virtual camera coordinate system. Since the relative positional relationships of the first virtual object, the virtual projection device, and the virtual imaging device in the virtual space are equal to the relative positional relationships of the object OB1 to be loaded, the projector 10, and the imaging device 20 in the real space, the relative positional relationships of the virtual space coordinate system, the virtual projector coordinate system, and the virtual camera coordinate system are equal to the relative positional relationships of the real space coordinate system, the projector coordinate system, and the camera coordinate system, respectively. Therefore, each of the virtual space coordinate system, the virtual projector coordinate system, and the virtual camera coordinate system is associated with each other using the above-described external parameters.

[0033] When the operator introduces the object OB2 to be loaded into the imaging field of view of the imaging device 20 in order to load the object OB2 onto the object OB1 to be loaded, the virtual space management unit 311 arranges a second virtual object corresponding to the object OB2 in the virtual space. More specifically, when the virtual space management unit 311 recognizes all of the AR markers AM1 to AM4 arranged on the upper surface of the object OB2 by analyzing the captured image by the imaging device 20, based on the above-described PnP principle, the relative position and relative orientation of the object OB2 with respect to the imaging device 20 are estimated.

[0034] As described above, in this embodiment, at the start of the assembly process, since the internal parameter matrix F and the matrix RT are known, for each of the AR markers AM1 to AM4, by substituting the coordinates (mx, my) in the camera coordinate system into Equation (1), the coordinates (X, Y, Z) in the real space coordinate system are obtained. Based on the coordinates in the real space coordinate system obtained for each of the AR markers AM1 to AM4 and the second object data, the virtual space management unit 311 estimates the relative position and relative orientation of the loaded object OB2 with respect to the imaging device 20, and based on this estimation result, the second virtual object in the virtual space is arranged in the virtual space so that the relative positional relationship between the second virtual object and the virtual imaging device is equal to the relative positional relationship between the loaded object OB2 and the imaging device 20 in the real space.

[0035] As described above, in the process of loading the loaded object OB2 onto the object to be loaded OB1, since the position and orientation of the loaded object OB2 in the real space change from moment to moment, the relative positional relationship between the loaded object OB2 introduced into the field of view of the imaging device 20 and the imaging device 20 also changes from moment to moment. According to this embodiment, even if the relative positional relationship between the loaded object OB2 and the imaging device 20 changes, based on the AR markers AM1 to AM4, the relative positional relationship between the loaded object OB2 and the imaging device 20 can be estimated at any time, and the position and orientation of the second virtual object in the virtual space can be made to follow the position and orientation of the loaded object OB2 in the real space.

[0036] The calculation unit 312 sets an axis that passes through the center of the first virtual object in the virtual space coordinate system and is along the z-axis in the virtual space coordinate system as the guide axis when orthographically projecting the guide image G1 onto the projection target object. The z-axis in the virtual space coordinate system is an example of an axis along the z-axis in the real space, that is, along the first direction. FIG. 6 is a diagram for explaining orthographic projection and perspective projection. As shown in FIG. 6, in orthographic projection, a three-dimensional object is projected onto the projection plane as it is, while in perspective projection, a three-dimensional object is projected so as to converge at one point. Also, the projection target object is the first virtual object if it is before the entire loaded object OB2 is introduced into the imaging field of the imaging device 20, and is the second virtual object after the entire loaded object OB2 is introduced into the imaging field of the imaging device 20. At least, if the entire loaded object OB2 is not introduced into the imaging field of the imaging device 20, it is impossible to recognize all of the AR markers AM1 to AM4.

[0037] Based on the relative positional relationship between the projection target object and the virtual projection device, the calculation unit 312 calculates image data representing an orthographic guide image corresponding to the guide image G1 orthographically projected from the virtual projection device along the guide axis onto the projection target object. For example, when rendering information such as coloring is given to the projection target object (in this embodiment, the first virtual object or the second virtual object), the calculation unit 312 updates the rendering information based on the guide image data, thereby orthographically projection-mapping the guide image G1 onto the surface of the projection target object. Then, the calculation unit 312 generates image data representing the orthographic guide image based on an image obtained by looking into the projection target object onto which the guide image G1 is projection-mapped on the surface with the frustum of the virtual projection device. As described above, even if the position and orientation of the loaded object OB2 in the real space change, the position and orientation of the second virtual object in the virtual space can be made to follow the position and orientation of the loaded object OB2 in the real space. Therefore, when the second virtual object is the projection target object, even if the position and orientation of the loaded object OB2 change, it is possible to generate an orthographic guide image following the change.

[0038] The projection control unit 313 controls the projector 10 to project the guide image G1 onto the projection target by orthographic projection. The projection control unit 313 causes the projector 10 to project the guide image onto the projection target by orthographic projection by transmitting the image data representing the orthographic projection guide image to the projector 10. For example, if the load OB2 is before being introduced into the imaging field of view of the imaging device 20, the image data of the orthographic projection guide image corresponding to the guide image G1 orthographically projected onto the first virtual object along the guide axis is generated by the calculation unit 312. Thus, as shown in FIG. 7, the orthographic projection guide image G2 is projected from the projector 10 onto the object to be loaded OB1. On the other hand, after the entire load OB2 is introduced into the imaging field of view of the imaging device 20, the image data of the orthographic projection guide image corresponding to the guide image G1 orthographically projected onto the second virtual object along the guide axis is generated by the calculation unit 312. Thus, as shown in FIG. 8, the orthographic projection guide image G2 is projected from the projector 10 onto the load OB2. Since the orthographic projection guide image G2 is displayed on the upper surface of the load OB2, the operator can position the load OB2 so that the four protrusions on the upper surface of the load OB2 are each within the inside of the markers M1 to M4 without peering into the space between the load OB2 and the object to be loaded OB1. Since the orthographic projection guide image G2 is displayed on the upper surface of the load OB2, as shown in FIG. 9, when the load OB2 is tilted with respect to the object to be loaded OB1, or as shown in FIG. 10, when the load OB2 is displaced with respect to the object to be loaded OB1 in at least one of the long side direction or the short side direction of the upper surface of the object to be loaded OB1, the load OB2 can be positioned so that the four protrusions on the upper surface of the load OB2 are each within the inside of the markers M1 to M4. Note that FIG. 10 illustrates a state in which the load OB2 is displaced in the long side direction of the upper surface of the object to be loaded OB1 with respect to the object to be loaded OB1.

[0039] The display control unit 314 displays on the display device 330 a virtual image, which is an image of a virtual space in which the first virtual object, the second virtual object, the virtual projection device, and the virtual detection device are arranged. By displaying the virtual image on the display device 330, the user of the positioning system 1 can check the state of the operation of loading the load OB2 onto the object to be loaded OB1 by checking the virtual image displayed on the display device 330. As an example of the user of the positioning system 1, there is a work supervisor who gives various instructions to a worker who performs the operation of loading the load OB2 onto the object to be loaded OB1. Even if the imaging device 20 images the state of the above operation, there will always be blind spots, but there are no blind spots in the case of a virtual image, and it becomes smoother to confirm that the positioning is being performed correctly. Note that the display control unit 314 may display the virtual image and the real image captured by the imaging device 20 side by side on the display device 330. According to this aspect, the user can check the state of the above operation by comparing the virtual image and the real image, and it becomes easier to confirm whether the positioning is being performed correctly.

[0040] Further, the processing device 310 operating according to the program PRA executes a positioning method that prominently shows the features of the present disclosure. At the time of disclosure of this positioning method, the above-described external parameters have been calculated. FIG. 11 is a flowchart showing the flow of this positioning method. As shown in FIG. 11, this positioning method includes each process of a virtual space initialization process SA110, a guide axis setting process SA120, a calculation process SA130, a projection control process SA140, a display control process SA150, and a determination process SA160. The processing contents of each of the virtual space initialization process SA110, the guide axis setting process SA120, the calculation process SA130, the projection control process SA140, the display control process SA150, and the determination process SA160 are as follows.

[0041] In the virtual space initialization process SA110, the processing device 310 functions as a virtual space management unit 311. In the virtual space initialization process SA110, the processing device 310 arranges in the virtual space a first virtual object corresponding to the object to be loaded OB1, a virtual projection device corresponding to the projector 10, a virtual detection device corresponding to the imaging device 20, while maintaining the relative positional relationship among the projector 10, the imaging device 20, and the object to be loaded OB1 in the real space based on the aforementioned external parameters. Note that the positioning method of the present disclosure is a process executed prior to the virtual space initialization process SA110, and may include a process of estimating external parameters using the principle of the aforementioned PnP.

[0042] In the guide axis setting process SA120, the processing device 310 functions as a calculation unit 312. In the guide axis setting process SA120, the processing device 310 sets, as the guide axis for orthogonally projecting the guide image G1 onto the object to be projected, an axis passing through the center of the first virtual object in the virtual space coordinate system and along the z-axis in the virtual space coordinate system.

[0043] In the calculation process SA130, the processing device 310 functions as a virtual space management unit 311 and a calculation unit 312. In the calculation process SA130, the processing device 310 first functions as a virtual space management unit 311 and determines whether the entire loaded object OB2 has been introduced into the imaging field of view of the imaging device 20 based on the presence or absence of detection of the AR markers AM1 to AM4. When all of the AR markers AM1 to AM4 are detected from the captured image by the imaging device 20, the processing device 310 determines that the entire loaded object OB2 has been introduced into the imaging field of view of the imaging device 20. On the contrary, when at least one of the AR markers AM1 to AM4 is not detected from the captured image by the imaging device 20, the processing device 310 determines that the entire loaded object OB2 has not been introduced into the imaging field of view of the imaging device 20.

[0044] When it is determined that the entire loaded object OB2 has not been introduced into the imaging field of view of the imaging device 20, the processing device 310 functions as a calculation unit 312, and based on the relative positional relationship between the first virtual object and the virtual projection device, calculates image data representing an orthographic guide image G2 corresponding to the guide image G1 projected orthographically from the virtual projection device along the guide axis onto the first virtual object, based on the guide image data. When it is determined that the entire loaded object OB2 has been introduced into the imaging field of view of the imaging device 20, the processing device 310 first estimates the relative positional relationship between the imaging device 20 and the loaded object OB2, and places the second virtual object in the virtual space while maintaining the relative positional relationship. Next, the processing device 310 functions as a calculation unit 312, and based on the relative positional relationship between the second virtual object and the virtual projection device, calculates image data representing an orthographic guide image G2 corresponding to the guide image G1 projected orthographically from the virtual projection device along the guide axis onto the second virtual object, based on the guide image data.

[0045] In the projection control process SA140, the processing device 310 functions as a projection control unit 313. In the projection control process SA140, the processing device 310 causes the projector 10 to project the orthographic guide image G2 onto the projection target by providing the image data calculated in the calculation process SA130. In the display control process SA150, the processing device 310 functions as a display control unit 314. In the display control process SA150, the processing device 310 displays a virtual image, which is an image of the virtual space, on the display device 330. Note that the execution order of the projection control process SA140 and the display control process SA150 may be swapped, and the display control process SA150 may be executed prior to the projection control process SA140.

[0046] In the determination process SA160, the processing device 310 determines whether the end of this positioning method is instructed by an operation on the input device 340. If the end of this positioning method is instructed by an operation on the input device 340, the determination result of the determination process SA160 is "Yes". When the determination result of the determination process SA160 is "Yes", the processing device 310 ends this positioning method. If the end of this positioning method is not instructed by an operation on the input device 340, the determination result of the determination process SA160 is "No". When the determination result of the determination process SA160 is "No", the processing device 310 re-executes the processes after the calculation process SA130.

[0047] As described above, according to this embodiment, an operator can place the load OB2 at a desired position according to the orthographic projection guide image G2 projected on the load OB2 without looking into the space between the load OB2 and the object to be loaded OB1. Further, according to this embodiment, since the orthographic projection guide image G2 that would be reflected on the load OB2 when the guide image G1 is projected orthographically is projected from the projector 10 onto the load OB2, even when the projector 10 is installed at a position deviated from directly above the object to be loaded OB1 in the real space, a decrease in positioning accuracy can be suppressed. Note that the display control unit 314 in this embodiment is not an essential component of the present disclosure and can be omitted. Similarly, the display control process A150 can also be omitted.

[0048] 2. Other Embodiments (1) Second Embodiment The calculation unit 312 may change the projection mode of the orthographic projection guide image G2 according to the number of markers among the markers M1 to M4 included in the guide image G1 projected onto the second virtual object in the virtual space that are projected onto the positions of the corresponding reference points. For example, when all of the markers M1 to M4 are projected onto the positions of the corresponding reference points, the color of each of the markers M1 to M4 is set to green, and when at least one marker is not projected onto the position of the corresponding reference point, an example is a mode in which the color of each of the markers M1 to M4 is set to red. Also, if the number of markers projected onto the position of the corresponding reference point is 0, the color of each of the markers M1 to M4 is set to red, if the number of markers projected onto the position of the corresponding reference point is 1 to 3, the color of each of the markers M1 to M4 is set to yellow, and when all of the markers are projected onto the position of the corresponding reference point, a mode in which the color of each of the markers M1 to M4 is set to green is also conceivable. According to these modes, it becomes possible to easily identify whether the positions of the loaded objects OB2 match for all of the reference points. Also, a mode is conceivable in which the color of the markers projected onto the position of the corresponding reference point is set to green, and the color of the markers not projected onto the position of the corresponding reference point is set to red. According to this mode, the operator can easily identify whether the positions of the loaded objects OB2 match for each reference point. Also, a mode may be adopted in which the shape of the marker, rather than the color of the marker, is changed according to the number of markers projected onto the position of the corresponding reference point. According to the second embodiment, it is possible to particularly improve workability when aligning a plurality of large-sized loaded objects OB2 by a plurality of people, etc.

[0049] (2) Third Embodiment When estimating the relative position and orientation between the imaging device 20 and the loaded object OB2, the virtual space management unit 311 may estimate the relative position and orientation between the imaging device 20 and the loaded object OB2 by using artificial intelligence based on the outer shape or feature points of the loaded object OB2 without using an AR marker. According to this aspect, it is not necessary to attach an AR marker to the loaded object OB2. According to this embodiment, since it is not necessary to attach an AR marker to the loaded object OB2, it is possible to improve the workability in the entire process of manufacturing a product by loading the loaded object OB2 on the object to be loaded OB1 as compared with the first embodiment.

[0050] (3) Fourth Embodiment The number of projectors 10 included in the positioning system 1 in the above embodiment was 1, but it may be 2 or more. For example, when the projection target is larger than the range in which an image can be projected by the projector 10, a plurality of projectors 10 may be used, and different projection areas may be assigned to each projector 10. Thereby, it becomes possible to perform positioning guidance even for a large loaded object OB2 that cannot be projected without using a plurality of projectors. For example, it can be applied to the positioning of sub-assembly parts of large manufactured products such as aircraft and ships. Similarly, when the entire loaded object OB2 does not fit within the imaging field of view of the imaging device 20, a plurality of imaging devices 20 may be used.

[0051] 3. Variations Each of the above embodiments can be modified as follows. (1) In each of the above embodiments, the position and orientation of the object to be loaded OB1 in the real space were fixed, but it is not essential that the position and orientation of the object to be loaded OB1 in the real space be fixed. For example, if an AR marker is given to the object to be loaded OB1 in the same manner as the loaded object OB2, it is possible to estimate the relative position and relative orientation of the object to be loaded OB1 with respect to the imaging device 20 based on the above-described PnP principle.

[0052] (2) In each of the above embodiments, the relative positional relationship between the projector 10 and the imaging device 20 in real space is estimated using the PnP principle, but the relative positional relationship between the projector 10 and the imaging device 20 may be fixed. An example of a case where the relative positional relationship between the projector 10 and the imaging device 20 is fixed is a case where the imaging device 20 is built into the projector 10, such as when the projector 10 is a projector with a built-in camera. When the relative positional relationship between the projector 10 and the imaging device 20 is fixed, the same effect as in the above embodiments can be obtained by storing information indicating this positional relationship in the storage device 350.

[0053] (3) In each of the above embodiments, the virtual space management unit 311, the calculation unit 312, the projection control unit 313, and the display control unit 314 are software modules. However, any one, any two, any three, or all of the virtual space management unit 311, the calculation unit 312, the projection control unit 313, and the display control unit 314 may be a hardware module such as an ASIC (Application Specific Integrated Circuit). Even if any one, any two, any three, or all of the virtual space management unit 311, the calculation unit 312, the projection control unit 313, and the display control unit 314 are a hardware module, the same effects as those of the above embodiments can be achieved.

[0054] (4) The program PRA may be manufactured as a standalone product or may be provided free of charge or for a fee. Specific examples of providing the program PRA include providing the program PRA by writing it to a computer-readable recording medium such as a flash ROM, or providing the program PRA by downloading it via a telecommunications line such as the Internet. By operating a general computer in accordance with the program PRA provided in these ways, it becomes possible to cause the computer to execute the display method of the present disclosure.

[0055] 4. Each of the embodiments exemplified above can be variously modified. Specific modification modes applicable to each of the above embodiments are exemplified below. Two or more modes arbitrarily selected from the following examples can be appropriately combined within a range where they do not conflict with each other.

[0056] 4-1. Modification Example 1 In the above-described embodiment, the object to be loaded OB1 is the lower part of the product manufactured in the above assembly process, and the load OB2 is the upper part of the product, but it is not limited to this mode. For example, the object to be loaded OB1 may be a mounting table for mounting the load OB2. For example, the object to be loaded OB1 may be a platen on which a printing medium as the load OB2 to which ink is applied in a digital inkjet printer is placed. Also, the object to be loaded OB1 and the load OB2 may be film-like articles. That is, the technology according to the present disclosure is applicable to all devices in which a situation occurs where the load OB2 is loaded on the object to be loaded OB1.

[0057] 4-2. Modification Example 2 In the above-described embodiment, the loading direction for loading the load OB2 on the object to be loaded OB1 is the direction of gravity, but it is not limited to this mode. For example, the above loading direction may be a horizontal direction intersecting the direction of gravity.

[0058] 5. Summary of the Present Disclosure The present disclosure is not limited to the above-described embodiments and modification examples, and can be realized in various modes without departing from the gist thereof. For example, the present disclosure can also be realized by the following modes. The technical features in the above embodiments corresponding to the technical features in each of the following modes can be appropriately replaced or combined in order to solve part or all of the problems of the present disclosure or to achieve part or all of the effects of the present disclosure. Also, if the technical feature is not described as essential in this specification, it can be appropriately deleted. Hereinafter, a summary of the present disclosure is appended.

[0059] (Appendix 1) The positioning method of the present disclosure is a method for positioning a second object loaded on a first object in a first direction in real space, including a first virtual object corresponding to the first object, and a virtual projection device corresponding to a projection device that projects a guide image for guiding the loading position of the second object with respect to the first object onto at least one of the first object and the second object, and a virtual detection device corresponding to a detection device that detects at least the second object in the real space. The virtual projection device and the virtual detection device are arranged in a virtual space while maintaining their relative positional relationship in the real space, and a second virtual object corresponding to the second object is arranged in the virtual space according to the detection result of the detection device. Among the three axes constituting the three-dimensional coordinate system associated with the first virtual object, an axis parallel to the first direction is set as a guide axis when orthographically projecting the guide image. A orthographic guide image corresponding to the guide image orthographically projected from the virtual projection device along the guide axis onto the second virtual object is generated based on the relative positional relationship between the second virtual object arranged in the virtual space and the virtual projection device according to the detection result of the detection device. The projection device is controlled so that the orthographic guide image is projected onto the second object. According to this aspect, since the orthographic guide image that would be reflected on the second object when the guide image for guiding the loading position of the second object with respect to the first object in the first direction is orthographically projected is projected from the projection device onto the second object, even when the projection device is installed at a position deviated from directly above the first object in the real space, a decrease in the positioning accuracy of the second object can be suppressed.

[0060] (Appendix 2) A more preferable positioning method is that the guide image includes a plurality of markers for each of a plurality of reference points on the surface of the first object on which the second object is loaded, and generating the orthographic guide image involves changing the projection mode of the orthographic guide image according to the number of markers projected at the positions of the corresponding reference points in the guide image projected orthographically onto the second virtual object. This is the positioning method described in (Appendix 1). According to this aspect, an operator performing the operation of loading the second object onto the first object can easily grasp whether the positioning is correctly performed based on the projection mode of the guide image.

[0061] (Appendix 3) A more preferable positioning method is that changing the projection mode of the orthographic guide image includes making the colors of the markers in the orthographic guide image different when all the markers are projected at the positions of the corresponding reference points and when at least one of the markers is not projected at the position of the corresponding reference point. This is the positioning method described in (Appendix 2). According to this aspect, an operator performing the operation of loading the second object onto the first object can easily grasp whether the positioning is correctly performed based on the colors of the plurality of markers included in the guide image.

[0062] (Appendix 4) Another preferable positioning method includes displaying, on a display device, a virtual image that is an image of the virtual space in which the first virtual object, the second virtual object, the virtual projection device, and the virtual detection device are arranged. This is the positioning method described in any one of (Appendix 1) to (Appendix 3). Since there are no blind spots in the case of a virtual image, according to this aspect, the confirmation that the positioning is correctly performed becomes smooth.

[0063] (Appendix 5) Another more preferable positioning method is characterized in that the detection device is an imaging device that images the first object and the second object, and the virtual image and the real image imaged by the imaging device are displayed on the display device. This is the positioning method described in (Supplementary Note 4). According to this aspect, it is possible to compare the virtual image and the real image to confirm the state of the above operation, and it becomes easier to confirm whether the positioning is correctly performed.

[0064] (Supplementary Note 6) The positioning system of the present disclosure is a positioning system for determining the position of a second object loaded on a first object in a first direction in a real space, including a projection device that projects a guide image for guiding the loading position of the second object with respect to the first object onto at least one of the first object and the second object, a detection device that detects at least the second object in the real space, and a processing device. The processing device arranges a first virtual object corresponding to the first object, a virtual projection device corresponding to the projection device, and a virtual detection device corresponding to the detection device in a virtual space while maintaining the relative positional relationship in the real space, and arranges a second virtual object corresponding to the second object in the virtual space according to the detection result by the detection device. Among the three axes constituting the three-dimensional coordinate system associated with the first virtual object, the axis parallel to the first direction is set as the guide axis when orthographically projecting the guide image. Based on the relative positional relationship of the second virtual object, the virtual detection device, and the virtual projection device arranged in the virtual space according to the detection result by the detection device, a orthographic guide image corresponding to the guide image orthographically projected from the virtual projection device along the guide axis onto the second virtual object is generated, and the projection device is controlled so that the orthographic guide image is projected onto the second object. According to this aspect, similar to the positioning method described in (Supplementary Note 1), even when the projection device is installed at a position deviated from directly above the first object in the real space, it is possible to suppress a decrease in the positioning accuracy of the second object.

Explanation of Reference Numerals

[0065] 1… Positioning system, 10… Projector, 20… Imaging device, 30… Control device, 310… Processing device, 311… Virtual space management unit, 312… Calculation unit, 313… Projection control unit, 314… Display control unit, 320… Communication device, 330… Display device, 340… Input device, 350… Storage device, PRA… Program.

Claims

1. A method for positioning a second object loaded on a first object in a first direction in real space, comprising: placing in a virtual space while maintaining a relative positional relationship in the real space a first virtual object corresponding to the first object, a virtual projection device corresponding to a projection device that projects a guide image for guiding a loading position of the second object with respect to the first object onto at least one of the first object and the second object, and a virtual detection device corresponding to a detection device that detects at least the second object in the real space, and placing a second virtual object corresponding to the second object in the virtual space according to a detection result by the detection device; setting, among three axes constituting a three-dimensional coordinate system associated with the first virtual object, an axis parallel to the first direction as a guide axis when orthographically projecting the guide image; generating an orthographic guide image corresponding to the guide image orthographically projected along the guide axis from the virtual projection device onto the second virtual object based on a relative positional relationship between the second virtual object placed in the virtual space and the virtual projection device according to a detection result by the detection device; controlling the projection device so that the orthographic guide image is projected onto the second object; The positioning method comprising the above steps.

2. The guide image includes a plurality of markers for each of a plurality of reference points on a surface of the first object on which the second object is loaded, Generating the orthographic guide image includes: changing a projection mode of the orthographic guide image according to the number of markers projected at positions of the corresponding reference points in the guide image orthographically projected onto the second virtual object, The positioning method according to claim 1, comprising the above steps.

3. Changing the projection mode of the orthographic guide image includes: changing colors of the markers in the orthographic guide image between a case where all the markers are projected at positions of the corresponding reference points and a case where at least one of the markers is not projected at the position of the corresponding reference point, The positioning method according to claim 2, comprising the above steps.

4. The positioning method according to claim 1, further comprising displaying, on a display device, a virtual image which is an image of the virtual space in which the first virtual object, the second virtual object, the virtual projection device, and the virtual detection device are arranged.

5. The detection device is an imaging device that images the first object and the second object. The positioning method according to claim 4, characterized in that the virtual image and the real image captured by the imaging device are displayed on the display device.

6. A positioning system for determining the position of a second object loaded on a first object in a first direction of a real space, A projection device that projects a guide image for guiding the loading position of the second object with respect to the first object onto at least one of the first object and the second object, A detection device that detects at least the second object in the real space, A processing device, and includes: The processing device: Disposes a first virtual object corresponding to the first object, a virtual projection device corresponding to the projection device, and a virtual detection device corresponding to the detection device in a virtual space while maintaining their relative positional relationships in the real space, and disposes a second virtual object corresponding to the second object in the virtual space according to the detection result by the detection device; Among the three axes constituting the three-dimensional coordinate system associated with the first virtual object, an axis parallel to the first direction is set as a guide axis when orthographically projecting the guide image; Based on the relative positional relationships of the second virtual object, the virtual detection device, and the virtual projection device arranged in the virtual space according to the detection result by the detection device, generates an orthographic guide image corresponding to the guide image orthographically projected from the virtual projection device along the guide axis onto the second virtual object; Controls the projection device so that the orthographic guide image is projected onto the second object, and executes. Positioning system.

Images