Authoring device, authoring procedure and authoring program

Abstract

Authoring device (1, 2), comprising: a user interface unit (11) that receives an operation to determine a target object existing in real space; a target definition unit (12) which uses the user interface unit on a reference plane (S p ) in relation to the target object defines a reference point (p) of a target as the target object; an arrangement position calculation unit (13) which includes a first arrangement level (S q ) specifies, arranged in a position that includes the reference point (p) and on which a virtual object can be placed, based on the reference plane (S) p ) and the reference point (p); and a calculation unit (14) for multiple viewpoints, which includes one or more second arrangement levels (S r1 , S r2 , S r3 ), obtained by rotating the first arrangement plane (S q), specifies on which the virtual object can be arranged, where the authoring device provides information that constitutes the first order level (S q ) and assign the virtual object to each other, and information that defines the second order levels (S r1 , S r2 , S r3 ) and assign the virtual object to each other, outputting it as authoring data.

Description

TECHNICAL FIELD

[0001] The present invention relates to an authoring device, an authoring method and an authoring program. STATE OF THE ART

[0002] In recent years, a technology for presenting an augmented reality (AR) image by overlaying virtual information on a real-world image has been gaining considerable attention. For example, a technology for displaying a virtual object as a piece of virtual information related to a specific target object in the vicinity of the specific target object is known when the user has performed an operation to identify the target object as an object in the real world.

[0003] Patent Reference 1 proposes a device that obtains a reference surface (e.g., palm) as the surface of an object (e.g., hand) existing in real space by analyzing information related to the real space captured by a camera, and changes a virtual object displayed on an image display unit based on the reference surface. Patent Reference 2 describes an information processing device including an image acquisition unit configured to acquire an input image of a real space captured by an imaging device, a user interface unit configured to display the input image on a screen of a display device and recognize a position indicated by a user on the screen, and a computing unit configured tothat it calculates a three-dimensional position of a virtual object in an augmented reality space from the specified position detected by the user interface unit, and an authoring unit configured to link the three-dimensional position of the virtual object calculated by the computing unit with the virtual object in order to store the three-dimensional position in a storage medium. Patent reference 3 describes a method for providing an augmented reality interaction service, wherein the method comprises: generating reference coordinates based on a three-dimensional image containing depth information obtained via a camera; segmenting an area corresponding to a preset object from the 3D image containing the depth information obtained via the camera,based on depth information of the preset object and color space conversion; segmenting a sub-object with a motion component from the preset object in the segmented area and detecting a feature point by modeling the sub-object and a palm area associated with the sub-object based on a preset algorithm; and controlling a 3D object to use an augmented reality service by estimating a posture of the sub-object based on joint information of the preset object provided via a specific user interface. Patent reference 4 describes a portable, head-mounted display system comprising an eye-level display for showing an augmented reality object that is perceptible to a wearer of the head-mounted display system at an apparently real depth and in an apparently real location.and a controller to adjust the seemingly real location of the augmented reality object as a function of the wearer's field of vision. REFERENCE ON THE STATE OF THE TECHNOLOGY PATENT REFERENCE Patent reference 1: JP 2018-84886 A (e.g. sections 0087 to 0102, Fig. 8 to 11) Patent reference 2: US 2012 / 0210255A1 Patent reference 3: US 2017 / 0154471A1 Patent reference 4: US 2016 / 0210784A1 SUMMARY OF THE INVENTIONAL TASK OF THE INVENTION

[0004] However, a problem with the aforementioned conventional device is that there are cases in which the visibility of the virtual object decreases because the shape and inclination of a plane on which the virtual object is arranged change depending on the shape and inclination of the object present in real space.

[0005] An object of the present invention, which has been made to achieve the above-described problem, is to provide an authoring apparatus, an authoring method, and an authoring program that enable an augmented reality image to be displayed so that the visibility of a virtual object does not deteriorate. MEANS TO SOLVED THE TASK

[0006] An authoring device according to one aspect of the present disclosure comprises a user interface unit that receives an operation for determining a target object existing in real space, a target determination unit that, using the user interface unit on a reference plane in relation to the target object, determines a reference point of a target object as the target object, an arrangement position calculation unit that determines a first arrangement plane, arranged in a position that includes the reference point and on which a virtual object can be arranged based on the reference plane and the reference point, and a multi-viewpoint calculation unit that determines one or more second arrangement planes, obtained by rotating the first arrangement plane, on which the virtual object can be arranged.The authoring device outputs information associating the first arrangement level and the virtual object and information associating the second arrangement levels and the virtual object as authoring data.

[0007] An authoring method according to another aspect of the present disclosure includes a step of receiving an operation for specifying a target object existing in real space, a step of specifying a reference point of a determination target as the specified target object on a reference plane in relation to the target object, a step of specifying a first arrangement plane arranged in a position including the reference point and on which a virtual object can be arranged based on the reference plane and the reference point, a step of specifying one or more second arrangement planes obtained by rotating the first arrangement plane on which the virtual object can be arranged, and a step of outputting information associating the first arrangement plane and the virtual object with each other and information associating the second arrangement planes and the virtual object with each other.as authoring data., EFFECT OF THE INVENTION

[0008] According to the invention, it becomes possible to display an augmented reality image in such a way that the visibility of the virtual object is not impaired. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a diagram showing an example of a hardware configuration of an authoring device according to a first embodiment of the present invention. Fig. Figure 2 is a functional block diagram that schematically shows a configuration of the authoring device according to the first embodiment. The Fig. 3A to 3D are diagrams showing data processed by a data acquisition unit of the authoring device according to the first embodiment, and parameters specifying positions and orientations of cameras capturing images of real space. Fig. Figure 4 is a diagram showing an example of target objects existing in real space and object IDs associated with the target objects. Fig. Figure 5 is a diagram showing an example of a planar virtual object. Fig. Figure 6 is a diagram showing an example of a massive virtual object. Fig. 7 is a diagram showing a first determination method for determining a determination target by a user operation of surrounding a region on a target object as the determination target by straight lines. Fig. 8 is a diagram showing a second determination method for determining a determination target by a user operation for determining a point on the target object as the determination target. Fig. 9A is a diagram showing an example of a determination target region and a reference point determined by a user operation, Fig. 9B is a diagram showing an example of the reference point and a reference plane, and Fig. 9C is a diagram showing an example of a horizontal plane. The Fig. 10A, Fig. 10B and Fig. 10C are diagrams that show a process of deriving an arrangement plane from the reference plane and the horizontal plane. The Fig. 11A and Fig. 11B are diagrams showing a first derivation method and a second derivation method for deriving the arrangement plane on which the virtual object is arranged from the reference point and the reference plane. Fig. Figure 12A is a diagram indicating that virtual objects displayed on the arrangement plane can be seen when the destination region is viewed from the front, and Fig. 12B is a diagram indicating that the virtual objects displayed on the arrangement plane cannot be seen when the destination target region is viewed from above. Fig. 13 is a diagram showing an example of displaying the virtual objects using billboard rendering in the state of Fig. 12B shows. Fig. Figure 14 is a diagram showing an arrangement plane derived from a multi-viewpoint calculation unit. Fig. 15 is a diagram showing an arrangement plane derived from the multi-viewpoint calculation unit. Fig. 16 is a diagram showing an arrangement plane derived from the multi-viewpoint calculation unit. Fig. 17 is a flowchart showing the operation of the authoring apparatus according to the first embodiment. Fig. Figure 18 is a diagram showing an example of a hardware configuration of an authoring device according to a second embodiment of the present invention. Fig. Figure 19 is a functional block diagram that schematically shows a configuration of the authoring device according to the second embodiment. Fig. Figure 20 is a flowchart showing an operation of the authoring device according to the second embodiment. MODE FOR CARRYING OUT THE INVENTION

[0009] Authoring devices, authoring methods, and authoring programs according to embodiments of the present invention will be described below with reference to the drawings. The following embodiments are only examples, and various modifications are possible within the scope of the present invention. (1) First embodiment(1-1) Configuration(1-1-1) Hardware configuration

[0010] Fig. 1 is a diagram showing an example of a hardware configuration of an authoring apparatus 1 according to a first embodiment. Fig. 1 does not show a configuration for performing rendering as a process for displaying an AR image based on authoring data including a virtual object. However, the authoring device 1 may include a configuration for acquiring information about real space, such as a camera or a sensor.

[0011] As in Fig. 1, the authoring device 1 includes, for example, a memory 102 as a storage device that stores a program as software, that is, an authoring program according to the first embodiment, and a processor 101 as an arithmetic processing unit that executes the program stored in the memory 102. The processor 101 is an information processing circuit such as a CPU (Central Processing Unit). The memory 102 is a volatile storage device such as a RAM (Random Access Memory). The authoring device 1 is, for example, a computer. The authoring program according to the first embodiment is stored in the memory 102 from a recording medium on which information is stored via a medium information reading device (not shown) or via a communication interface (not shown) connectable to the Internet.

[0012] Furthermore, the authoring device 1 includes an input device 103, e.g., a mouse, a keyboard, or a touch panel, as a user operation unit. The input device 103 is a user operation device that receives user operations. The input device 103 includes a head-mounted display (HMD) that receives input via a gesture operation, a device that receives input via a line-of-sight operation, or the like. The HMD for receiving input via a gesture operation includes a small camera that captures images of a part of the user's body and recognizes a gesture operation, which is the movement of the body, as an input operation for the HMD.

[0013] Furthermore, the authoring device 1 includes a display device 104 that displays images. The display device 104 is a display that presents information to the user when the authoring process is performed. The display device 104 displays an application. The display device 104 can also be a transparent display of the HMD.

[0014] Authoring device 1 can also include a storage device 105 on which various types of information are stored. Storage device 105 is a storage device such as a hard disk drive (HDD) or a solid-state drive (SSD). Programs, data to be used at the time of authoring, data generated by the authoring process, and so on are also stored on storage device 105. Storage device 105 can also be located outside of authoring device 1. For example, storage device 105 can be located in the cloud and connected via a communication interface (not shown).

[0015] The authoring device 1 can be implemented by the processor 101 executing a program stored in the memory 102. It is also possible to implement part of the authoring device 1 by the processor 101 executing a program stored in the memory 102. (1-1-2) Authoring device 1

[0016] Fig. Figure 2 is a functional block diagram that schematically shows a configuration of the authoring device 1 according to the first embodiment. The authoring device 1 is a device capable of performing an authoring procedure according to the first embodiment. The authoring device 1 performs the authoring taking into account the depth of the virtual object.

[0017] The authoring device 1 (1) receives a user operation to determine a target object existing in real space, (2) determines a reference point of a target as the specified target object on a reference plane in relation to the target object (this process is described in the Fig. 9A to 9C shown, which will be explained later), (3) determines a first arrangement plane, arranged in a position that includes the reference point and on which the virtual object can be arranged, based on the reference plane and the reference point (this process is described in the Fig. 10A to 10C shown, which will be explained later), (4) determines one or more second arrangement planes, obtained by rotating the first arrangement plane, on which the virtual object can be arranged (this process is described in the Fig. 14 to 16 shown, which will be explained later), and (5) outputs information relating to the first array level and the virtual object, and information relating to the second array level(s) and the virtual object, as authoring data to memory 105, for example.

[0018] As in Fig. As shown in Figure 2, the authoring device 1 comprises an authoring unit 10, a data acquisition unit 20, and a recognition unit 30. The authoring unit 10 performs the authoring in response to a user operation as an input operation performed by the user. The data acquisition unit 20 captures data to be used at the time the authoring is executed from memory 105 (in Fig. (1 shown). The recognition unit 30 performs processing, for example image recognition, which is required in the authoring process performed by the authoring unit 10. While the memory 105 in the first embodiment in Fig. As shown in Figure 1, the memory 105 can also be, in whole or in part, a storage device outside the authoring device 1. (1-1-3) Data acquisition unit 20

[0019] The Fig. Figures 3A to 3D are diagrams showing data processed by the data acquisition unit 20 of the authoring device 1 according to the first embodiment, and parameters specifying the positions and orientations of cameras capturing images of the real space. The cameras are described in a second embodiment. The data acquisition unit 20 acquires data to be used by the authoring unit 10 at the time of authoring. The data to be used at the time of authoring may include three-dimensional model data representing a three-dimensional model, virtual object data representing the virtual object, and sensor data output by the sensor. This data may also already be stored in memory 105. (Three-dimensional model data)

[0020] The three-dimensional model data is data that represents information about the real space in which the AR image is displayed in three dimensions. The three-dimensional model data can contain the Fig. The data shown in Figures 3A to 3C are included. The three-dimensional model data can be obtained, for example, using SLAM (Simultaneous Localization and Mapping) technology. In SLAM technology, the three-dimensional model data is obtained by capturing images of real space using a camera capable of obtaining a color image (i.e., RGB image) and a depth image (i.e., depth image) of real space (hereinafter also referred to as an "RGBD camera").

[0021] Fig. Figure 3A shows an example of a three-dimensional point set. The three-dimensional point set represents target objects as objects present in real space. Target objects present in real space may include, for example, a floor, a wall, a door, a ceiling, an object placed on the floor, an object suspended from the ceiling, an object attached to a wall, etc.

[0022] Fig. Figure 3B shows an example of a plane obtained in the process of generating the three-dimensional model data. This plane is derived from the data in Fig. 3A shows a three-dimensional point set.

[0023] Fig. Figure 3C shows an example of images obtained by capturing images from multiple viewpoints and at multiple angles. In SLAM technology, the three-dimensional model data is generated by capturing images of the real space from multiple viewpoints and at multiple angles using RGBD cameras or the like. Fig. 3C shown images (ie image data) obtained at the time of image acquisition are combined with the Fig. 3A shown three-dimensional point set, which in Fig. 3B shown plane or both the three-dimensional point set and the plane stored in memory 105.

[0024] The Fig. The information shown in 3D is information that specifies the position and orientation of the camera relative to each image. Assume that k = 1, 2, ..., N (N: positive integer), p kthe position of the k-th camera and r k represents the position of the k-th camera, i.e., an image capture direction of the camera.

[0025] Fig. Figure 4 is a diagram showing an example of target objects that exist in real space and object IDs (identifications) assigned to the target objects. Fig. Section 4 describes "A1", "A2", "A3", and "A4" as examples of object IDs. The three-dimensional model data is used in a procedure for determining a three-dimensional arrangement position of the virtual object, a procedure for deriving the position, the orientation, or both the position and orientation of a target object in an image, etc. The three-dimensional model data is one of several instances of input data for the authoring unit 10.

[0026] The three-dimensional model data may contain different information than that in the Fig. 3A to 3D. The three-dimensional model data may include data of any target object existing in real space. For example, as shown in Fig. 4, the three-dimensional model data may include the object ID assigned to each target object and partial three-dimensional model data of each target object to which an object ID has been assigned.

[0027] In the Fig. 4, the partial three-dimensional model data of each target object can be obtained, for example, using semantic segmentation technology. For example, the partial three-dimensional model data of each target object can be obtained by segmenting data of the Fig. 3A shown three-dimensional point set, data of the Fig. 3B or both of these data sets into regions corresponding to the target objects, respectively. Furthermore, Non-Patent Reference 1 describes a technology for detecting regions of target objects included in point set data based on the point set data.

[0028] Non-patent reference 1: Florian Walch, “Deep Learning for Image-Based Localization”, Department of Computer Science, Technical University of Munich (TUM), October 15, 2016. (Virtual object data)

[0029] Fig. Figure 5 is a diagram showing an example of a two-dimensional virtual object. Fig. Figure 6 is a diagram showing an example of a solid virtual object. Virtual object data is data that stores information representing a virtual object to be displayed as an AR image. The virtual objects discussed here have two types of attributes.

[0030] One in Fig. The virtual object V1 shown in Figure 5 is represented by a plane. The virtual object V1 corresponds to an image, motion video, or the like. A barycenter position of the virtual object V1 is represented as Zv1. The barycenter position Zv1 is stored in the memory or storage 105 in the form of coordinates in a local coordinate system.

[0031] One in Fig. The virtual object V2 shown in Figure 6 is represented by a solid object. The virtual object V2 corresponds to data generated by a three-dimensional modeling tool or similar. The barycenter position of the virtual object V2 is represented as Zv2. The barycenter position Zv2 is stored in memory location 105 as coordinates in a local coordinate system. (Sensor data)

[0032] Sensor data is data intended to support a method for estimating the position and orientation of a camera at the time of image acquisition. For example, sensor data may include tilt data output by a gyroscope that measures the tilt of the camera capturing an image of real space, acceleration data output by an accelerometer that measures the camera's acceleration, and so on. Sensor data is not limited to information associated with the camera; for instance, it may include position data measured by a GPS (Global Positioning System) as a position information measurement system. (1-1-4) Recognition unit 30

[0033] The recognition unit 30 uses the three-dimensional model data acquired by the data acquisition unit 20 to detect a surface or target object located at a specific position in an image. The recognition unit 30 detects the surface or target object at the specified location in the image by transforming two-dimensional positions in the image into three-dimensional positions in real space according to a pinhole camera model and comparing these three-dimensional positions with the three-dimensional model data. The two-dimensional position in the image is represented by pixel coordinates.

[0034] Furthermore, the recognition unit 30 receives an image as input and recognizes the position and orientation of the camera that captured the image based on the received image. A method for estimating the position-orientation pair of a camera that captured an image, based on the image, is known to use a neural network, for example, PoseNet. This method is described, for example, in non-patent reference 2.

[0035] Non-Patent Reference 2: Charles R. Qi and three others: “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation,” Stanford University

[0036] Another method for estimating the position-location pair of a camera that has taken an image is a method using SLAM technology. (1-1-5) Authoring Unit 10

[0037] Authoring Unit 10 performs the authoring of the virtual object using the three-dimensional model data acquired by Data Acquisition Unit 20, the virtual object data, or both of these datasets. Authoring Unit 10 outputs the result of the authoring as the authoring data. Authoring Unit 10 performs the authoring such that the virtual object relating to a user-defined location—that is, a region at the user-defined target—has a position in a depth direction that corresponds to the position of the target area in that depth direction. As described in Fig. 2, the authoring unit 10 includes a user interface unit 11, a destination target setting unit 12, an arrangement position calculation unit 13, and a multiple viewpoint calculation unit 14. (1-1-6) User Interface Unit 11

[0038] The user interface unit 11 provides a user interface for authoring. The user interface unit 11 includes, for example, the input device 103 and the display device 104, which are located in Fig. 1, and so on. The user interface unit 11 may include a GUI (Graphical User Interface) application. Specifically, the user interface unit 11 causes the display device 104 to display an image or three-dimensional data to be used for authoring (e.g., three-dimensional point set data, plane data, etc.) and receives user operations required for authoring from the input device 103. Here, the three-dimensional data may include, for example, three-dimensional point set data, plane data, etc.

[0039] The user input operations performed using the input device 103 are described below. In an "Operation U1," the user specifies an image to be used for authoring. For example, in "Operation U1," the user selects an image from the images stored in the Fig. 3A, Fig. 3B and Fig. The images shown in 3C illustrate this. In "Operation U2," the user defines the target as the reference for the AR image. In "Operation U3," the user performs an operation to arrange the virtual object. In "Operation U4," the user determines the number of layer patterns. The number of layer patterns is the number of layers obtained by calculation using the computational unit 14 for multiple viewpoints, which will be described later.

[0040] In the image determined in “Operation U1”, the user determines the target in “Operation U2”, on the basis of which the target determination unit 12 and the arrangement position calculation unit 13 obtain the three-dimensional position of the target and an arrangement plane as a plane on which the virtual object related to the target is arranged.

[0041] On the obtained plane, the user specifies the arrangement position of the virtual object in "Operation U3," based on which the arrangement position calculation unit 13 calculates the three-dimensional position and posture of the virtual object. Furthermore, the user specifies the number G (G: positive integer) of plane patterns in "Operation U4," based on which the multi-viewpoint calculation unit 14 is able to obtain the arrangement positions of the virtual object in cases where the destination is viewed from G viewpoints (in the line-of-sight directions of G patterns). (1-1-7) Target definition unit 12

[0042] The target determination unit 12 defines a reference point p and a reference plane S pbased on the target specified by the user via user interface unit 11. There are two methods for determining the target: a first and a second method. The methods for deriving the reference point p and the reference plane S are also available. p The target determination unit 12 uses a different procedure for each procedure to determine the target. (First determination procedure)

[0043] In the first determination method, the user, using a graphical user interface (GUI), performs an operation by enclosing a region to be set as the target with lines of a rectangle, polygon, or similar shape. The area enclosed by the lines becomes the region and thus the target. Once the target has been determined using the first method, the reference point p and the reference plane S are defined. p as follows:

[0044] The vertices of an n-sided polygon region, which is defined as the target region, are called H1, ..., H1. n defined. Here, n is an integer greater than or equal to 3. The vertices H1, ..., H n are represented in the GUI image by pixel coordinates (u, v). These coordinates are converted into three-dimensional coordinates a according to the pinhole camera model. i = (x, y, z) transformed. Here, i = 1, 2, ..., n.

[0045] Suppose b1, b2 and b3 represent three points arbitrarily chosen from the three-dimensional coordinates a1, ..., a n If points b1, b2, and b3 are selected, then a plane Sm containing these points is uniquely obtained. Furthermore, among the vertices H1, ..., H n The region of the n-sided polygon is represented by a set C of points that were not chosen as the three points b1, b2 and b3, as follows: C={c1,c2,…,cn−3}

[0046] As a method for selecting three points from the three-dimensional coordinates a1, ..., a n There are J possibilities, which are given by the following expression (1). Here, J is a positive integer. J nC3

[0047] There exist J planes, obtained from any three points selected from the vertices of the n-sided polygon. The J planes are represented as Sm1, ..., Sm J .

[0048] Furthermore, there are sentences C1, ..., C J of points obtained by excluding any three points from the vertices H1, ..., H n the region of the n-sided polygon is obtained, J sentences, as shown below. C1={c1,1,c1,2,…,c1,n−3},⋯,CJ={cJ,1,cJ,2,…,cJ,n−3}

[0049] The element c 1,n-3 For example, represents the (n-3)th element, that is, the (n-3)th point in sentence C1.

[0050] Suppose D(S, X) represents the distance between a plane S and a point X, the reference plane S p obtained by the following expression (2). Among a plurality of planes obtained from three points selected from the vertices of the n-sided polygon, S is the reference plane. p defines a plane whose mean distance to other points is the smallest. The term "other points" refers to points that do not form the plane. Sp=arg minSmi1J∑j=1JD(Smi,Ci,j)

[0051] Here is element C i,j the j-th element in sentence C i .

[0052] Furthermore, A G represent the coordinates of the barycenter of the n-sided polygon, where the reference point p is an intersection point of one of the coordinates A G to the reference plane S obtained by expression (2) p drawn perpendiculars and the reference plane Sp is defined. (Second determination procedure)

[0053] On an image in which a GUI is displayed, the user performs an operation to specify a point to be set as the determination target. If the point has been specified by the user as the determination target region according to the first determination method, the reference point p and the reference plane S are specified in the second determination method. p obtained as follows:

[0054] Assuming that the point in the image where the reference point p was determined is M = (u, v), M can be expressed in three-dimensional coordinates a according to the pinhole camera model. i = (x, y, z). In the second determination method, the three-dimensional coordinates a i directly used as the coordinates of the reference point p.

[0055] The recognition unit 30 determines the reference plane S pby detecting a plane containing the reference point p in the plane data in the three-dimensional model data. If no corresponding plane exists, the detection unit 30 can detect a quasi-plane by using point set data in the vicinity of the reference point p, for example, using RANSAC (RANdom Sample Consensus).

[0056] Fig. 7 is a diagram showing the first determination method for determining the determination target by a user operation of surrounding a region on the target object as the determination target by straight lines. Fig. Fig. 8 is a diagram showing the second determination method for determining the determination target by a user operation of designating a point on the target object as the determination target. Fig. In the second determination method shown in Fig. 8, a plane is detected based on only one point, and thus there are cases where the reference plane S p cannot be adequately detected if the target object is not a plane. Using the Fig. However, in the first determination method shown in Figure 7, the reference plane S p can also be derived if the shape of the target object as the determination target is not flat. (1-1-8) Arrangement position calculation unit 13

[0057] The arrangement position calculation unit 13 executes a first process 13a and a second process 13b, which will be described below.

[0058] In the first process 13a, the arrangement position calculation unit 13 calculates an arrangement plane S q on which the virtual object is arranged. The arrangement position calculation unit 13 derives the arrangement plane S qfrom the reference point p and the reference plane S obtained by the destination target setting unit 12 p as a plane on which the virtual object is arranged. As a method for deriving the arrangement plane S q There is a first derivation method and a second derivation method. (First derivative method)

[0059] In the first derivation method, the arrangement position calculation unit 13 treats the reference plane S p directly as the arrangement level S q . (Second derivative method)

[0060] In the second derivation process, the arrangement position calculation unit 13 first detects a horizontal plane S h based on the three-dimensional model data in real space. The horizontal plane S hIt can also be selected by a user operation performed by the user using user interface unit 11. Furthermore, the horizontal plane S h can also be determined automatically using image recognition and spatial recognition technology. Fig. 9A is a diagram showing an example of the target region and reference point p, which is determined by the user operation. Fig. 9B is a diagram showing an example of the reference point p and the reference plane S p shows. Fig. 9C is a diagram that shows an example of the horizontal plane S h shows.

[0061] The Fig. 10A, Fig. 10B and Fig. 10C are diagrams that illustrate a process of deriving the arrangement plane S q from the reference plane S p and the horizontal plane S hshow. In this case, the arrangement position calculation unit 13 derives the arrangement level S in the second derivation procedure. q after the in the Fig. 10A, Fig. 10B and Fig. 10C shown process.

[0062] First, as in Fig. As shown in 10A, the intersection line of the reference plane S p and the horizontal plane S h defined as L. Then, as in Fig. 10B shows the reference plane S p rotated about the intersection line L as the central axis around a plane S v vertical to the horizontal plane S h to train. Subsequently, as in Fig. 10C shown, which relates to the horizontal plane S h vertical plane S v It is shifted so that it contains the reference point p. Then, the horizontal plane S is... h vertical plane S v , which contains the reference point p, as the arrangement plane S q defined.

[0063] In the first derivation method, the arrangement plane with low visibility can be obtained depending on the inclination of the target area. By obtaining the horizontal plane S h vertical plane S v , which contains the reference point p, as the arrangement plane S q , as in the second derivation method, however, regardless of the inclination of the target area, the position of the virtual object in the depth direction can be aligned with the reference point p as the reference position of the target region in the depth direction.

[0064] The Fig. 11A and Fig. 11B are diagrams showing the first derivative method and the second derivative method for deriving the arrangement level S q , on which the virtual object is arranged, from the reference point p and the reference plane S p show.

[0065] In the second process 13b, the arrangement position calculation unit 13 calculates a three-dimensional arrangement position q of the virtual object. After the arrangement plane S q The arrangement position of the virtual object, on which the virtual object is arranged, was derived by the first process 13a from the arrangement position calculation unit 13. The user determines the arrangement position of the virtual object using a GUI. For example, the user determines the arrangement position of the virtual object by clicking on a location in an image where the user wants to arrange the virtual object using the input device 103, such as a mouse. At this point, it is also possible to support the user's arrangement position determination operation by the arrangement plane S q is projected onto the GUI image.

[0066] Assuming that D = (u, v) represents the coordinates in the image obtained by the user's determination, three-dimensional coordinates E = (x, y, z) can be obtained from the coordinates D according to the pinhole camera model. Assuming F = (x c , y c , z c ) would represent the three-dimensional coordinates of the camera obtained from the three-dimensional model data, then an intersection of the arrangement plane S is q and a vector formed by two points at coordinates E and F is defined as the arrangement position q. Furthermore, it is also possible to arrange a multitude of virtual objects with respect to a target. When t (t: positive integer) virtual objects are arranged, the arrangement positions q1, q2, ..., q1 are defined. t derived using the same procedure.

[0067] Furthermore, after determining the arrangement position, the size of the virtual object can be changed by the user through a user operation, such as a drag-and-drop operation. In this case, it is desirable to display the virtual object obtained as a result of the rendering on the display device 104 at the time of the user operation.

[0068] Furthermore, at this point, the user can change the direction (i.e., position) in which the virtual object is arranged by performing a user operation, such as a drag-and-drop operation. In this case, information about the rotation of the virtual object in storage 105 is also stored as authoring data. By performing the procedure described above, the three-dimensional arrangement position, area, and location of the virtual object are obtained. (1-1-9) Calculation unit 14 for multiple viewpoints

[0069] Through processing by the components for the arrangement position calculation unit 13, the position of the target area in the depth direction and the position of the virtual object in the depth direction were aligned with each other when viewed from a specific direction. Fig. 12A is a diagram indicating that virtual objects No. 1 and No. 2, which are on the arrangement plane S q They will be displayed when the target region is viewed from the front. Fig. 12B is a diagram indicating that virtual objects No. 1 and No. 2, which are on the arrangement level S q They are displayed but not visible when the target region is viewed from above.

[0070] Fig. Figure 13 is a diagram showing an example of displaying the virtual No. 1 and No. 2 objects using billboard rendering. When the rendering is performed using billboard rendering, so that each virtual object is constantly in a position perpendicular to a line-of-sight vector of the camera, the virtual object can be displayed as shown in Figure 13. Fig. Figure 13 shows that positions L1 and L2 of virtual objects No. 1 and No. 2 in the depth direction are, however, defined by a position L. p the target region is derived in the depth direction.

[0071] To ensure that the position of each virtual object in the depth direction corresponds to the position of the target area in the depth direction, even with a significant change in viewpoint as described above, the multi-viewpoint computation unit 14 prepares a multitude of array levels for a target and calculates the array position of the virtual object with respect to each array level. The multi-viewpoint computation unit 14 repeats a first viewpoint calculation process 14a and a second viewpoint calculation process 14b, which are described below, a number of times equal to the number of array levels added.

[0072] In the first viewpoint calculation process 14a, the calculation unit 14 determines a plane S for several viewpoints. r , which are obtained by rotating the arrangement plane S obtained by the arrangement position calculation unit 13 qwas obtained by means of an axis passing through the reference point p.

[0073] In the second viewpoint calculation process 14b, the calculation unit 14 receives arrangement positions q for several viewpoints. r1 , q r2 , ..., q rt of arranged virtual objects v1, v2, ..., v t , which are calculated by the arrangement position calculation unit 13 on level S r be obtained.

[0074] Regarding the first viewpoint calculation process 14a, it is also possible to allow the user to select level S. r to be set by a user operation, such as drag and drop. Furthermore, the calculation unit 14 can provide a function for automatically maintaining the level S for multiple viewpoints. r exhibit. The following are examples of the procedure for automatically preserving level S. r described.

[0075] Regarding the second viewpoint calculation process 14b, the multi-viewpoint calculation unit 14 is capable of calculating the arrangement positions q r1 , q r2 , ..., q rt on level S r using a relative positional relationship at the arrangement level S q between the reference point p and the arrangement positions q1, q2, ..., qt of the virtual objects v1, v2, ..., v obtained by the arrangement position calculation unit 13 t to obtain.

[0076] Furthermore, after obtaining temporary arrangement positions using the method described above, it is also possible to provide the user with a user interface to allow the user to adjust the arrangement positions. Additionally, for multiple viewpoints, the computation unit 14 can adjust the arrangement positions of the virtual objects after obtaining the temporary arrangement positions by assessing the collision of the virtual objects and the target objects in real space using the point set data in the three-dimensional submodel data, the plane data in the three-dimensional submodel data, or both of these datasets.

[0077] Below are examples of the procedure for automatically obtaining the level S r described in the first viewpoint calculation process 14a. An example in which the number of levels S rThe number of levels is described here. If the number of levels is three, the calculation unit 14 derives arrangement levels S for multiple viewpoints. r1 , S r2 and S r3 than the levels S r away. Fig. 14 is a diagram showing an arrangement plane S r1 shows, which is derived from the calculation unit 14 for multiple viewpoints. Fig. Figure 15 is a diagram that provides an example of the arrangement level S r2 shows, which is derived from the calculation unit 14 for multiple viewpoints. Fig. Figure 16 is a diagram that provides an example of the arrangement level S r3 shows, which is derived from the calculation unit 14 for multiple viewpoints. The examples from the Fig. 14 to Fig. 16 show the arrangement levels S r1 , S r2 and S r3, taking into account that the designation target is viewed from the front and back, top and bottom, and left and right. In the examples, the arrangement levels S r1 , S r2 and S r3 as follows, without user operation.

[0078] The Fig. Example 14 shown is an example where the arrangement level S derived from the arrangement position calculation unit 13 q directly as arrangement level S r1 is being treated.

[0079] The Fig. 15 shown arrangement level S r2 is a plane that is created by rotating the arrangement plane S q is obtained around a horizontal axis that passes through the reference point p, in order to be parallel to the horizontal plane S h to be the one recognized by the arrangement position calculation unit 13.

[0080] The Fig. 16 shown arrangement level S r3is a level that can be created by changing the arrangement level S q in a direction perpendicular to both the arrangement plane S r1 as well as the arrangement level S r2 is obtained and includes the reference point p.

[0081] As above, the arrangement position calculation unit 13 calculates a plurality of arrangement planes and arrangement positions and outputs the calculation result as the authoring data. At the time of rendering execution, by switching the plane as the rendering target depending on the camera angle, the depth direction positions of a plurality of virtual objects related to the designation target can be consistent with the depth direction position of the designation target, even when the virtual objects are viewed from a plurality of viewpoints. (1-1-10) Authoring data

[0082] The authoring data is stored in memory 105 and indicates the result of the authoring performed by authoring unit 10. The authoring data includes, for example, the following first through sixth pieces of information, I1 to I6:

[0083] The first information I1 is information about the target and contains information about the reference point p and the reference plane S. p The second set of information, I2, is information about the arrangement level and contains information about the arrangement level S. q and the level S rThe third piece of information I3 is information about the virtual objects and contains information about the virtual objects v1, v2, etc. The fourth piece of information I4 is information indicating the arrangement positions of the virtual objects. The fifth piece of information I5 is information indicating the arrangement ranges of the virtual objects. The sixth piece of information I6 is information indicating the locations of the virtual objects. The information indicating the locations is also called information indicating the directions of the virtual objects.

[0084] The three-dimensional arrangement positions of the virtual objects received by the authoring unit 10 are managed while they are assigned to the arrangement plane, the labeling target, or both of these pieces of information. (1-2) Functionality

[0085] Fig. Figure 17 is a flowchart illustrating the operation of the authoring device 1 according to the first embodiment. Initially, in step S11, the authoring device 1 starts an authoring application using the functions of the authoring unit 10, as instructed by the user.

[0086] In step S12, the authoring device 1 captures an image, a three-dimensional point configured as three-dimensional data, or a plane to be used for authoring, which is determined by the user via the user interface unit 11 of the authoring unit 10, and displays the captured image or three-dimensional data on the display device 104. The user determines the object using a mouse, touchpad, or the like as the user interface unit 11.

[0087] In step S13, the authoring device 1 defines the destination for the user-specified image or three-dimensional data via the user interface unit 11. The authoring device 1 receives the reference point p and the reference plane S. p based on the user-defined destination.

[0088] In step S14, the authoring device 1 sets the arrangement level S q fixed, on which the virtual object is positioned.

[0089] In step S15, authoring device 1 receives information about the arrangement position, size, rotation, etc., of the virtual object entered through user operations. Based on the received information, authoring device 1 calculates information such as the three-dimensional arrangement position and the location of the virtual object.

[0090] In step S16, the authoring device 1 for handling rendering from a plurality of viewpoints acquires the layout plane, the layout position of the virtual object placed on the layout plane, etc., for the number of times corresponding to the number of added planes. At this time, there are a case where the added layout plane is determined on a GUI by a user operation, and a case where the added layout plane is determined automatically without user operation.

[0091] In step S17, after obtaining the authoring information regarding the virtual object at a plurality of levels, the authoring device 1 outputs the authoring information obtained through the previous processing as authoring data and stores the authoring data in the memory 105. (1-3) Effect

[0092] As described above, in the first embodiment, when authoring is performed based on the target object as the destination in real space and a virtual object that refers to the target object as the destination, the reference point p and the reference plane S are p The target is obtained from the user's target by the target definition unit 12. Accordingly, the position of the virtual object in the depth direction can correspond to the position of the target in the depth direction, regardless of the shape and inclination of the target.

[0093] Furthermore, the calculation unit 14 provides a multitude of virtual object arrangement planes for multiple viewpoints. Accordingly, even if the camera's direction or position changes, the position of the virtual object in the depth direction can coincide with the position of the target in the depth direction.

[0094] Furthermore, even in cases where a large number of contents have been registered for a target, the positions of the virtual objects in the depth direction can be adjusted to match the position of the target in the depth direction, even if the direction or position of the camera is changed. (2) Second embodiment(2-1) Configuration(2-1-1) Hardware configuration

[0095] While the authoring device 1 according to the first embodiment is a device that generates and outputs authoring data, the authoring device may also include a configuration for performing rendering.

[0096] Fig. Figure 18 is a diagram showing an example of a hardware configuration of an authoring device 2 according to a second embodiment of the present invention. Fig. 18 was assigned to each component that was identical to one in Fig. The component shown in point 1, or which corresponds to it, has the same reference symbol as in point 1. Fig. 1 assigned. The authoring device 2 according to the second embodiment differs from the authoring device 1 according to the first embodiment in that it comprises a sensor 106 and a camera 107.

[0097] Sensor 106 is an IMU (Inertial Measurement Unit), an infrared sensor, a LiDAR (Light Detection and Ranging), or the like. The IMU is a detection device that integrates various types of sensors, such as an acceleration sensor, a geomagnetic sensor, and a gyro sensor. Camera 107 is an image capture device, such as a monocular camera, a stereo camera, or an RGBD camera.

[0098] The authoring device 2 estimates the position and attitude of the camera 107 from image data output from the camera 107 capturing an image of the real space, selects a display plane on which the virtual object is arranged from a first arrangement plane and one or more second arrangement planes based on the estimated position and attitude of the camera 107 and the authoring data, and outputs display image data based on the image data and the virtual object arranged on the display plane.

[0099] From the first arrangement plane and one or more second arrangement planes, the authoring device 2 selects an arrangement plane with which an angle formed by the first arrangement plane and a vector determined by the position of the camera 107 and the reference point p or an angle formed by the second arrangement plane and the vector is closest to 90 degrees, as the display plane on which the virtual object is displayed. (2-1-2) Authoring Device 2

[0100] Fig. 19 is a functional block diagram schematically showing a configuration of the authoring apparatus 2 according to the second embodiment. In Fig. 19, each component that is identical to one in Fig. 2 shown component or corresponds to it, the same reference numeral as in Fig. 2 assigned. The authoring device 2 according to the second embodiment differs from the authoring device 1 according to the first embodiment in that it comprises an image acquisition unit 40 and an AR display unit 50, which outputs image data to the display device 104.

[0101] The image acquisition unit 40 captures image data output by the camera 107. The image data captured by the image acquisition unit 40 is input to the authoring unit 10, the recognition unit 30, and the AR display unit 50. In one case, when authoring is performed using the image data output by the camera 107, the image data output by the camera 107 is input to the authoring unit 10. In other cases, the image data output by the camera 107 is input to the AR display unit 50. (2-1-3) AR display unit 50

[0102] The AR display unit 50 performs the rendering to generate image data for displaying the virtual object on the display device 104 using the authoring data output by the authoring unit 10 or stored in memory 105. As in Fig. As shown in Figure 19, the AR display unit 50 comprises a position-location estimation unit 51, a display level setting unit 52 and a rendering unit 53. (Position-Location Estimation Unit 51)

[0103] The position-orientation estimation unit 51 estimates the position and orientation of camera 107 connected to authoring device 2. Image data from an image captured by camera 107 via image acquisition unit 40 is provided to the recognition unit 30. Recognition unit 30 receives the image data as input and, based on the received image data, determines the position and orientation of the camera that captured the image. Based on the result of the recognition by recognition unit 30, position-orientation estimation unit 51 estimates the position and orientation of camera 107 connected to authoring device 2. (Display level setting unit 52)

[0104] In the authoring data of the second embodiment, there is a case where, due to the computational unit 14, a multitude of arrangement levels exist for multiple viewpoints for a user-defined target. These multiple arrangement levels are, for example, those in Fig. 14 to Fig. 16 shown arrangement levels S r1 , S r2 and S r3 The display plane setting unit 52 sets a plane as the rendering target from the plurality of arrangement planes using the current position and attitude information of the camera 107. A reference point corresponding to a specific determination target is represented as p, and t (t: positive integer) display planes are represented as S1, S2, ..., St. Further, assume that θ1, θ2, ..., θ teach represents an angle [°] formed by a vector defined by the three-dimensional position of camera 107 and the reference point p and the display planes S1, S2, ..., S t to be determined, and assuming that i represents an integer greater than or equal to 1 and less than or equal to t, the plane S R as the rendering target, as specified by the following expression (3), for example, when 0° < θ i ≤ 90°. The vector determined by the three-dimensional position of the camera 107 and the reference point p is, for example, a vector in a direction connecting the position of the optical axis of the camera 107 and the reference point p. SR=arg minSi(90−θi)

[0105] However, if 90° < θ i If the angle is ≤ 180°, then the plane S is obtained. R as the goal of rendering, as indicated, for example, by the following expression (4). SR=arg minSi(θi−90)

[0106] After obtaining level S R Data such as the arrangement position of the in the plane S R contained virtual object, captured from the authoring data and output to the rendering unit 53. As level S R a display plane is selected with which the angle formed by the display plane and the vector determined by the three-dimensional position of the camera 107 and the reference point p is closest to 90 degrees. (Rendering Unit 53)

[0107] Based on information about the position and attitude of the camera 107 obtained by the position-attitude estimation unit 51 and the arrangement plane and arrangement position of the virtual object obtained by the display plane setting unit 52, the rendering unit 53 transforms the three-dimensional coordinates of the virtual object into two-dimensional coordinates on the display of the display device 104 and displays the virtual object on the display of the display device 104 in superposition on the two-dimensional coordinates obtained by the transformation. (2-1-4) Display device 104

[0108] The display device 104 is a device for rendering the AR image. The display device 104 is, for example, a display of a PC (personal computer), a display of a smartphone, a display of a tablet terminal, a head-mounted display, or the like. (2-2) Functionality

[0109] Fig. Figure 20 is a flowchart illustrating the operation of the authoring device 2 according to the second embodiment. The authoring performed by the authoring device 2 according to the second embodiment is the same as that in the first embodiment.

[0110] In step S21, the authoring device 2 starts an AR application.

[0111] After the authoring application has been started in step S22, the authoring device 2 captures the authoring data as display data in step S23.

[0112] In step S24, the authoring device 2 captures the image data of the captured image output by camera 107, which is connected to the authoring device 2.

[0113] In step S25, the authoring device 2 estimates the position and orientation of camera 107.

[0114] In step S26, the authoring device 2 extracts information about the identified target(s) from the authoring data and performs the processing of step S27 for one target or each of a multitude of targets.

[0115] In step S27, the authoring device 2 determines, from a multitude of arrangement levels corresponding to the target, an arrangement level on which the virtual object is displayed. The authoring device 2 then extracts information from the authoring data about the arrangement position, size, position, orientation, etc., of the virtual object arranged on the specified arrangement level. Finally, the authoring device 2 performs the rendering of the virtual object.

[0116] In step S28, the authoring device 2 assesses whether the AR display process should continue or whether the process is complete for all registered destinations. If the AR display process should continue, the processing from step S24 to step S27 is repeated. (2-3) Effect

[0117] As described above, in the second embodiment, when the target is defined as the target of the virtual object and the virtual object is rendered relative to the target, the rendering is performed based on the authoring data output by the authoring unit 10. Accordingly, it becomes possible to perform a rendering in which the position of the virtual object in the depth direction corresponds to the position of the target in the depth direction, regardless of the shape or inclination of the target.

[0118] Furthermore, from a multitude of content arrangement layers obtained by the calculation unit 14 for multiple viewpoints, the layer as the rendering target is determined by the display layer definition unit 52 based on the position of camera 107, the orientation of camera 107, or both of these pieces of information. Accordingly, the position of the virtual object in the depth direction can be designed so that it remains consistent with the position of the target in the depth direction even if the position of camera 107, its orientation, or both of these pieces of information change. DESCRIPTION OF REFERENCE SYMBOLS

[0119] 1, 2: Authoring device, 10: Authoring unit, 11: User interface unit, 12: Destination target setting unit, 13: Arrangement position calculation unit, 14: Multi-viewpoint calculation unit, 20: Data acquisition unit, 30: Recognition unit, 40: Image acquisition unit, 50: AR display unit, 51: Position-attitude estimation unit, 52: Display plane setting unit, 53: Rendering unit, 101: Processor, 102: Memory, 103: Input device, 104: Display device, 105: Tray, 106: Sensor, 107: Camera, p: Reference point, S p : Reference plane, S h : horizontal plane, S q : Arrangement level, S r1 , S r2 , S r3 : Arrangement level.

Claims

[1] Authoring device (1, 2) comprising: a user interface unit (11) receiving an operation for determining a target object existing in real space; a destination setting unit (12) which, using the user interface unit on a reference plane (S p ) in relation to the target object, sets a reference point (p) of a destination target as the target object; an arrangement position calculation unit (13) which calculates a first arrangement plane (S q ), arranged in a position that includes the reference point (p) and on which a virtual object can be placed, based on the reference plane (S p ) and the reference point (p); and a calculation unit (14) for multiple viewpoints, which comprises one or more second arrangement planes (S r1 , S r2 , S r3 ), obtained by rotating the first arrangement plane (S q), on which the virtual object can be placed, wherein the authoring device contains information that defines the first arrangement level (S q ) and the virtual object to each other, and information that the second arrangement levels (S r1 , S r2 , S r3 ) and the virtual object to each other, outputs as authoring data. [2] The authoring device (1, 2) according to claim 1, wherein the operation using the user interface unit (11) is an operation of surrounding a region representing the target object as the determination target by an n-sided polygon, where n is an integer greater than or equal to three. [3] Authoring device (1, 2) according to claim 2, wherein the destination setting unit (12) one of the planes that includes three vertices among vertices of the n-sided polygon as the reference plane (S p ) and the reference point (p) based on a barycenter position of the n-sided polygon and the reference plane (S p ) certainly. [4] Authoring device (1, 2) according to one of claims 1 to 3, wherein the calculation unit (14) for multiple viewpoints comprises the one or more arrangement planes (S r1 , S r2 , S r3 ) by rotating the first arrangement level (S q ) around an axis line encompassing the reference point (p). [5] Authoring device (2) according to one of claims 1 to 4, further comprising: a position-attitude estimation unit (51) that estimates a position and an attitude of a camera (107) that captures an image of a real space based on image data output from the camera (107); a display plane setting unit (52) which sets a display plane on which the virtual object is arranged from the first arrangement plane (S q) and the one or more second arrangement levels (S r1 , S r2 , S r3 ) based on the estimated position and attitude of the camera (107) and the authoring data; and a rendering unit (53) that outputs display image data based on the image data and the virtual object arranged on the display plane. [6] Authoring device (2) according to claim 5, wherein the display level setting unit (52) is selected from the first arrangement level (S q ) and the one or more second arrangement levels (S r1 , S r2 , S r3 ) an arrangement level with which a pattern from the first arrangement level (S q ) and an angle formed by the position of the camera (107) and the reference point (p) or an angle formed by the second arrangement plane and the vector is closest to 90 degrees as the display plane on which the virtual object is displayed. [7] Authoring process, including: a step of receiving an operation for determining a target object existing in real space; a step of setting a reference point (p) of a determination target as the determined target object on a reference plane (S p ) in relation to the target object; a step of defining a first arrangement level (S q ), arranged in a position which includes the reference point (p) and on which a virtual object can be arranged, based on the reference plane (S p ) and the reference point (p); a step of defining one or more second arrangement levels (S r1 , S r2 , S r3 ), obtained by rotating the first arrangement plane (S q ) on which the virtual object can be placed; and a step of outputting information that represents the first arrangement level (Sq ) and the virtual object to each other, and from information that the second arrangement levels (S r1 , S r2 , S r3 ) and the virtual object to each other as authoring data. [8] Authoring program that causes a computer to execute: a process of receiving an operation for determining a target object existing in real space; a process of setting a reference point (p) of a determination target as the determined target object on a reference plane (S p ) in relation to the target object; a process of defining a first arrangement level (S q ), arranged in a position which includes the reference point (p) and on which a virtual object can be arranged, based on the reference plane (S p ) and the reference point (p); a process of defining one or more second arrangement levels (Sr1 , S r2 , S r3 ), obtained by rotating the first arrangement plane (S q ) on which the virtual object can be placed; and a process of outputting information that the first arrangement level (S q ) and the virtual object to each other, and from information that the second arrangement levels (S r1 , S r2 , S r3 ) and the virtual object to each other as authoring data.

Images