Display control device, display control method, and display control program

Abstract

A display control device (100) comprises: a sensor (106) that outputs sensor data that is data for detecting the direction of gravity; a construction unit (130) that constructs a world coordinate system using the sensor data; an acquisition unit (120) that acquires an image including a marker in which the direction of one side of the marker is the direction of gravity, and AR information; an image processing unit (160) that detects the marker on the basis of the image, acquires a conversion matrix for the world coordinate system when the marker is detected, extracts a parallel shift component and a rotational component from the conversion matrix, calculates the Euler angle of an axis that corresponds to the direction of gravity by using the rotational component, and calculates a first correction conversion matrix which is a matrix by using the Euler angle and the parallel shift component; a correction processing unit (170) that corrects the coordinates of the AR information by using the first correction conversion matrix; and a display control unit (180) that displays the AR information.

Description

Display control device, display control method, and display control program

[0001] The present disclosure relates to a display control device, a display control method, and a display control program.

[0002] Augmented reality (AR) is known, and a technology relating to AR has been proposed (see Patent Document 1). Patent Document 1 describes a method for aligning a virtual space with a real space.

[0003] JP 2023-77070 A

[0004] <URL: https: / / learn. microsoft. com / en-us / uwp / api / windows. perception. spatial. spatial coordinate system? view=winrt-22621><URL: https: / / learn. microsoft. com / ja-jp / dotnet / api / system. numerics. matrix4x4. Decompose? view=net-8.0>

[0005] However, if a device such as an HMD (Head Mounted Display) cannot accurately detect a marker, the AR information will be displayed at a position different from the expected position.

[0006] An object of the present disclosure is to display AR information at an appropriate position.

[0007] According to one aspect of the present disclosure, there is provided a display control device including: a sensor that outputs sensor data for detecting a direction of gravity; a constructing unit that constructs a world coordinate system using the sensor data; an acquiring unit that acquires an image including a marker whose one side is oriented in the direction of gravity and AR information; an image processing unit that detects the marker based on the image, acquires a transformation matrix for the world coordinate system upon detecting the marker, extracts a translation component and a rotation component from the transformation matrix, calculates an Euler angle of an axis corresponding to the direction of gravity using the rotation component, and calculates a first correction transformation matrix using the Euler angle and the translation component; a correction processing unit that corrects coordinates of the AR information using the first correction transformation matrix; and a display control unit that displays the AR information.

[0008] According to the present disclosure, AR information can be displayed at an appropriate position.

[0009] 1 is a diagram illustrating a display control device according to a first embodiment; FIG. 2 is a diagram illustrating hardware included in the display control device according to the first embodiment; FIG. 3 is a block diagram illustrating functions of the display control device according to the first embodiment; FIG. 4 is a flowchart illustrating an example (part 1) of processing executed by the display control device according to the first embodiment; FIG. 5 is a flowchart illustrating an example (part 2) of processing executed by the display control device according to the first embodiment; FIG. 6 is a diagram illustrating a specific example (part 1) of processing executed by the display control device according to the first embodiment; FIG. 7 is a block diagram illustrating functions of a display control device according to a second embodiment; FIG. 8 is a flowchart illustrating an example (part 1) of processing executed by the display control device according to the second embodiment; FIG. 9 is a flowchart illustrating an example (part 2) of processing executed by the display control device according to the second embodiment; FIG. 10 is a diagram illustrating a line detection process according to the second embodiment; FIG. 11 is a diagram for explaining an overview of a third embodiment; FIG. 12 is a flowchart illustrating an example (part 1) of processing executed by the display control device according to the third embodiment; FIG. 13 is a flowchart illustrating an example (part 2) of processing executed by the display control device according to the third embodiment;

[0010] Hereinafter, an embodiment will be described with reference to the drawings.

[0011] Embodiment 1. FIG. 1 is a diagram illustrating a display control device according to embodiment 1. The display control device 100 is a device that executes a display control method. For example, the display control device 100 is an HMD. The display control device 100 may be a smartphone or a tablet terminal. FIG. 1 illustrates a marker 200. The marker 200 is also referred to as an AR marker. The display control device 100 can display AR information by detecting the marker 200. FIG. 1 illustrates a floor 300 and an object 301. For example, the marker 200 is attached to the object 301. Here, the direction of one side of the marker 200 is the direction of gravity. In other words, the direction of gravity of the marker 200 is the same as the Y-axis direction of the marker coordinate system.

[0012] Next, a description will be given of the hardware included in the display control device 100. Fig. 2 is a diagram showing the hardware included in the display control device according to embodiment 1. The display control device 100 includes a processor 101, a volatile storage device 102, a non-volatile storage device 103, an imaging device 104, a display device 105, and a sensor 106.

[0013] The processor 101 controls the entire display control device 100. For example, the processor 101 is a central processing unit (CPU), a field programmable gate array (FPGA), a graphics processing unit (GPU), or the like. The processor 101 may be a multiprocessor. The display control device 100 may also include a processing circuit.

[0014] The volatile storage device 102 is a main storage device of the display control device 100. For example, the volatile storage device 102 is a random access memory (RAM). The nonvolatile storage device 103 is an auxiliary storage device of the display control device 100. For example, the nonvolatile storage device 103 is a hard disk drive (HDD) or a solid state drive (SSD).

[0015] The imaging device 104 captures an image. For example, the imaging device 104 captures an image of the marker 200. The display device 105 displays AR information. For example, the display device 105 is a display. The sensor 106 is a gyro sensor, an acceleration sensor, a magnetic sensor, a depth sensor, or the like.

[0016] Next, a description will be given of the functions of the display control device 100. Fig. 3 is a block diagram showing the functions of the display control device of embodiment 1. The display control device 100 has a storage unit 110, an acquisition unit 120, a construction unit 130, a preprocessing unit 140, a position and orientation estimation unit 150, an image processing unit 160, a correction processing unit 170, and a display control unit 180.

[0017] The storage unit 110 may be realized as a storage area secured in the volatile storage device 102 or the non-volatile storage device 103. The storage unit 110 may also be called a memory. Some or all of the acquisition unit 120, construction unit 130, pre-processing unit 140, position and orientation estimation unit 150, image processing unit 160, correction processing unit 170, and display control unit 180 may be realized by processing circuits. Furthermore, some or all of the acquisition unit 120, construction unit 130, pre-processing unit 140, position and orientation estimation unit 150, image processing unit 160, correction processing unit 170, and display control unit 180 may be realized as program modules executed by the processor 101. For example, the program executed by the processor 101 is also called a display control program or a display control program product. For example, the display control program is recorded on a recording medium.

[0018] The storage unit 110 stores various information. The functions of the acquisition unit 120, construction unit 130, preprocessing unit 140, position and orientation estimation unit 150, image processing unit 160, correction processing unit 170, and display control unit 180 will be described later.

[0019] Next, the processing executed by the display control device 100 will be described using a flowchart. FIG. 4 is a flowchart showing an example (part 1) of the processing executed by the display control device of embodiment 1. (Step S11) The acquisition unit 120 acquires sensor data. The sensor data is data for detecting the direction of gravity. The acquisition unit 120 acquires sensor data from a sensor 106 (e.g., a gyro sensor, an acceleration sensor) that outputs sensor data. (Step S12) The construction unit 130 constructs a world coordinate system using the sensor data. In detail, the construction unit 130 constructs a world coordinate system using the sensor data such that the positive direction of the axis corresponding to the direction of gravity in the world coordinate system (i.e., the Y-axis) is opposite to the direction of gravity. Note that the world coordinate system may also be called a real space coordinate system.

[0020] (Step S13) The acquisition unit 120 acquires an image from the imaging device 104. (Step S14) The preprocessing unit 140 performs preprocessing on the image. For example, the preprocessing unit 140 performs resizing, grayscale conversion, noise removal, etc. (Step S15) The acquisition unit 120 acquires sensor data from the sensor 106. (Step S16) The position and orientation estimation unit 150 estimates the position and orientation of the display control device 100 based on the preprocessed image and the sensor data acquired in step S15.

[0021] (Step S17) The image processing unit 160 executes a detection process for the marker 200 based on the preprocessed image and the sensor data acquired in step S15. The image processing unit 160 may execute the detection process for the marker 200 based on the preprocessed image. If the marker 200 is detected by the detection process, the process proceeds to step S21. If the marker 200 is not detected, the process proceeds to step S13.

[0022] 5 is a flowchart showing an example (part 2) of processing executed by the display control device according to the first embodiment. (Step S21) When the image processing unit 160 detects the marker 200, it acquires a transformation matrix (hereinafter, transformation matrix A) for the world coordinate system. In other words, when the image processing unit 160 detects the marker 200, it acquires the position and orientation of the marker 200 as a transformation matrix A for the world coordinate system. It may also be expressed that the image processing unit 160 acquires the transformation matrix A for the world coordinate system based on an image. For example, when acquiring the transformation matrix, the image processing unit 160 acquires the transformation matrix using an API (Application Programming Interface) (i.e., TryGetTransformTo( )) described in Non-Patent Document 1.

[0023] (Step S22) The image processing unit 160 extracts translation components and rotation components from the transformation matrix A. For example, when extracting translation components and rotation components, the image processing unit 160 extracts the translation components and rotation components using the API (i.e., Decompose()) described in Non-Patent Document 2.

[0024] (Step S23) The image processing unit 160 uses the rotational components to calculate the Euler angles of the axis corresponding to the direction of gravity (i.e., the Y-axis). The calculation method is described below. When the API described in Non-Patent Document 2 is used, the rotational components are extracted using quaternions. Therefore, in the following description, it is assumed that the rotational components are expressed using quaternions. Furthermore, it is assumed that the order in which the Euler angles are applied is ZXY. The image processing unit 160 calculates the Euler angles of the Y-axis using equation (1).

[0025]

[0026] (Step S24) The image processing unit 160 calculates the correction transformation matrix B using the calculated Euler angles and the extracted translation components. Specifically, the image processing unit 160 calculates the correction transformation matrix B using equation (2). The correction transformation matrix B is also called a first correction transformation matrix. T is a translation matrix indicating the translation components. T ｘ , Tｙ , T ｚ indicates the displacement from the origin of the world coordinate system to the origin of the marker coordinate system. R indicates a rotation matrix. S indicates a scaling matrix.

[0027]

[0028] (Step S25) The acquisition unit 120 acquires AR information. For example, the acquisition unit 120 acquires the AR information from the storage unit 110. Also, for example, the acquisition unit 120 acquires the AR information from an external device. Note that the external device is a device that exists outside the display control device 100. For example, the external device is a cloud server. The external device is not shown in the drawing.

[0029] (Step S26) The correction processing unit 170 corrects the coordinates of the AR information using the correction transformation matrix B. Specifically, the correction processing unit 170 corrects the coordinates of the AR information using equation (3). (x, y, z, 1) indicate the coordinates of the AR information before the change. (x', y', z', 1) indicate the coordinates of the AR information after the change. Note that the coordinates of the AR information are coordinates indicating the position where the AR information is displayed.

[0030]

[0031] (Step S27) The display control unit 180 displays the AR information on the display of the display control device 100.

[0032] In step S14, a case where preprocessing is performed has been described. The display control device 100 does not need to perform preprocessing. If preprocessing is not performed, the position and orientation estimation unit 150 estimates the position and orientation of the display control device 100 based on the image acquired in step S13 and the sensor data acquired in step S15. Furthermore, the image processing unit 160 performs detection processing of the marker 200 based on the image acquired in step S13 and the sensor data acquired in step S15. The image processing unit 160 may perform detection processing of the marker 200 based on the image acquired in step S13.

[0033] Next, a specific example of processing executed by the display control device 100 will be shown. FIG. 6 is a diagram showing a specific example (part 1) of processing executed by the display control device of embodiment 1. The acquisition unit 120 acquires sensor data from the sensor 106. The construction unit 130 uses the sensor data to construct a world coordinate system such that the positive direction of the Y axis of the world coordinate system is opposite to the direction of gravity. Note that FIG. 6 shows an origin 10 of the world coordinate system. FIG. 6 also shows a marker 200. FIG. 6 also shows a marker coordinate system. Furthermore, FIG. 6 also shows an origin 11 of the marker coordinate system.

[0034] The acquisition unit 120 acquires an image including the marker 200 from the imaging device 104. The preprocessing unit 140 performs preprocessing on the image. The acquisition unit 120 acquires sensor data from the sensor 106. The image processing unit 160 detects the marker 200 based on the preprocessed image and the sensor data. The image processing unit 160 acquires a transformation matrix A. The image processing unit 160 extracts a translation component and a rotation component from the transformation matrix A. The image processing unit 160 calculates Euler angles for the Y axis using the rotation component. The image processing unit 160 calculates a correction transformation matrix B using the Euler angles and the translation component.

[0035] 7 is a diagram showing a specific example (part 2) of the processing executed by the display control device according to the first embodiment. The acquisition unit 120 acquires AR information 400. The correction processing unit 170 corrects the coordinates of the AR information 400 using a correction transformation matrix B. The display control unit 180 displays the AR information 400 on the display of the display control device 100.

[0036] According to the first embodiment, even when the display control device 100 cannot accurately detect the marker 200, the display control device 100 can display the AR information at an accurate position by correcting the coordinates of the AR information using the correction transformation matrix B. That is, the display control device 100 can display the AR information at an accurate position by correcting a deviation due to detection of the marker 200 (in other words, inaccurate detection of the marker 200) using the correction transformation matrix B. In this way, the display control device 100 can display the AR information at an accurate position.

[0037] Second Embodiment Next, a second embodiment will be described. In the second embodiment, differences from the first embodiment will be mainly described. In the second embodiment, descriptions of the commonalities between the first embodiment and the second embodiment will be omitted.

[0038] 8 is a block diagram showing the functions of the display control device of embodiment 2. The display control device 100 further includes a detection unit 190 and a calculation unit 191. Part or all of the detection unit 190 and the calculation unit 191 may be realized by a processing circuit. Also, part or all of the detection unit 190 and the calculation unit 191 may be realized as a program module executed by the processor 101. The functions of the detection unit 190 and the calculation unit 191 will be described later.

[0039] Next, the processing executed by the display control device 100 will be described using a flowchart. FIG. 9 is a flowchart showing an example (part 1) of the processing executed by the display control device of embodiment 2. FIG. 10 is a flowchart showing an example (part 2) of the processing executed by the display control device of embodiment 2. The processing in FIGS. 9 and 10 differs from the processing in FIG. 5 in that steps S24a to 24e and 26a are executed. Therefore, steps S24a to 24e and 26a will be described in FIGS. 9 and 10. Further, a description of the processing other than steps S24a to 24e and 26a will be omitted.

[0040] (Step S24a) The detection unit 190 detects information on a first direction (i.e., vector information), which is a direction other than the direction of gravity, from the correction transformation matrix B. Specifically, the first direction is the X-axis direction or the Z-axis direction. In the following description, the first direction is the X-axis direction. (Step S24b) The detection unit 190 detects multiple straight lines from the image acquired in step S13. The detection unit 190 can detect multiple straight lines by performing image analysis processing. Note that the straight lines are preferably long. The detection unit 190 may detect multiple straight lines from a preprocessed image.

[0041] (Step S24c) The detection unit 190 detects, from among the multiple straight lines, the straight line that forms the smallest angle with the X-axis direction. In other words, the detection unit 190 detects, from among the multiple straight lines, the straight line that forms the smallest angle with the X-axis direction information (i.e., vector information). Note that the detected straight line is preferably a straight line near the marker 200. Examples of the detected straight lines are shown below.

[0042] FIG. 11 is a diagram illustrating a line detection process according to the second embodiment. The detection unit 190 detects multiple lines from an image 500. For example, the multiple lines are lines 501 and 502. The detection unit 190 detects the line that forms the smallest angle with the X-axis direction from the multiple lines. For ease of understanding, FIG. 11 illustrates a dashed line 501a in which the line 501 is lowered downward. For example, the detection unit 190 calculates the angle between the X-axis direction and dashed line 501a. In this way, the detection unit 190 calculates the angle, thereby detecting the line that forms the smallest angle with the X-axis direction from the multiple lines. For example, the detection unit 190 detects the line 501 that forms the smallest angle with the X-axis direction from the multiple lines. The detected line is referred to as line V.

[0043] (Step S24d) The calculation unit 191 calculates a rotation matrix R in which the X-axis direction is approximated to a straight line V. ｍ In other words, the calculation unit 191 calculates a rotation matrix R such that information in the X-axis direction (i.e., vector information) approximates a straight line V. ｍ (Step S24e) The calculation unit 191 calculates the correction transformation matrix B and the rotation matrix R ｍ and calculates the correction transformation matrix C. Specifically, the calculation unit 191 calculates the correction transformation matrix C using equation (4). The correction transformation matrix C is also referred to as a second correction transformation matrix. θ ｍ is the angle between the information in the X-axis direction and the line V.

[0044]

[0045] After step S24e, the process proceeds to step S25. (Step S26a) The correction processing unit 170 corrects the coordinates of the AR information using the correction transformation matrix C. Specifically, the correction processing unit 170 corrects the coordinates of the AR information as shown in equation (3).

[0046] According to the second embodiment, the display control device 100 calculates the rotation matrix R ｍ By correcting the coordinates of the AR information using the correction transformation matrix C that takes into account the above, the AR information can be displayed at a more accurate position.

[0047] Embodiment 3 Next, embodiment 3 will be described. In embodiment 3, differences from embodiments 1 and 2 will be mainly described. Furthermore, in embodiment 3, descriptions of matters common to embodiments 1 and 2 will be omitted.

[0048] A brief description of the third embodiment will be given. FIG. 12 is a diagram for explaining an overview of the third embodiment. The display control device 100 can display AR information 600. For example, the AR information 600 is explanatory information about an object (e.g., a robot 700). When the distance between the AR information 600 and the object is equal to or greater than a threshold, the display control device 100 performs processing to bring the AR information 600 closer to the object.

[0049] The processing will be described in detail below. Fig. 13 is a flowchart showing an example (part 1) of processing executed by the display control device of embodiment 3. Fig. 14 is a flowchart showing an example (part 2) of processing executed by the display control device of embodiment 3. The processing in Figs. 13 and 14 differs from the processing in Fig. 5 in that steps S26b to S26f are executed. Therefore, steps S26b to S26f will be described in Figs. 13 and 14. Description of processing other than steps S26b to S26f will be omitted.

[0050] (Step S26b) The detection unit 190 detects an object from the image acquired in step S13 based on the coordinates of the AR information. In other words, the detection unit 190 detects an object that exists near the coordinates of the AR information from the image acquired in step S13. The detection unit 190 may detect the object from a preprocessed image. (Step S26c) The detection unit 190 detects a feature point of the object. For example, the feature point is the feature point located closest to the coordinates of the AR information. (Step S26d) The calculation unit 191 calculates the distance between the coordinates of the AR information and the feature point.

[0051] (Step S26e) The correction processing unit 170 determines whether the distance is equal to or greater than a predetermined threshold. If the distance is equal to or greater than the threshold, the process proceeds to step S26f. If the distance is less than the threshold, the process proceeds to step S27. (Step S26f) The correction processing unit 170 performs correction on the coordinates of the AR information to bring the coordinates of the AR information closer to the feature point. For example, the correction processing unit 170 corrects the coordinates of the AR information so that the distance between the coordinates of the AR information and the feature point becomes a predetermined distance.

[0052] If there are multiple pieces of AR information, the display control device 100 executes steps S26b to S26f for each piece of AR information.

[0053] The display control device 100 may generate a correction transformation matrix to bring the AR information closer to the feature points, and may correct the AR information using the correction transformation matrix. Specifically, the calculation unit 191 calculates the correction transformation matrix D using equation (5). Note that α is a coefficient. T ｘｅ is the x-component of the distance between the coordinates of the AR information and the feature point. ｙｅ is the y component of the distance between the coordinate of the AR information and the feature point. ｚｅ is the z component of the distance between the coordinates of the AR information and the feature point.

[0054]

[0055] The correction processing unit 170 corrects the coordinates of the AR information using the correction transformation matrix D. Specifically, the correction processing unit 170 corrects the coordinates of the AR information as shown in equation (3).

[0056] According to the third embodiment, the display control device 100 can clarify the association between the AR information and the object by bringing the AR information closer to the object.

[0057] It is also possible to combine the second embodiment with the third embodiment. Specifically, the display control device 100 executes steps S26b to S26f after step S26a.

[0058] The embodiments can be modified in various ways within the scope of the present disclosure, and the features of the embodiments can be combined with each other as appropriate.

[0059] 10 Origin, 11 Origin, 100 Display control device, 101 Processor, 102 Volatile storage device, 103 Non-volatile storage device, 104 Imaging device, 105 Display device, 106 Sensor, 110 Storage unit, 120 Acquisition unit, 130 Construction unit, 140 Preprocessing unit, 150 Position and orientation estimation unit, 160 Image processing unit, 170 Correction processing unit, 180 Display control unit, 190 Detection unit, 191 Calculation unit, 200 Marker, 300 Floor, 301 Object, 400 AR information, 500 Image, 501, 502 Straight line, 501a Dashed line, 600 AR information, 700 Robot.

Claims

1. A sensor that outputs sensor data, which is data for detecting the direction of gravity, A construction unit that constructs a world coordinate system using the aforementioned sensor data, An acquisition unit that acquires an image including the marker, where the direction of one side of the marker is the direction of gravity, and AR (Augmented Reality) information, An image processing unit that, based on the aforementioned image, detects the marker, obtains a transformation matrix for the world coordinate system when the marker is detected, extracts a translation component and a rotation component from the transformation matrix, calculates the Euler angle of the axis corresponding to the direction of gravity using the rotation component, and calculates a first correction transformation matrix, which is a matrix, using the Euler angle and the translation component. A correction processing unit that corrects the coordinates of the AR information using the first correction transformation matrix, A display control unit that displays the AR information, A display control device that uses a thumbnail.

2. A detection unit detects information of a first direction other than the direction of gravity from the first correction transformation matrix, detects a plurality of straight lines from the image, and detects the straight line from the plurality of straight lines that has the smallest angle with the first direction, A calculation unit calculates a second correction transformation matrix using the rotation matrix in which the first direction approximates the straight line and the first correction transformation matrix, It further possesses, The correction processing unit corrects the coordinates of the AR information using the second correction transformation matrix. The display control device according to claim 1.

3. A detection unit detects an object from the image based on the coordinates of the AR information and detects the characteristic points of the object. A calculation unit that calculates the distance between the coordinates of the AR information and the feature point, It further possesses, The correction processing unit, when the distance is greater than or equal to a predetermined threshold, performs a correction on the coordinates of the AR information to bring them closer to the feature point. The display control device according to claim 1 or 2.

4. The system further includes a preprocessing unit that performs preprocessing on the aforementioned image, The image processing unit detects the marker based on the pre-processed image, The aforementioned image is a pre-processed image. The display control device according to claim 1 or 2.

5. A display control device having a sensor that outputs sensor data, which is data for detecting the direction of gravity, Using the aforementioned sensor data, a world coordinate system is constructed. An image containing the marker, whose side is oriented in the direction of gravity, and AR information are acquired; the marker is detected based on the image; a transformation matrix for the world coordinate system is acquired when the marker is detected; a translation component and a rotation component are extracted from the transformation matrix; the Euler angle of the axis corresponding to the direction of gravity is calculated using the rotation component; and a first correction transformation matrix, which is a matrix, is calculated using the Euler angle and the translation component. Using the first correction transformation matrix, the coordinates of the AR information are corrected. Displaying the aforementioned AR information, Display control method.

6. A display control device having a sensor that outputs sensor data, which is data for detecting the direction of gravity, Using the aforementioned sensor data, a world coordinate system is constructed. An image containing the marker, whose side is oriented in the direction of gravity, and AR information are acquired; the marker is detected based on the image; a transformation matrix for the world coordinate system is acquired when the marker is detected; a translation component and a rotation component are extracted from the transformation matrix; the Euler angle of the axis corresponding to the direction of gravity is calculated using the rotation component; and a first correction transformation matrix, which is a matrix, is calculated using the Euler angle and the translation component. Using the first correction transformation matrix, the coordinates of the AR information are corrected. Displaying the aforementioned AR information, A display control program that executes a process.