Display control device, display control method, and display control program

Abstract

A display control device (100) comprises: a sensor (106) that outputs sensor data that is data for detecting the direction of gravity; a construction unit (130) that constructs a world coordinate system using the sensor data; an acquisition unit (120) that acquires an image including a marker in which the direction of one side of the marker is the direction of gravity, and AR information; an image processing unit (160) that detects the marker on the basis of the image, acquires a conversion matrix for the world coordinate system when the marker is detected, extracts a parallel shift component and a rotational component from the conversion matrix, calculates the Euler angle of an axis that corresponds to the direction of gravity by using the rotational component, and calculates a first correction conversion matrix which is a matrix by using the Euler angle and the parallel shift component; a correction processing unit (170) that corrects the coordinates of the AR information by using the first correction conversion matrix; and a display control unit (180) that displays the AR information.

Description

[Technical Field]

[0001] This disclosure relates to a display control device, a display control method, and a display control program. [Background technology]

[0002] Augmented Reality (AR) is well known. Technologies related to AR have been proposed (see Patent Document 1). Patent Document 1 describes a method for aligning a virtual space with the real space. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-77070 [Non-patent literature]

[0004] [Non-Patent Document 1] <URL:https: / / learn.microsoft.com / en-us / uwp / api / windows.perception.spatial.spatialcoordinatesystem?view=winrt-22621> [Non-Patent Document 2] <URL:https: / / learn.microsoft.com / ja-jp / dotnet / api / system.numerics.matrix4x4.decompose?view=net-8.0> [Overview of the project] [Problems that the invention aims to solve]

[0005] By the way, if a device such as an HMD (Head Mounted Display) cannot accurately detect the marker, the AR information will be displayed in a different location than intended.

[0006] The purpose of this disclosure is to display AR information in the appropriate location. [Means for solving the problem]

[0007] A display control device according to one aspect of the present disclosure is provided. The display control device includes a sensor that outputs sensor data which is data for detecting the direction of gravity; a construction unit that constructs a world coordinate system using the sensor data; an acquisition unit that acquires an image including a marker whose side direction is the direction of gravity, and AR information; an image processing unit that detects the marker based on the image, acquires a transformation matrix for the world coordinate system when the marker is detected, extracts a translation component and a rotation component from the transformation matrix, calculates the Euler angle of the axis corresponding to the direction of gravity using the rotation component, and calculates a first correction transformation matrix which is a matrix using the Euler angle and the translation component; a correction processing unit that corrects the coordinates of the AR information using the first correction transformation matrix; and a display control unit that displays the AR information. [Effects of the Invention]

[0008] According to this disclosure, AR information can be displayed in the correct location. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing the display control device of Embodiment 1. [Figure 2] This diagram shows the hardware of the display control device of Embodiment 1. [Figure 3] This block diagram shows the functions of the display control device in Embodiment 1. [Figure 4] This flowchart shows an example (part 1) of the processing performed by the display control device of Embodiment 1. [Figure 5] This flowchart shows an example (part 2) of the processing performed by the display control device of Embodiment 1. [Figure 6] This figure shows a specific example (part 1) of the processing performed by the display control device of Embodiment 1. [Figure 7] It is a diagram showing a specific example (part 2) of the processing executed by the display control device of Embodiment 1. [Figure 8] It is a block diagram showing the functions of the display control device of Embodiment 2. [Figure 9] It is a flowchart showing an example (part 1) of the processing executed by the display control device of Embodiment 2. [Figure 10] It is a flowchart showing an example (part 2) of the processing executed by the display control device of Embodiment 2. [Figure 11] It is a diagram showing the straight line detection processing of Embodiment 2. [Figure 12] It is a diagram for explaining the outline of Embodiment 3. [Figure 13] It is a flowchart showing an example (part 1) of the processing executed by the display control device of Embodiment 3. [Figure 14] It is a flowchart showing an example (part 2) of the processing executed by the display control device of Embodiment 3.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described with reference to the drawings.

[0011] Embodiment 1. FIG. 1 is a diagram showing the display control device of Embodiment 1. The display control device 100 is a device that executes a display control method. For example, the display control device 100 is an HMD. The display control device 100 may also be a smartphone or a tablet terminal. FIG. 1 shows a marker 200. The marker 200 is also referred to as an AR marker. The display control device 100 can display AR information by detecting the marker 200. FIG. 1 shows a floor 300 and an object 301. For example, the marker 200 is attached to the object 301. Here, the direction of one side of the marker 200 is the gravitational direction. In other words, the gravitational direction of the marker 200 is the same as the Y-axis direction of the marker coordinate system.

[0012] Next, the hardware of the display control device 100 will be described. Figure 2 shows the hardware of the display control device of Embodiment 1. The display control device 100 includes a processor 101, a volatile storage device 102, a non-volatile storage device 103, an imaging device 104, a display device 105, and a sensor 106.

[0013] The processor 101 controls the entire display control device 100. For example, the processor 101 may be a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), or a GPU (Graphics Processing Unit). The processor 101 may also be a multiprocessor. The display control device 100 may also have processing circuits.

[0014] The volatile storage device 102 is the main memory of the display control device 100. For example, the volatile storage device 102 is RAM (Random Access Memory). The non-volatile storage device 103 is the auxiliary storage device of the display control device 100. For example, the non-volatile storage device 103 is an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0015] The imaging device 104 performs imaging. For example, the imaging device 104 images the marker 200. The display device 105 displays AR information. For example, the display device 105 is a display. The sensors 106 include a gyro sensor, an accelerometer, a magnetic sensor, a depth sensor, etc.

[0016] Next, the functions of the display control device 100 will be described. Figure 3 is a block diagram showing the functions of the display control device of Embodiment 1. The display control device 100 includes a storage unit 110, an acquisition unit 120, a construction unit 130, a preprocessing unit 140, a position and orientation estimation unit 150, an image processing unit 160, a correction processing unit 170, and a display control unit 180.

[0017] The storage unit 110 may be implemented as a storage area reserved in a volatile storage device 102 or a non-volatile storage device 103. The storage unit 110 may also be called a memory. Some or all of the acquisition unit 120, construction unit 130, preprocessing unit 140, position and orientation estimation unit 150, image processing unit 160, correction processing unit 170, and display control unit 180 may be implemented by processing circuits. Alternatively, some or all of the acquisition unit 120, construction unit 130, preprocessing unit 140, position and orientation estimation unit 150, image processing unit 160, correction processing unit 170, and display control unit 180 may be implemented as modules of a program executed by the processor 101. For example, the program executed by the processor 101 is also called a display control program or a display control program product. For example, the display control program is recorded on a recording medium.

[0018] The memory unit 110 stores various types of information. The functions of the acquisition unit 120, construction unit 130, preprocessing unit 140, position and orientation estimation unit 150, image processing unit 160, correction processing unit 170, and display control unit 180 will be explained later.

[0019] Next, the processes performed by the display control device 100 will be explained using a flowchart. Figure 4 is a flowchart showing an example (part 1) of the processing performed by the display control device of Embodiment 1. (Step S11) The acquisition unit 120 acquires sensor data. The sensor data is data for detecting the direction of gravity. The acquisition unit 120 acquires sensor data from the sensor 106 (for example, a gyro sensor, an accelerometer) that outputs sensor data. (Step S12) The construction unit 130 constructs a world coordinate system using the sensor data. Specifically, the construction unit 130 constructs a world coordinate system using the sensor data such that the positive direction of the axis corresponding to the direction of gravity in the world coordinate system (i.e., the Y-axis) is opposite to the direction of gravity. The world coordinate system may also be called the real-world space coordinate system.

[0020] (Step S13) The acquisition unit 120 acquires an image from the imaging device 104. (Step S14) The preprocessing unit 140 performs preprocessing on the image. For example, the preprocessing unit 140 performs resizing, grayscale conversion, noise reduction, etc. (Step S15) The acquisition unit 120 acquires sensor data from the sensor 106. (Step S16) The position and orientation estimation unit 150 estimates the position and orientation of the display control device 100 based on the pre-processed image and the sensor data acquired in step S15.

[0021] (Step S17) The image processing unit 160 performs marker detection processing based on the pre-processed image and the sensor data acquired in step S15. The image processing unit 160 may also perform marker detection processing based on the pre-processed image. If marker 200 is detected during the detection process, the process proceeds to step S21. If marker 200 is not detected, the process proceeds to step S13.

[0022] Figure 5 is a flowchart showing an example (part 2) of the processing performed by the display control device of Embodiment 1. (Step S21) When the image processing unit 160 detects the marker 200, it obtains a transformation matrix for the world coordinate system (hereinafter referred to as transformation matrix A). In other words, when the image processing unit 160 detects the marker 200, it obtains the position and orientation of the marker 200 as a transformation matrix A for the world coordinate system. It may also be said that the image processing unit 160 obtains a transformation matrix A for the world coordinate system based on the image. For example, when the image processing unit 160 obtains a transformation matrix, it obtains the transformation matrix using the API (Application Programming Interface) described in Non-Patent Document 1 (i.e., TryGetTransformTo()).

[0023] (Step S22) The image processing unit 160 extracts the translation component and the rotation component from the transformation matrix A. For example, when the image processing unit 160 extracts the translation component and the rotation component, it uses the API described in Non-Patent Document 2 (i.e., Decompose()) to extract the translation component and the rotation component.

[0024] (Step S23) The image processing unit 160 uses the rotation component to calculate the Euler angle of the axis corresponding to the direction of gravity (i.e., the Y-axis). The calculation method is explained below. Furthermore, when the API described in Non-Patent Document 2 is used, the rotation component is extracted as a quaternion. Therefore, in the following explanation, the rotation component will be represented as a quaternion. Additionally, in the following explanation, the order in which the Euler angles are applied will be ZXY. The image processing unit 160 calculates the Euler angle of the Y axis using equation (1).

[0025]

number

[0026] (Step S24) The image processing unit 160 calculates the correction transformation matrix B using the calculated Euler angles and the extracted translation components. Specifically, the image processing unit 160 calculates the correction transformation matrix B using equation (2). The correction transformation matrix B is also called the first correction transformation matrix. T is a translation matrix that represents the translation component. x , T y , T z R represents the displacement from the origin of the world coordinate system to the origin of the marker coordinate system. R represents the rotation matrix. S represents the scaling matrix.

[0027]

number

[0028] (Step S25) The acquisition unit 120 acquires AR information. For example, the acquisition unit 120 acquires AR information from the storage unit 110. Alternatively, for example, the acquisition unit 120 acquires AR information from an external device. The external device is a device located outside the display control device 100. For example, the external device is a cloud server. The diagram of the external device is omitted.

[0029] (Step S26) The correction processing unit 170 corrects the coordinates of the AR information using the correction transformation matrix B. Specifically, the correction processing unit 170 corrects the coordinates of the AR information using equation (3). (x,y,z,1) represents the coordinates of the AR information before the change. (x',y',z',1) represents the coordinates of the AR information after the change. Note that the coordinates of the AR information are the coordinates that indicate the position where the AR information is displayed.

[0030]

number

[0031] (Step S27) The display control unit 180 displays AR information on the display of the display control device 100.

[0032] Step S14 describes the case where preprocessing is performed. The display control device 100 does not have to perform preprocessing. If preprocessing is not performed, the position and orientation estimation unit 150 estimates the position and orientation of the display control device 100 based on the image acquired in step S13 and the sensor data acquired in step S15. The image processing unit 160 also performs marker 200 detection processing based on the image acquired in step S13 and the sensor data acquired in step S15. The image processing unit 160 may also perform marker 200 detection processing based on the image acquired in step S13.

[0033] Next, a specific example of the processing performed by the display control device 100 is shown. Figure 6 shows a specific example (part 1) of the processing performed by the display control device of Embodiment 1. The acquisition unit 120 acquires sensor data from the sensor 106. The construction unit 130 uses the sensor data to construct a world coordinate system such that the positive direction of the Y axis of the world coordinate system is opposite to the direction of gravity. Figure 6 shows the origin 10 of the world coordinate system. Figure 6 shows marker 200. Figure 6 also shows the marker coordinate system. Furthermore, Figure 6 shows the origin 11 of the marker coordinate system.

[0034] The acquisition unit 120 acquires an image including the marker 200 from the imaging device 104. The preprocessing unit 140 performs preprocessing on the image. The acquisition unit 120 acquires sensor data from the sensor 106. The image processing unit 160 detects the marker 200 based on the preprocessed image and sensor data. The image processing unit 160 obtains a transformation matrix A. The image processing unit 160 extracts the translation component and the rotation component from the transformation matrix A. The image processing unit 160 calculates the Euler angle of the Y-axis using the rotation component. The image processing unit 160 calculates a correction transformation matrix B using the Euler angle and the translation component.

[0035] Figure 7 shows a specific example (part 2) of the processing performed by the display control device of Embodiment 1. The acquisition unit 120 acquires AR information 400. The correction processing unit 170 corrects the coordinates of the AR information 400 using the correction transformation matrix B. The display control unit 180 displays the AR information 400 on the display of the display control device 100.

[0036] According to Embodiment 1, even if the marker 200 cannot be accurately detected, the display control device 100 can display the AR information in the correct position by correcting the coordinates of the AR information using the correction transformation matrix B. In other words, the display control device 100 can display the AR information in the correct position by correcting the deviation caused by the detection of the marker 200 (in other words, the inaccurate detection of the marker 200) using the correction transformation matrix B. In this way, the display control device 100 can display the AR information in the correct position.

[0037] Embodiment 2. Next, Embodiment 2 will be described. Embodiment 2 will mainly describe the differences from Embodiment 1. In Embodiment 2, the explanation of matters common to Embodiment 1 will be omitted.

[0038] Figure 8 is a block diagram showing the functions of the display control device of Embodiment 2. The display control device 100 further includes a detection unit 190 and a calculation unit 191. Some or all of the detection unit 190 and the calculation unit 191 may be implemented by processing circuits. Alternatively, some or all of the detection unit 190 and the calculation unit 191 may be implemented as modules of a program executed by the processor 101. The functions of the detection unit 190 and the calculation unit 191 will be described later.

[0039] Next, the processes performed by the display control device 100 will be explained using a flowchart. Figure 9 is a flowchart showing an example of processing (part 1) performed by the display control device of Embodiment 2. Figure 10 is a flowchart showing an example of processing (part 2) performed by the display control device of Embodiment 2. The processing in Figures 9 and 10 differs from the processing in Figure 5 in that steps S24a to 24e and 26a are executed. Therefore, steps S24a to 24e and 26a will be explained in Figures 9 and 10. The explanation of processing other than steps S24a to 24e and 26a will be omitted.

[0040] (Step S24a) The detection unit 190 detects information for a first direction other than the direction of gravity (i.e., vector information) from the correction transformation matrix B. Specifically, the first direction is either the X-axis direction or the Z-axis direction. In the following description, the first direction will be the X-axis direction. (Step S24b) The detection unit 190 detects multiple straight lines from the image acquired in step S13. The detection unit 190 can detect multiple straight lines by performing image analysis processing. Long straight lines are preferable. The detection unit 190 may also detect multiple straight lines from a pre-processed image.

[0041] (Step S24c) The detection unit 190 detects the line with the smallest angle with the X-axis from among multiple lines. In other words, the detection unit 190 detects the line with the smallest angle with the X-axis information (i.e., vector information) from among multiple lines. It is desirable that the detected line be a line near the marker 200. An example of a detected line is shown.

[0042] Figure 11 shows the line detection process of Embodiment 2. The detection unit 190 detects multiple lines from the image 500. For example, the multiple lines are lines 501 and 502. The detection unit 190 detects the line with the smallest angle with the X-axis direction from among several lines. Figure 11 shows the dashed line 501a, which is the line 501 extended downwards, for ease of understanding. For example, the detection unit 190 calculates the angle between the X-axis direction and the dashed line 501a. In this way, the detection unit 190 detects the line with the smallest angle with the X-axis direction from among several lines by calculating the angle. For example, the detection unit 190 detects the line 501 with the smallest angle with the X-axis direction from among several lines. The detected line is called line V.

[0043] (Step S24d) The calculation unit 191 calculates a rotation matrix R that approximates the X-axis direction to a straight line V. m The calculation unit 191 calculates the rotation matrix R that approximates the information in the X-axis direction (i.e., vector information) to a straight line V. m Calculate. (Step S24e) The calculation unit 191 calculates the correction transformation matrix B and the rotation matrix R m The correction transformation matrix C is calculated using the following. Specifically, the calculation unit 191 calculates the correction transformation matrix C using equation (4). The correction transformation matrix C is also called the second correction transformation matrix. θ m This is the angle between the information in the X-axis direction and the line V.

[0044]

number

[0045] After step S24e, the process proceeds to step S25. (Step S26a) The correction processing unit 170 corrects the coordinates of the AR information using the correction transformation matrix C. Specifically, the correction processing unit 170 corrects the coordinates of the AR information as shown in equation (3).

[0046] According to Embodiment 2, the display control device 100 uses a rotation matrix R m By correcting the coordinates of the AR information using a correction transformation matrix C that takes this into account, the AR information can be displayed in a more accurate position.

[0047] Embodiment 3. Next, Embodiment 3 will be described. Embodiment 3 will mainly describe the differences from Embodiments 1 and 2. In Embodiment 3, the explanation of matters common to Embodiments 1 and 2 will be omitted.

[0048] Embodiment 3 will be briefly described. Figure 12 is a diagram illustrating the overview of Embodiment 3. The display control device 100 can display AR information 600. For example, the AR information 600 is descriptive information about an object (e.g., a robot 700). If the distance between the AR information 600 and the object is greater than or equal to a threshold, the display control device 100 processes the AR information 600 to move it closer to the object.

[0049] The process will be explained in detail below. Figure 13 is a flowchart showing an example of processing (part 1) performed by the display control device of Embodiment 3. Figure 14 is a flowchart showing an example of processing (part 2) performed by the display control device of Embodiment 3. The processing in Figures 13 and 14 differs from the processing in Figure 5 in that steps S26b to S26f are executed. Therefore, steps S26b to S26f will be explained in Figures 13 and 14. The explanation of processing other than steps S26b to S26f will be omitted.

[0050] (Step S26b) The detection unit 190 detects the target object from the images acquired in step S13 based on the coordinates of the AR information. In other words, the detection unit 190 detects the target object existing near the coordinates of the AR information from the images acquired in step S13. The detection unit 190 may detect the target object from the pre-processed images. (Step S26c) The detection unit 190 detects the feature points of the target object. For example, the feature points are the feature points at the position closest to the coordinates of the AR information. (Step S26d) The calculation unit 191 calculates the distance between the coordinates of the AR information and the feature points.

[0051] (Step S26e) The correction processing unit 170 determines whether the distance is equal to or greater than a predetermined threshold value. If the distance is equal to or greater than the threshold value, the process proceeds to step S26f. If the distance is less than the threshold value, the process proceeds to step S27. (Step S26f) The correction processing unit 170 performs correction on the coordinates of the AR information to bring the coordinates of the AR information closer to the feature points. For example, the correction processing unit 170 corrects the coordinates of the AR information so that the distance between the coordinates of the AR information and the feature points becomes a predetermined distance.

[0052] When there are multiple pieces of AR information, the display control device 100 executes steps S26b to 26f for each piece of AR information.

[0053] The display control device 100 may generate a correction transformation matrix to bring the AR information closer to the feature points, and correct the AR information using the correction transformation matrix. Specifically, the calculation unit 191 calculates the correction transformation matrix D using Equation (5). Here, α is a coefficient. T xe is the x component of the distance between the coordinates of the AR information and the feature points. T ye is the y component of the distance between the coordinates of the AR information and the feature points. T ze is the z component of the distance between the coordinates of the AR information and the feature points.

[0054]

Equation

[0055] The correction processing unit 170 corrects the coordinates of the AR information using the correction transformation matrix D. Specifically, the correction processing unit 170 corrects the coordinates of the AR information as shown in equation (3).

[0056] According to Embodiment 3, the display control device 100 can clarify the relationship between the AR information and the object by bringing the AR information closer to the object.

[0057] Embodiment 2 and Embodiment 3 may be combined. Specifically, the display control device 100 executes steps S26b to S26f after step S26a.

[0058] The embodiments can be modified in various ways within the scope of this disclosure. Furthermore, the features of each embodiment can be combined with each other as appropriate. [Explanation of Symbols]

[0059] 10 Origin, 11 Origin, 100 Display control device, 101 Processor, 102 Volatile memory device, 103 Non-volatile memory device, 104 Imaging device, 105 Display device, 106 Sensor, 110 Memory unit, 120 Acquisition unit, 130 Construction unit, 140 Preprocessing unit, 150 Position and orientation estimation unit, 160 Image processing unit, 170 Correction processing unit, 180 Display control unit, 190 Detection unit, 191 Calculation unit, 200 Marker, 300 Floor, 301 Object, 400 AR information, 500 Image, 501, 502 Line, 501a Dashed line, 600 AR information, 700 Robot.

Claims

1. A sensor that outputs sensor data, which is data for detecting the direction of gravity, A construction unit that constructs a world coordinate system using the aforementioned sensor data, An acquisition unit that acquires an image including a marker whose side is oriented in the direction of gravity, and AR (Augmented Reality) information, An image processing unit that, based on the aforementioned image, detects the marker, obtains a transformation matrix for the world coordinate system when the marker is detected, extracts a translation component and a rotation component from the transformation matrix, calculates the Euler angle of the axis corresponding to the direction of gravity using the rotation component, and calculates a first correction transformation matrix, which is a matrix, using the Euler angle and the translation component. A correction processing unit that corrects the coordinates of the AR information using the first correction transformation matrix, A display control unit that displays the AR information, A display control device that uses a thumbnail.

2. A detection unit detects information of a first direction other than the direction of gravity from the first correction transformation matrix, detects a plurality of straight lines from the image, and detects the straight line from the plurality of straight lines that has the smallest angle with the first direction, A calculation unit calculates a second correction transformation matrix using the rotation matrix in which the first direction approximates the straight line and the first correction transformation matrix, It further possesses, The correction processing unit corrects the coordinates of the AR information using the second correction transformation matrix. The display control device according to claim 1.

3. A detection unit detects an object from the image based on the coordinates of the AR information and detects the characteristic points of the object. A calculation unit that calculates the distance between the coordinates of the AR information and the feature point, It further possesses, The correction processing unit, when the distance is greater than or equal to a predetermined threshold, performs a correction on the coordinates of the AR information to bring them closer to the feature point. The display control device according to claim 1 or 2.

4. The system further includes a preprocessing unit that performs preprocessing on the aforementioned image, The image processing unit detects the marker based on the pre-processed image, The aforementioned image is a pre-processed image. The display control device according to claim 1 or 2.

5. A display control device having a sensor that outputs sensor data, which is data for detecting the direction of gravity, Using the aforementioned sensor data, a world coordinate system is constructed. An image containing the marker, whose side is oriented in the direction of gravity, and AR information are acquired; the marker is detected based on the image; a transformation matrix for the world coordinate system is acquired when the marker is detected; a translation component and a rotation component are extracted from the transformation matrix; the Euler angle of the axis corresponding to the direction of gravity is calculated using the rotation component; and a first correction transformation matrix, which is a matrix, is calculated using the Euler angle and the translation component. Using the first correction transformation matrix, the coordinates of the AR information are corrected. Displaying the aforementioned AR information, Display control method.

6. A display control device having a sensor that outputs sensor data, which is data for detecting the direction of gravity, Using the aforementioned sensor data, a world coordinate system is constructed. An image containing the marker, whose side is oriented in the direction of gravity, and AR information are acquired; the marker is detected based on the image; a transformation matrix for the world coordinate system is acquired when the marker is detected; a translation component and a rotation component are extracted from the transformation matrix; the Euler angle of the axis corresponding to the direction of gravity is calculated using the rotation component; and a first correction transformation matrix, which is a matrix, is calculated using the Euler angle and the translation component. Using the first correction transformation matrix, the coordinates of the AR information are corrected. Displaying the aforementioned AR information, A display control program that executes a process.

Images