Image display device

Abstract

An image display device is provided that can easily correct the positional deviation even if the user moves and the alignment deviates. The image display device (100) has a correction processing unit (102) that corrects the deviation when the image data (G) in the user's field of vision deviates from the steel reinforcement frame (T) due to the user's (U) movement. The correction processing unit (102) uses the design data stored in the storage unit (103) and the user's (U) own position relative to the steel reinforcement frame (T) to obtain a coordinate modification position (S) as a reference position for correction processing from some components, such as the steel reinforcement frame (T1). The correction processing unit (102) performs positional alignment of the image data (G) based on this coordinate modification position (S), thereby performing correction.

Description

Technical Field

[0001] This invention relates to an image display device that utilizes Mixed Reality (MR) space. Background Technology

[0002] It is known that there are techniques for combining images with objects located in real space using MR technology. For example, Patent Document 1 describes an overlay depiction of real-space images and virtual-space images.

[0003] Existing technical documents

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2008-293209 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] In recent years, there has been a desire to use MR technology to synthesize image data of decorative materials and other components with the framework and beams during the construction phase of a building, thereby making it easier to control the finished product.

[0008] However, there are many situations where it is difficult to align the image data position on a structure. Generally, it is considered to attach markers to the structure for image data alignment. In this case, when a user wearing MR goggles or other protective eyewear that allows observation of the MR space moves away from the marked structure, the alignment between the structure and the image data deviates. Therefore, it is also considered to place markers corresponding to the position at various locations on the structure, but setting the markers and this placement is very time-consuming.

[0009] Therefore, in order to solve the above-mentioned problems, the objective is to provide an image display device that can easily correct the positional alignment deviation caused by the user's movement in a mixed space.

[0010] Methods for solving problems

[0011] The image display device of the present invention synthesizes and displays a predetermined image of a composite object located in real space. The image display device includes: a storage unit that stores design data representing the positional relationships of various parts of the composite object; a display unit that generates and displays a composite image obtained by synthesizing a predetermined image using one part of the composite object as a reference position; and a correction unit that corrects deviations between the predetermined image and the composite object caused by user movement. The correction unit uses the design data and the user's position relative to the composite object to obtain a coordinate modification position of the reference position in the one part of the composite object used for correction processing, and corrects the composite position of the predetermined image based on this coordinate modification position.

[0012] Invention Effects

[0013] According to the present invention, it is possible to easily correct positional alignment deviations in mixed reality space. Attached Figure Description

[0014] Figure 1 A diagram showing the structure, namely the steel reinforcement cage T, in this embodiment, and a diagram showing the composite image data G of the column in the steel reinforcement cage T.

[0015] Figure 2 This is an explanatory diagram about modifying the coordinate position.

[0016] Figure 3 This is an explanatory diagram showing the composite process that uses coordinates to modify the position.

[0017] Figure 4 This is a block diagram showing the functional structure of the image display device 100.

[0018] Figure 5 This is a flowchart illustrating the operation of the image display device 100.

[0019] Figure 6 This is a flowchart showing the detailed actions of processing S101.

[0020] Figure 7 This is a flowchart showing the detailed process of selecting and correcting the self-position and attitude estimation results in the correction processing unit 102.

[0021] Figure 8 This is a flowchart illustrating the process of selecting a coordinate modification position to be used from the candidates for coordinate modification positions.

[0022] Figure 9 This is a flowchart showing the calculation process of the deviation of the coordinate modification position in the correction processing unit 102.

[0023] Figure 10 This is a flowchart illustrating additional processing steps when calculating the positional deviation between image data G and the reinforcing steel skeleton T.

[0024] Figure 11 This is a diagram illustrating an example of the hardware structure of an image display device 100 according to one embodiment of the present disclosure. Detailed Implementation

[0025] Embodiments of the present invention will be described with reference to the accompanying drawings. Where possible, the same reference numerals are used to denote the same parts, and repeated descriptions are omitted.

[0026] Figure 1 This is a diagram showing the structure, namely the steel reinforcement cage T, in this embodiment, and a diagram showing image data G, representing decorative material that is a column, synthesized in the steel reinforcement cage T. Figure 1 The image shown is what a user wearing the image display device of this embodiment, i.e., MR goggles, can see. Figure 1 (a) is a diagram showing the structure, namely the steel frame T, existing in real space before the image data G of the decorative material is synthesized. Figure 1 In the design, the reinforcing cage T is composed of reinforcing cages T to Tn. Figure 1 The diagram shows up to the rebar cage T9, but it is also possible for up to the rebar cage Tn (where n is a natural number). The same applies to the image data Gn (where n is a natural number, for example, G1 to G9).

[0027] Furthermore, in this embodiment, the column is set as a steel reinforcement frame, but the column is not limited to a steel reinforcement frame; it can also be a concrete column, or it can be covered by decorative materials. The image synthesized and displayed is not limited to decorative materials; it can also be pipes, wiring, or furniture, etc.

[0028] Figure 1 (b) is a diagram showing the state of the actual steel reinforcement skeleton T covered by decorative materials. It is a diagram created in mixed reality space by combining the image data G representing the decorative material with the steel reinforcement skeleton T. Design data indicating the position and size of the steel reinforcement skeletons T1-Tn, as well as the image data G1-Gn of the decorative materials, are pre-stored. The image data G is a three-dimensional image composed of the image data G1-Gn. The image data G1-Gn of multiple decorative materials are pre-associated with the positions of the steel reinforcement skeletons T1-Tn.

[0029] The modified corresponding parts, described later, are set in each image data Gn. For example, when the modified corresponding part of the decorative material's image data G1 is aligned with the coordinate modification position set on the reinforcing steel frame T1 for composite composition, the image data G2 of other decorative materials is also aligned with the corresponding reinforcing steel frame T2. Similarly, the image data Gn of other decorative materials are also aligned with the corresponding reinforcing steel frame Tn. However, there may be cases where the alignment of the modified corresponding parts of image data G2, etc., with the coordinate modification position set on the reinforcing steel frame T2, etc., contains errors.

[0030] When the image data G of the reinforcing steel skeleton T and the decorative material is initially aligned, the alignment is performed, for example, by attaching an initial mark for image data G1 to a predetermined position on the reinforcing steel skeleton T1. Since image data G2 and the like constitute image data G with mutually fixed positional relationships, the alignment is also automatically performed for other reinforcing steel skeletons T2 and the like. Furthermore, since image data G is a three-dimensional image, it is rotated according to the direction in which the user observes the reinforcing steel skeleton T. In this way, by performing alignment based on the initial mark, the alignment of all reinforcing steel skeletons T1 to Tn with image data G1 to Gn can be performed.

[0031] The alignment of the positions will be explained in further detail. Figure 2 Indicate the location where the coordinates are modified. Figure 1 The text only describes the alignment process. For example... Figure 2 As shown, the reinforcing steel skeleton T1 has surfaces in four directions. The image data G1 of the decorative material is aligned with one of these surfaces as a reference. The coordinate modification position indicates the reference position used for aligning the position of the reinforcing steel skeleton T with the position of the image data G of the decorative material.

[0032] Figure 2 (a) is a schematic diagram viewed from the upper surface of the reinforcing cage T. In this embodiment, the reinforcing cage T is a prism with the intersection point C of the positioning axes at its center, and its cross-section is quadrilateral. Here, we focus on the reinforcing cage T1. Coordinate modification positions S11 to S14 are set on each face of the reinforcing cage T1.

[0033] When the user moves while observing the rebar skeleton T1, the positions of other rebar skeletons T2-T4 (decorative materials K2-K4) deviate from the image data G2-G4 of the decorative materials. Therefore, this deviation needs to be corrected. Specifically, when the user moves while observing the rebar skeleton T1, the initially set reference marker disappears from the field of view, making it impossible to track the position and angle based on that reference marker, and thus preventing SLAM-based self-position and pose estimation tracking. This self-position and pose estimation introduces estimation errors due to movement.

[0034] In this embodiment, the nearest rebar cage T to the user and its coordinate modification position S are determined, thereby determining the rebar cage T as the modification reference.

[0035] Figure 2 (b) is a diagram showing the nearest steel reinforcement frame T2 to the user U when the user U, wearing the image display device 100 as MR goggles, moves, and the coordinate modification positions S21 to S24. In this embodiment, the image display device 100 can use known self-position estimation techniques to determine the position of the user U relative to the steel reinforcement frame T and its relative positioning relationship.

[0036] By having user U observe the rebar cage T2, the image display device 100 determines which of the coordinate modification positions S21 to S24 of the rebar cage T2 closest to user U should be set as the modification reference. For example... Figure 3 As shown in (a), the image display device 100 selects the opposite direction vector of the user U's line of sight (which is equivalent to the shooting direction of the camera in the image display device) and the normal vector of the surface of the steel frame T as the coordinate modification position S within a predetermined angle.

[0037] Furthermore, based on the distance to the reinforcing bar cage T measured by the depth sensor, the actual coordinate modification position defined on the surface of the reinforcing bar cage T is calculated. Figure 3 (b)). The depth sensor is not limited to measuring the rebar cage T from the front. Therefore, the coordinates are modified to a predetermined position on the surface of the rebar cage T (e.g., the central part).

[0038] Then, based on these coordinates, the position is modified to shift the corresponding part Sh2 of the image data G. Figure 3 (c)). Image data G2 and the information of the corresponding modified parts associated with each coordinate modification position S (in Figure 3 The relative position information in the image data G is associated with the position S22. That is, each face of each image data G is associated with a reference position used for aligning with the coordinate modification position set on each face of each steel reinforcement cage T. This aligns the corresponding modification part Sh2 with the coordinate modification position S12, thereby enabling the alignment of the image data G with the steel reinforcement cage T.

[0039] By performing this processing, user U can see the state where the steel frame T is covered by the image data G of the decorative material while moving around. Additionally, in Figure 2 and Figure 3 In this design, the cross-section of the reinforcing steel frame T is quadrilateral, but it is not limited to this. It can also be circular or other polygonal shapes, as long as the coordinates of the reinforcing steel frame T are modified and associated with the corresponding parts of the image data G of the decorative material.

[0040] Next, the functional structure of the image display device 100 of this embodiment will be described. Figure 4 This is a block diagram showing the functional structure of the image display device 100. As shown, the image display device 100 is configured to include a camera 101, a correction processing unit 102 (correction unit), a storage unit 103, a display unit 104, a depth sensor 105 (sensor unit), a calculation processing unit 106, and a self-position and pose estimation unit 107. This image display device 100 is a device called a so-called MR goggles, and is a head-mounted display type device. The user wears the image display device 100 on their head, thereby being able to view the images captured by the camera 101 through the display unit 104. In a video see-through type head-mounted display, the user can view the images captured by the camera 101 through the display unit 104 by wearing the image display device 100 on their head. Furthermore, this embodiment is not limited to these methods, and can also be implemented in the same way in optical see-through type devices (e.g., viewing a steel frame through glass).

[0041] Camera 101 is the part that captures the real space as observed by the user.

[0042] The correction processing unit 102 is the part that synthesizes image data from images captured by the camera 101 in the mixed reality space into virtual space components. Here, processing is performed to synthesize image data G representing decorative materials with the steel frame T, as well as correction processing for positional deviations.

[0043] Storage unit 103 stores design data for structures such as steel reinforcement cages T and image data G representing decorative materials for those structures. The design data includes information such as the size and spacing of each steel reinforcement cage T. Furthermore, the image data G contains images of the decorative materials, as well as information indicating their correspondence with the steel reinforcement cages T (particularly coordinate modification positions).

[0044] Display unit 104 is the part that displays the image captured by camera 101. When image data G is synthesized for the steel reinforcement skeleton T by correction processing unit 102, display unit 104 displays the synthesized image.

[0045] The depth sensor 105 is a sensor that measures the distance between the user U and the steel reinforcement cage T.

[0046] The calculation processing unit 106 calculates the initial position of the image display device 100 relative to the structure (steel skeleton T) and the positional deviation between the steel skeleton T and the image data G that occurs whenever the user moves, based on the design data stored in the storage unit 103.

[0047] The self-position and pose estimation unit 107 estimates the position and pose changes of the image display device 100 in the real space captured by the camera 101 based on the images captured by the camera 101. The self-position and pose estimation unit 107, based on the initial position calculated by the calculation processing unit 106, the image of the initial marker at this time, and the currently captured image of the steel reinforcement frame T, determines the direction and relative distance of the steel reinforcement frame T from the captured images, thereby estimating the relative current position (current position) with respect to the steel reinforcement frame T. This self-position and pose estimation unit 107 is a known technology. For example, it can be implemented using SLAM (Simultaneous Localization and Mapping). Furthermore, the self-position and pose estimation function is not limited to the above; GPS or other sensors can also be used for self-position estimation.

[0048] The operation of the image display device 100 configured in this manner will be explained. Figure 5 This is a flowchart illustrating the action.

[0049] The calculation and processing unit 106 calculates the coordinate modification position S in each steel reinforcement cage T based on the design data stored in the storage unit 103, and stores it in the storage unit 103 (S101).

[0050] Next, the calculation and processing unit 106 calculates the initial position of its own position and posture estimation unit 107 and grasps the positional relationship of the steel reinforcement cage T (S102).

[0051] When the user moves (S103), an error occurs in the estimation of their own position and pose (S104).

[0052] Then, the correction processing unit 102 selects a coordinate modification position as a reference from the coordinate modification position candidates based on its own position estimated by the self-position and posture estimation unit 107. Then, the correction processing unit 102 corrects the position of the image data G of the decorative material based on the self-position estimation result (self-position) and the selected coordinate modification position (S105). S103 to S106 are repeated until the program ends (S106).

[0053] Figure 6This is a flowchart illustrating the detailed actions of processing S101. The calculation processing unit 106, referring to the storage unit 103, reads the design data of the building (steel frame T) (S201). The calculation processing unit 106 calculates the coordinate modification position based on the floor height information and the positioning axis information (S202). Here, the floor height information indicates the number of floors and the height of the building. The positioning axis information represents the centerline of the steel frame, indicating the coordinates within the building. The coordinate modification position refers to the reference position of the image data G used to synthesize the decorative material; here, it refers to the three-dimensional coordinates calculated based on the coordinates and height of the coordinate modification position defined at the intersection C of the positioning axes.

[0054] The calculation and processing unit 106 sets the position after shifting the coordinate modification position by Y meters in the height direction as the coordinate modification position (S203). This processing is to align with the position that is in line with the viewer's line of sight.

[0055] The calculation and processing unit 106 determines whether the final coordinate modification position is located within the steel reinforcement cage T of a wall or column, etc. (S204). For example, the coordinates (horizontal coordinates) of the intersection of the positioning axes are determined to be located inside the steel reinforcement cage T. The same applies when the steel reinforcement cage T is embedded in the wall. This information is determined based on the size and configuration of the steel reinforcement cage T contained in the design data.

[0056] When the calculation processing unit 106 determines that there is a coordinate modification position in the steel reinforcement cage T, it splits the coordinate modification position in four directions centered on the intersection of the positioning axis, calculates the intersection point of the design data with the surface of the steel reinforcement cage T as the coordinate modification position, and sets it (S205).

[0057] The calculation processing unit 106 stores the coordinate modification positions calculated based on these processes in the storage unit 103 (S206). The storage unit 103 stores, according to each reinforcing bar skeleton T (reinforcing bar skeletons T1 to Tn), the identification information of the decorative material associated with that reinforcing bar skeleton T, the arrangement position (relative position information) of the decorative material, the corresponding modification part in the image data G of the decorative material corresponding to the coordinate modification position set for each reinforcing bar skeleton T, and the image data G (image data G1 to Gn) of the decorative material. That is, the storage unit 103 stores various information to associate each coordinate modification position of the reinforcing bar skeleton T with which image data G (G1 to Gn) is synthesized.

[0058] Next, the detailed processing of S105 will be explained. Figure 7 This is a flowchart showing the detailed processes of selecting and correcting the self-position and attitude estimation results in the correction processing unit 102.

[0059] The correction processing unit 102 selects a coordinate modification position as a reference from the coordinate modification position candidates stored in the storage unit 103 (S301). If there are candidates, the correction processing unit 102 overlays the message information indicating that it is a coordinate modification position according to the selected coordinate modification position, and the display unit 104 displays the meaning (S303).

[0060] The correction processing unit 102 calculates the deviation between the coordinate modification position and the coordinates in actual space (S304). That is, the correction processing unit 102 calculates the positional deviation between the specified coordinate modification position in the reinforcing steel frame T and the positional deviation between the image data G of the decorative material being synthesized. For example, in... Figure 3 In (b), the depth sensor 105 measures the distance to the surface of the reinforcing steel skeleton T (coordinate modification position) at the location where the height of the intersection point C2 of the positioning axes has shifted by Y. The correction processing unit 102 calculates the actual position of the coordinate modification position based on this distance. On the other hand, the correction processing unit 102 determines the position of the image data G2 of the decorative material (set as the position to be aligned with the coordinate modification position, here set as the corresponding modification location), and calculates the difference as the position deviation (error). Since the coordinate modification position is the position shifted by Y in the height direction and is located at the center of the thickness of the reinforcing steel skeleton in the lateral direction, this position can be determined by calculation or by user instruction using the user input function of this device.

[0061] When the correction processing unit 102 calculates a positional deviation (the deviation is greater than or equal to a predetermined value) (S305: Yes), it modifies the overlapping position of the image data G based on the amount of deviation (S306).

[0062] Next, regarding Figure 7 The detailed processing of S301 in the process will be explained. Figure 8 This is a flowchart illustrating the process of selecting a coordinate modification position to be used from the candidates for coordinate modification positions.

[0063] The correction processing unit 102 reads the coordinate modification position in the steel reinforcement cage T from the storage unit 103 (S401) and obtains the result of the self-position and posture estimation based on the self-position and posture estimation unit 107 (relative self-position information relative to the steel reinforcement cage T) (S402).

[0064] Based on its own position and posture estimation results, the correction processing unit 102 extracts multiple coordinate modification positions S located near its own position (within a predetermined range) (S403). Furthermore, based on its own position and posture estimation results, the correction processing unit 102 calculates the direction N1 of the user toward the reinforcing bar cage T (S404). That is, the self-position and posture estimation unit 107 calculates the angle between the direction vector of the normal N2 of the surface of each reinforcing bar cage T and the opposite direction vector of direction N1.

[0065] The correction processing unit 102 excludes surfaces with coordinate modification positions having an angle of φ degrees or more. That is, when the user observes the surface of the reinforcing bar cage T from an inclined direction, coordinate modification positions on the surface of these reinforcing bar cage Ts are excluded (S405).

[0066] The correction processing unit 102 determines whether the distance between the nearest coordinate modification position S and the second nearest coordinate modification position S is less than a predetermined distance (S406). If the distance is less than the predetermined distance, it is determined that there are no candidates (S408). If the distance is greater than or equal to the predetermined distance, the nearest coordinate modification position S is selected (S407).

[0067] The correction processing unit 102 selects a coordinate modification position S through this processing.

[0068] Next, regarding Figure 7 The process S304 in the document describes the detailed process. Figure 9 This is a flowchart showing the calculation process of the deviation of the coordinate modification position in the correction processing unit 102. Figure 9 The treatment using reference marks is shown for each part of the reinforcing steel cage T.

[0069] When a reference marker is captured in the image by camera 101, the correction processing unit 102 calculates the position P1 of the reference marker (S501, S502) based on its own position estimated by the self-position and posture estimation unit 107. The correction processing unit 102 has prior information such as the shape and size of the reference marker, and can determine from which direction and how far the image was captured based on the image of the reference marker captured by camera 101.

[0070] Furthermore, the correction processing unit 102 obtains the modified corresponding part, i.e., position P2, in the image data Gn of the currently displayed decorative material that corresponds to the coordinate modification position of the steel skeleton T (S503). The correction processing unit 102 knows the modified corresponding parts and their positions in each image data Gn that correspond to the coordinate modification position of the steel skeleton T, and obtains the position information of the modified corresponding parts in the image data Gn that is to be synthesized near position P1.

[0071] The correction processing unit 102 calculates the difference between position P1 and position P2 as the estimation error (S504).

[0072] As the user moves, a deviation arises between the image data G and the rebar skeleton T. This is because, initially, the image data G and the rebar skeleton T are aligned using initial markers, but it is difficult to perfectly align the positions to the details, and the position and pose estimation error of SLAM accumulates with movement. Therefore, the deviation increases with user movement. The above processing describes the handling of this deviation.

[0073] The above Figure 9 It uses the reference marker, but it is also possible to use the reference marker without it. Figure 10 This is a flowchart illustrating additional processing steps when calculating the positional deviation between image data G and the reinforcing steel skeleton T.

[0074] When the depth sensor 105 receives coordinate input indicating a modified coordinate position (S601: Yes), it calculates the distance to the modified coordinate position on the rebar skeleton T located in the shooting direction (user's line of sight) of the camera 101, and calculates position P1 (S602). Furthermore, the correction processing unit 102 obtains the modified corresponding part in the currently displayed image data G, namely position P2, which corresponds to the modified coordinate position of the rebar skeleton T (S603). The correction processing unit 102 calculates the difference between position P1 and position P2 as an estimation error (S604).

[0075] Therefore, it is possible to calculate position P1 and position deviation based on position P1 without using a reference mark.

[0076] Next, the effects of one embodiment will be explained. In one embodiment, the image display device 100 displays predetermined image data G (including image data G1 to Gn) of a steel frame T, which is the object of a composite image, and consists of multiple parts, namely steel frame T1 to Tn, captured by a camera 101 and located in the user's field of vision. This image display device 100 includes: a storage unit 103 that stores design data of the steel frame T, which is the object of the composite image; a display unit 104 that generates and displays a composite image representing a predetermined decorative material based on a reference position (reference mark or coordinate modification position) in one part of the steel frame T (e.g., steel frame T1); and a correction processing unit 102 that corrects deviations between the image data G in the user's field of vision and the steel frame T caused by user U's movement.

[0077] The correction processing unit 102 uses the design data stored in the storage unit 103 and the user U's own position relative to the steel reinforcement cage T to obtain the coordinate modification position S, which is a reference position for correction processing, from the steel reinforcement cage T1 and other components.

[0078] The correction processing unit 102 modifies the position S according to the coordinate and performs position alignment of the image data G, thereby performing correction.

[0079] This process enables the identification of which steel reinforcement skeleton Tn is being used without attaching labels corresponding to each part of the steel reinforcement skeleton T (steel reinforcement skeletons T1 to Tn), and allows the synthesis of image data Gn representing the decorative material corresponding to that steel reinforcement skeleton Tn.

[0080] In the image display device 100, when the correction processing unit 102 obtains a coordinate modification position as a reference position for correction processing, it obtains a plurality of coordinate modification positions and selects a coordinate modification position from the plurality of coordinate modification positions that satisfies a predetermined condition (e.g., the most recent) for the user U's own position.

[0081] Therefore, it is possible to select more appropriate coordinates to modify the position.

[0082] Furthermore, in this embodiment, the predetermined image data G represents the decorative material (covering component) configured to cover the steel skeleton T, which is the composite object.

[0083] This allows for the alignment of the steel reinforcement frame T with the image data G, enabling the user to see the building, including the decorative materials.

[0084] Furthermore, in this embodiment, the reinforcing steel cage T is composed of multiple surfaces. For example, the cross-section is quadrilateral. The coordinate modification position is defined on these multiple surfaces. Then, when selecting a coordinate modification position S, the correction processing unit 102 selects a coordinate modification position on one of the multiple surfaces of the reinforcing steel cage T that has a predetermined angle relative to the orientation from the user U's own position toward each surface of the reinforcing steel cage T.

[0085] Therefore, it is possible to select more appropriate coordinates to modify the position, thereby enabling more accurate synthesis of image data.

[0086] Furthermore, in this image display device 100, the correction processing unit 102 selects the coordinate modification position closest to the user U as a coordinate modification position. Therefore, an appropriate coordinate modification position can be selected.

[0087] Furthermore, the image display device 100 also includes a depth sensor 105, which measures the distance to the reinforcing bar cage T. The correction processing unit 102 uses the distance measured by the depth sensor 105 to calculate the actual coordinate modification position. Then, the correction processing unit 102 calculates the deviation between the actual coordinate modification position and the position corresponding to the coordinate modification position S in the synthesized image data G before correction, and performs correction based on the deviation.

[0088] Furthermore, the image display device 100 includes an identification unit (not shown) for identifying marks attached to the composite object. The correction processing unit 102 calculates the actual coordinate modification position based on the position of the identified mark. Then, the correction processing unit 102 calculates the deviation between the actual coordinate modification position and the position corresponding to the coordinate modification position in the composite image data G before correction, and performs correction based on this deviation.

[0089] These deviations can be corrected.

[0090] The block diagrams used in the description of the above embodiments illustrate blocks based on function. These functional blocks (structural units) are implemented through any combination of at least one of hardware and software. Furthermore, there are no particular limitations on the implementation method of each functional block. That is, each functional block can be implemented using a single device that is physically or logically combined, or by directly or indirectly (e.g., using wired, wireless, etc.) connecting two or more physically or logically separate devices. Functional blocks can also be implemented by combining software with the aforementioned single or multiple devices.

[0091] The functions include, but are not limited to, judgment, determination, calculation, processing, derivation, investigation, search, confirmation, receiving, sending, output, access, resolution, selection, choosing, establishing, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assigning. For example, the functional block (structural unit) that performs the sending function is called the transmitting unit or transmitter. In short, as mentioned above, there are no particular limitations on the implementation method.

[0092] For example, the image display device 100 in one embodiment of this disclosure can also function as a computer performing the image processing method of this disclosure. Figure 11This is a diagram illustrating an example of the hardware structure of an image display device 100 according to an embodiment of the present disclosure. The image display device 100 described above can be configured as a computer device that physically includes a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, and a bus 1007, etc.

[0093] Additionally, in the following description, the term "device" can be replaced with "circuit," "device," "unit," etc. The hardware structure of the image display device 100 can be configured to include one or more of the devices shown in the figures, or it can be configured to not include any of the devices.

[0094] The functions in the image display device 100 are implemented by reading predetermined software (programs) into hardware such as the processor 1001 and memory 1002, so that the processor 1001 performs calculations and controls the communication of the communication device 1004, or controls at least one of reading out and writing data in the memory 1002 and the storage device 1003.

[0095] The processor 1001 controls the computer as a whole, for example, by enabling the operating system to operate. The processor 1001 may also be a central processing unit (CPU) that includes interfaces with peripheral devices, control devices, arithmetic units, registers, etc. For example, the aforementioned correction processing unit 102, calculation processing unit 106, and self-position and attitude estimation unit 107 can also be implemented by the processor 1001.

[0096] Furthermore, the processor 1001 reads programs (program code), software modules, data, etc., from at least one of the memory 1002 in the memory 1003 and the communication device 1004, and performs various processes accordingly. The program is used to cause the computer to perform at least a portion of the actions described in the above embodiments. For example, the correction processing unit 102 of the image display device 100 can be implemented by a control program stored in the memory 1002 and operating in the processor 1001, and other functional blocks can be implemented similarly. Although it has been described that the various processes described above are performed by one processor 1001, the various processes described above can also be performed simultaneously or sequentially by two or more processors 1001. The processor 1001 can also be installed using more than one chip. Additionally, the program can also be transmitted from a network via a telecommunications line.

[0097] Memory 1002 is a computer-readable recording medium, and may be composed of at least one of ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. Memory 1002 may also be referred to as a register, cache, main memory (main storage device), etc. Memory 1002 can store programs (program code), software modules, etc., that are executable for implementing the image processing method according to one embodiment of this disclosure.

[0098] The memory 1003 is a computer-readable recording medium, such as at least one of the following: CD-ROM (Compact Disc ROM), hard disk drive, floppy disk, magneto-optical disk (e.g., compact disc, digital multipurpose disk, Blu-ray disc, smart card, flash memory (e.g., card, stick, key drive), floppy disk, magnetic stripe, etc.). The memory 1003 may also be referred to as an auxiliary storage device. The aforementioned storage medium may also be, for example, a database, server, or other suitable medium that includes at least one of the memory 1002 and the memory 1003.

[0099] The communication device 1004 is hardware (transceiver) used for communication between computers via at least one of a wired network and a wireless network. It is also referred to as a network device, network controller, network interface card (NIC), communication module, etc. For example, to implement at least one of Frequency Division Duplex (FDD) and Time Division Duplex (TDD), the communication device 1004 may be configured to include a high-frequency switch, duplexer, filter, frequency synthesizer, etc.

[0100] Input device 1005 is an input device that accepts input from external sources (e.g., keyboard, mouse, microphone, switch, button, sensor, etc.). Output device 1006 is an output device that performs output to external sources (e.g., display, speaker, LED, etc.). Alternatively, input device 1005 and output device 1006 can also be integrated (e.g., a touch panel).

[0101] Furthermore, the processor 1001, memory 1002, and other devices are connected via a bus 1007 for communication. The bus 1007 can be configured using a single bus or different buses for each device.

[0102] Furthermore, the image display device 100 can be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or a FPGA (Field Programmable Gate Array), and can also implement some or all of the functional blocks using this hardware. For example, the processor 1001 can also be implemented using at least one of these hardware components.

[0103] The processing procedures, timing, and flow of the various forms / implementations described in this disclosure may be changed in order, provided there is no contradiction. For example, the elements of various steps are indicated using an illustrative order in the methods described in this disclosure, but are not limited to the specific order indicated.

[0104] Input or output information can be stored in a specific location (e.g., memory) or managed using a management table. Input or output information can be overwritten, updated, or appended. Output information can also be deleted. Input information can also be sent to other devices.

[0105] The determination can be made by the value represented by 1 bit (0 or 1), by a Boolean value (Boolean: true or false), or by comparing numerical values ​​(e.g., comparing with a predetermined value).

[0106] The various forms / implementations described in this disclosure can be used individually or in combination, and can be switched depending on the execution. Furthermore, the notification of predetermined information is not limited to explicit notification (e.g., a "Yes X" notification) but can also be implicit notification (e.g., not notifying the predetermined information).

[0107] The present disclosure has been described in detail above, but it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented as modifications and variations without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the present disclosure is for illustrative purposes only and is not intended to be limiting.

[0108] Software, whether referred to as software, firmware, middleware, microcode, hardware description language, or by other names, should be broadly interpreted as commands, command sets, code, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc.

[0109] Furthermore, software, commands, and information can be sent and received via a transmission medium. For example, when software is sent from a website, server, or other remote source using at least one of wired technologies (coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), etc.) and wireless technologies (infrared, microwave, etc.), at least one of these wired and wireless technologies is included within the definition of a transmission medium.

[0110] The information, signals, etc., described in this disclosure can also be represented using any of a variety of different technologies. For example, the data, commands, instructions, information, signals, bits, symbols, chips, etc., that may be involved in the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any combination of these.

[0111] Furthermore, the terms used in this disclosure and those necessary for understanding this disclosure may be replaced with terms that have the same or similar meanings. For example, at least one of the channel and symbol may also be a signal (signaling). Additionally, a signal may also be a message. Furthermore, a component carrier (CC) may also be referred to as a carrier frequency, cell, frequency carrier, etc.

[0112] The terms “system” and “network” as used in this disclosure are used interchangeably.

[0113] Furthermore, the information, parameters, etc., described in this disclosure may be represented using absolute values, relative values ​​to predetermined values, or other corresponding information. For example, wireless resources may also be indicated by indexes.

[0114] The names used for the above parameters are not restrictive in any way.

[0115] The terms "determining" and "determining" as used in this disclosure sometimes encompass a variety of actions. For example, "determining" and "determining" can include actions such as judging, calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or other data structure), and ascertaining, which are considered as actions of "determining" and "determining." Furthermore, "determining" and "determining" can include actions such as receiving (e.g., receiving information), transmitting (e.g., sending information), inputting, outputting, and accessing (e.g., accessing data in memory), which are considered as actions of "determining" and "determining." Additionally, "determining" and "determining" can include actions such as resolving, selecting, choosing, establishing, and comparing, which are considered as actions of "determining" and "determining." That is, "judgment" and "decision" can include matters that are considered to have been "judged" or "decided" to take any action. In addition, "judgment (decision)" can also be replaced by "assuming", "expecting", "considering", etc.

[0116] The terms “connected,” “coupled,” or any variations thereof are intended to indicate any direct or indirect connection or combination between two or more elements, including cases where there is one or more intermediate elements between the two elements that are “connected” or “coupled.” The combination or connection between elements can be physical, logical, or a combination of these. For example, “access” can be used instead of “connected.” In the context of this disclosure, for two elements, it can be considered that they are “connected” or “coupled” by using at least one of one or more wires, cables, and printed electrical connections, and as some non-limiting and non-inclusive examples, by using electromagnetic energy with wavelengths having wireless frequency domains, microwave regions, and light (including both visible and invisible regions).

[0117] As used in this disclosure, the word "based on" does not mean "based on only" unless otherwise expressly stated. In other words, the word "based on" means both "based on only" and "based on at least".

[0118] Alternatively, the "unit" in the structure of the above devices can be replaced with "section", "circuit", "equipment", etc.

[0119] When the terms "include," "including," and their variations are used in this disclosure, these terms, like the term "comprising," imply inclusion. Furthermore, the term "or" as used in this disclosure implies non-exclusivity.

[0120] In this disclosure, for example, in cases where articles are added through translation, such as in English (e.g., a, an, and the), this disclosure may also include cases where the noun following these articles is in a plural form.

[0121] In this disclosure, the phrase "A and B are different" can also mean "A and B are not the same." Furthermore, this phrase can also mean "A and B are each different from C." Terms such as "separate" and "combined" can also be interpreted in the same way as "different."

[0122] Label Explanation

[0123] 100: Image display device; 101: Camera; 102: Correction processing unit; 103: Storage unit; 104: Display unit; 105: Depth sensor; 106: Calculation processing unit; 107: Self-position and pose estimation unit.

Claims

1. An image display device that synthesizes and displays a predetermined image of a composite object located in real space, wherein, The image display device has: The storage unit stores design data representing the positional relationships of the various parts of the composite object; The display unit generates and displays a composite image obtained by compositing a predetermined image using a portion of the composite object as a reference position. as well as The correction unit corrects for deviations between the predetermined image and the composite object caused by user movement. The correction unit uses the design data and the user's position relative to the composite object to obtain the coordinate modification position of the reference position in the one part of the composite object for correction processing. The correction unit modifies its position based on these coordinates to correct the predetermined composite position of the image. When the correction unit obtains the coordinate modification position of the reference position used for correction processing, it obtains multiple coordinate modification positions. The correction unit selects from the plurality of coordinate modification positions a coordinate modification position that meets predetermined conditions for the user's own position.

2. The image display device according to claim 1, wherein, The predetermined image represents a cover element configured to cover the composite object.

3. The image display device according to claim 1, wherein, When selecting a coordinate modification position, the correction unit selects a coordinate modification position defined among multiple composite objects that has a predetermined angle relative to the orientation of each composite object from the user's own position.

4. The image display device according to claim 3, wherein, The calibration unit selects the coordinate modification position closest to the user as a coordinate modification position.

5. The image display device according to any one of claims 1 to 4, wherein, The image display device also includes a sensor unit that measures the distance to the composite object. The calibration unit uses the distance measured by the sensor unit to calculate the actual coordinate correction position. The correction unit calculates the deviation between the actual coordinate modification position and the portion of the image synthesized before correction corresponding to the coordinate modification position. The correction unit performs correction based on the deviation.

6. The image display device according to any one of claims 1 to 4, wherein, The image display device also has a recognition unit that identifies marks attached to the composite object. The correction unit calculates the actual coordinate correction position based on the identified mark's location. The correction unit calculates the deviation between the actual coordinate modification position and the portion of the image synthesized before correction corresponding to the coordinate modification position. The correction unit performs correction based on the deviation.

Images