Image display device
The image display device addresses misalignment issues in MR by using design data and self-position estimation to correct deviations, enabling efficient alignment of image data with structures.
Patent Information
- Application Number
- JP2021571157
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-01-14
- Filing Date
- 2021-01-06
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-01-06
AI Technical Summary
Existing MR technologies face challenges in aligning image data with structures during construction, leading to misalignment when users move, which is time-consuming and difficult to correct.
An image display device that synthesizes and displays images in real space, using design data to store positional relationships, correct deviations through coordinate correction positions, and align images based on self-position and orientation estimation.
Facilitates easy correction of misalignment in mixed reality spaces, allowing accurate alignment of image data with structures without the need for extensive marker setup.
Smart Images

Figure 0007592031000001 
Figure 0007592031000002 
Figure 0007592031000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an image display device that utilizes mixed reality (MR). [Background technology]
[0002] It is generally known that an image is synthesized with an object in a real space by using MR technology. For example, Patent Document 1 describes a superimposed depiction of a real space image and a virtual space image. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2008-293209 A Summary of the Invention [Problem to be solved by the invention]
[0004] In recent years, there has been interest in using MR technology to combine image data of decorative materials and other elements with structures such as framework or beams during the construction stage of a building, making it easier to grasp the finished product.
[0005] However, it is often difficult to align the image data with the structure. In general, it is considered to align the image data by attaching a marker to the structure. In this case, when a user wearing goggles that can view an MR space, such as MR goggles, moves away from the structure to which the marker is attached, a deviation occurs in the alignment between the structure and the image data. For this reason, it is possible to place markers in accordance with the positions throughout the structure, but it is very time-consuming to set and place markers in accordance with the positions.
[0006] Therefore, in order to solve the above-mentioned problems, an object of the present invention is to provide an image display device that can easily correct misalignment that occurs when a user moves in a complex space. [Means for solving the problem]
[0007] The image display device of the present invention is an image display device that synthesizes a predetermined image with respect to a synthetic object in real space and displays it, and is equipped with a memory unit that stores design data indicating the positional relationship of each part of the synthetic object, a display unit that generates and displays a synthetic image by synthesizing the predetermined image using one part of the synthetic object as a reference position, and a correction unit that corrects the deviation between the predetermined image and the synthetic object caused by the movement of the user, wherein the correction unit uses the design data and the user's own position relative to the synthetic object to obtain a coordinate correction position that serves as a reference position for correction processing of the one part of the synthetic object, and corrects the synthesis position of the predetermined image based on the coordinate correction position. Effect of the Invention
[0008] According to the present invention, misalignment in mixed reality space can be easily corrected. [Brief description of the drawings]
[0009] [Figure 1] 1 shows a diagram of a steel frame T, which is a structure in this embodiment, and a diagram in which image data G of a column is superimposed on the steel frame T. [Diagram 2] FIG. 11 is an explanatory diagram of a coordinate correction position. [Diagram 3] FIG. 13 is an explanatory diagram showing a synthesis process using a coordinate correction position. [Figure 4] FIG. 1 is a block diagram showing a functional configuration of an image display device 100. [Diagram 5] 4 is a flowchart showing the operation of the image display device 100. [Figure 6] 13 is a flowchart showing a detailed operation of processing S101. [Figure 7]11 is a flowchart showing detailed processing of selection and correction of self-position and orientation estimation results in a correction processing unit 102. [Figure 8] 13 is a flowchart showing a process of selecting a coordinate correction position to be used from coordinate correction position candidates. [Figure 9] 13 is a flowchart showing a calculation process of a deviation of a coordinate correction position in a correction processing unit 102. [Figure 10] 13 is a flowchart showing another process for calculating the positional deviation between image data G and a steel frame T. [Figure 11] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image display device 100 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described with reference to the accompanying drawings. Whenever possible, the same reference numerals are used to designate the same parts, and redundant description will be omitted.
[0011] FIG. 1 shows a diagram of a steel frame T, which is a structure in this embodiment, and a diagram in which the steel frame T has been combined with image data G of a decorative material indicating that it is a pillar. FIG. 1 shows a diagram that can be seen by a user wearing MR goggles, which is an image display device in this embodiment. FIG. 1(a) is a diagram showing the steel frame T, which is a structure existing in real space, before the image data G of the decorative material has been combined. In FIG. 1, the steel frame T is made up of steel frames T to Tn. Although FIG. 1 shows up to steel frame T9, it goes without saying that up to steel frame Tn (n is a natural number) may also exist. The same applies to image data Gn (n is a natural number, for example, G1 to G9).
[0012] In this embodiment, the pillars are steel frames, but the pillars are not limited to steel frames, and may be concrete pillars or may be covered with decorative materials. The image to be synthesized and displayed is not limited to decorative materials, and may be piping, wiring, furniture, or the like.
[0013] FIG. 1(b) is a diagram showing an actual steel frame T covered with decorative materials for decoration, and is a diagram showing image data G representing the decorative materials superimposed on the steel frame T in mixed reality space. Design data indicating the positions and sizes of the steel frames T1-Tn, and image data G1-Gn of the decorative materials are stored in advance. The image data G is a three-dimensional image composed of the image data G1-Gn. The image data G1-Gn of the multiple decorative materials are associated in advance with the positions of the steel frames T1-Tn.
[0014] A correction corresponding portion corresponding to a coordinate correction position described later is set in each image data Gn. For example, when the correction corresponding portion of the image data G1 of a decorative material is aligned with the coordinate correction position set in the steel frame T1 and synthesized, the image data G2 of another decorative material is also aligned with the corresponding steel frame T2. The image data Gn of other decorative materials are similarly aligned with the corresponding steel frame Tn. However, there may be an error in the alignment between the correction corresponding portion of the image data G2, etc. and the coordinate correction position set in the steel frame T2, etc.
[0015] When the steel frame T and the image data G of the decorative materials are aligned for the first time, for example, an initial marker for the image data G1 is attached to a predetermined location on the steel frame T1 to align them. Since the image data G2, etc. constitute the image data G with a fixed positional relationship with each other, the other steel frames T2, etc. are also aligned automatically. Since the image data G is a three-dimensional image, the image data G is rotated according to the direction from which the user views the steel frame T. In this way, by aligning the positions based on the initial marker, it becomes possible to align all the steel frames T1 to Tn with the image data G1 to Gn.
[0016] The alignment will be described in more detail. FIG. 2 explains the coordinate correction position. FIG. 1 simply describes alignment. As shown in FIG. 2, the steel frame T1 has faces in each of the four directions. The image data G1 of the decorative material is aligned based on one of the faces. The coordinate correction position indicates the reference position for aligning the position of the steel frame T with the image data G of the decorative material.
[0017] 2(a) is a schematic diagram of a steel frame T viewed from above. In this embodiment, the steel frame T is a rectangular column with the intersection point C of the grid line at its center, and its cross section is a square. Here, attention is focused on the steel frame T1. Coordinate correction positions S11 to S14 are set on each face of the steel frame T1.
[0018] When the user moves while looking at the steel frame T1, a misalignment occurs between the positions of the other steel frames T2-T4 (decorative materials K2-K4) and the image data G2-G4 of the decorative materials. Therefore, when a misalignment occurs, it is necessary to correct the misalignment. In other words, when the user moves while looking at the steel frame T1, the initially set reference marker disappears from view, making it impossible to track the position and angle using this reference marker, and tracking is performed using self-position and orientation estimation by SLAM. This self-position and orientation estimation generates estimation errors due to movement.
[0019] In this embodiment, the steel frame T that serves as the reference for correction is identified by identifying the steel frame T that is closest to the user and its coordinate correction position S.
[0020] 2(b) is a diagram showing the steel frame T2 and coordinate correction positions S21 to S24 that are closest to a user U when the user U moves while wearing the image display device 100, which is an MR goggle. In this embodiment, the image display device 100 can grasp the relative position of the user U with respect to the steel frame T by using a known self-position estimation technique.
[0021] When the user U looks at the steel frame T2, the image display device 100 determines which of the coordinate correction positions S21 to S24 of the steel frame T2 closest to the user U will be used as the correction reference. As shown in Fig. 3(a), the image display device 100 selects the coordinate correction position S where the reverse vector of the line of sight of the user U (corresponding here to the shooting direction of the camera in the image display device) and the normal vector of the surface of the steel frame T are within a predetermined angle.
[0022] Furthermore, based on the distance to the steel beam T measured by the depth sensor, the actual coordinate correction position defined on the surface of the steel beam T is calculated (Figure 3(b)). The depth sensor does not necessarily measure from the front of the steel beam T. Therefore, the coordinate correction position is corrected to a specified position on the surface of the steel beam T (e.g., the center).
[0023] Then, a process is performed to shift the correction corresponding portion Sh2 of the image data G based on the coordinate correction position (FIG. 3(c)). The image data G2 is associated with correction corresponding portion information (position S22 in FIG. 3) associated with each coordinate correction position S and with relative position information in the image data G. That is, each surface of each image data G is associated with a reference position for alignment with the coordinate correction position set for each surface of each steel frame T. By aligning this correction corresponding portion Sh2 with the coordinate correction position S12, it is possible to align the image data G with the steel frame T.
[0024] By carrying out such processing, the user U can visually confirm the state in which the steel frame T is covered with the decorative material image data G while moving. Note that, although the cross section of the steel frame T is rectangular in Fig. 2 and Fig. 3, this is not limited to this. It may be circular or another polygonal shape, and it is sufficient that the coordinate correction position of the steel frame T and the corresponding correction part of the decorative material image data G can be associated with each other.
[0025] Next, the functional configuration of the image display device 100 of this embodiment will be described. FIG. 4 is a block diagram showing the functional configuration of the image display device 100. As shown in the figure, the image display device 100 is configured to include a camera 101, a correction processing unit 102 (correction unit), a storage unit 103, a display unit 104, a depth sensor 105 (sensor unit), a calculation processing unit 106, and a self-position and orientation estimation unit 107. This image display device 100 is a device called MR goggles, and is a head-mounted display type device. By wearing this image display device 100 on the head, a user can view the image captured by the camera 101 through the display unit 104. In a video see-through type head-mounted display, by wearing this image display device 100 on the head, a user can view the image captured by the camera 101 through the display unit 104. In addition, this embodiment is not limited to these forms, and can be similarly realized with an optical see-through type device (for example, viewing a steel frame through glasses).
[0026] The camera 101 is a part that captures the real space seen by the user.
[0027] The correction processing unit 102 is a part that synthesizes, in a mixed reality space, image data of virtual space members with respect to the video captured by the camera 101. Here, the correction processing unit 102 performs processing for synthesizing image data G representing decorative materials with the steel frame T and performing correction processing in the event of positional deviation.
[0028] The storage unit 103 is a part that stores design data of structures such as steel beams T and image data G that indicates decorative materials for the structures. The design data is information that indicates the size of each steel beam T, the spacing between them, etc. Furthermore, the image data G has information that indicates the correspondence with the steel beams T (particularly the coordinate correction position, etc.) in addition to the image of the decorative materials.
[0029] The display unit 104 is a part that displays the image captured by the camera 101. When the image data G is synthesized with the steel frame T by the correction processing unit 102, the display unit 104 displays the synthesized image.
[0030] The depth sensor 105 is a sensor that measures the distance between the user U and the steel frame T.
[0031] The calculation processing unit 106 is a part that calculates the initial position of the relative positional relationship of the image display device 100 with respect to the structure (steel frame T) and the positional deviation between the steel frame T and the image data G that occurs each time the user moves, based on the design data stored in the memory unit 103.
[0032] The self-position and orientation estimation unit 107 is a part that estimates a position in real space captured and displayed by the camera 101 and a change in the orientation of the image display device 100 based on the image captured by the camera 101. The self-position and orientation estimation unit 107 can estimate a current positional relationship (current position) relative to the steel frame T by grasping the direction in which the steel frame T is viewed from the captured image and its relative distance based on the initial position calculated by the calculation processing unit 106, the image of the initial marker at that time, and the steel frame T currently being captured. This self-position and orientation estimation unit 107 is a known technology and can be realized by, for example, SLAM (Simultaneous Localization and Mapping). Note that the self-position and orientation estimation function is not limited to the above, and the self-position may be estimated using a GPS or other sensors.
[0033] The operation of the image display device 100 configured as above will be described below with reference to the flowchart of FIG.
[0034] The calculation processing unit 106 calculates a coordinate correction position S for each steel frame T based on the design data stored in the storage unit 103, and stores the calculated coordinate correction position S in the storage unit 103 (S101).
[0035] Next, the calculation processing unit 106 calculates the initial position of the self-position and orientation estimation unit 107, and grasps the positional relationship of the steel frame T (S102).
[0036] When the user moves (S103), an error occurs in the estimation of the user's position and orientation (S104).
[0037] Then, the correction processing unit 102 selects a reference coordinate correction position from the coordinate correction position candidates based on the self-position estimated by the self-position and orientation estimation unit 107. Then, the correction processing unit 102 corrects the position of the image data G of the cosmetic material based on the self-position estimation result (self-position) and the selected coordinate correction position (S105). These steps S103 to S106 are repeated until the program ends (S106).
[0038] 6 is a flowchart showing the detailed operation of process S101. The calculation processing unit 106 reads design data of the building (steel frame T) by referring to the storage unit 103 (S201). The calculation processing unit 106 calculates the coordinate correction position from the floor height information and center line information (S202). Here, the floor height information indicates the number of floors of the building and its height. The center line information is information indicating the center line of the steel frame, and indicates coordinates within the building. The coordinate correction position is a reference position for synthesizing image data G of the decorative material, and indicates three-dimensional coordinates obtained from the coordinates of the coordinate correction position defined at the intersection C of the center line and its height.
[0039] The calculation processing unit 106 determines the position obtained by shifting the coordinate correction position by Y meters in the height direction as the coordinate correction position (S203). This process is for aligning the position with the line of sight of a person.
[0040] The calculation processing unit 106 judges whether or not the final coordinate correction position is inside a steel frame T such as a wall or a pillar (S204). For example, the coordinates (horizontal coordinates) of the intersection of the grid line are judged to be inside the steel frame T. The same applies when the steel frame T is embedded in a wall. This information is judged based on the size and arrangement of the steel frame T included in the design data.
[0041] When the calculation processing unit 106 determines that the coordinate correction position is located within the steel frame T, it divides the coordinate correction position in four directions centered on the intersection of the grid line, and calculates and sets the intersection that intersects with the surface of the steel frame T on the design data as the coordinate correction position (S205).
[0042] The calculation processing unit 106 stores the coordinate correction positions calculated based on these processes in the storage unit 103 (S206). The storage unit 103 stores, for each steel frame T (steel frames T1 to Tn), the identification information of the decorative material associated therewith, the arrangement position of the decorative material (relative position information), the correction corresponding portion in the image data G of the decorative material corresponding to the coordinate correction position set for each steel frame T, and the image data G of the decorative material (image data G1 to Gn) in association with each other. In other words, the storage unit 103 stores various information so that each coordinate correction position of the steel frame T is associated with which image data G (G1 to Gn) is to be synthesized.
[0043] Next, the detailed process of step S105 will be described. Fig. 7 is a flowchart showing the detailed process of the selection process and correction process of the self-position and orientation estimation result in the correction processing unit 102.
[0044] The correction processing unit 102 selects a reference coordinate correction position from the coordinate correction position candidates stored in the storage unit 103 (S301). If there is a candidate, the correction processing unit 102 superimposes message information indicating that the selected coordinate correction position is a coordinate correction position, and the display unit 104 displays that fact (S303).
[0045] The correction processing unit 102 calculates the deviation of the coordinate correction position from the coordinates in the real space (S304). That is, the correction processing unit 102 calculates the position deviation between the specified coordinate correction position in the steel frame T and the image data G of the decorative material to be synthesized. For example, in FIG. 3(b), the depth sensor 105 measures the distance to the surface of the steel frame T (coordinate correction position) at a position where the height of the intersection point C2 of the grid center is shifted by Y. The correction processing unit 102 calculates the actual position of the coordinate correction position based on the distance. On the other hand, the correction processing unit 102 obtains the position of the image data G2 of the decorative material (the position to be aligned with the coordinate correction position, here the correction corresponding position), and calculates the difference as the position deviation (error). The coordinate correction position is a position shifted by Y in the height direction and is at the center position of the thickness of the steel frame in the horizontal direction, so the position may be calculated, or may be specified by a user's instruction using the user input function of this device.
[0046] When the correction processing unit 102 calculates a positional deviation (eg, the deviation is equal to or greater than a predetermined value) (S305: YES), the correction processing unit 102 corrects the overlay position of the image data G in accordance with the amount of deviation (S306).
[0047] Next, a detailed process of step S301 in Fig. 7 will be described. Fig. 8 is a flowchart showing a process of selecting a coordinate correction position to be used from the coordinate correction position candidates.
[0048] The correction processing unit 102 reads the corrected coordinate position on the steel frame T from the memory unit 103 (S401), and also obtains the result of self-position and orientation estimation by the self-position and orientation estimation unit 107 (information on the self's position relative to the steel frame T) (S402).
[0049] The correction processing unit 102 extracts multiple coordinate correction positions S in the vicinity (predetermined range) of the self-position based on the self-position and orientation estimation result (S403). The correction processing unit 102 also calculates the direction N1 in which the user faces with respect to the steel frame T from the self-position and orientation estimation result (S404). That is, the self-position and orientation estimation unit 107 calculates the angle between the direction vector of the normal N2 to the surface of each steel frame T and the reverse direction vector of the direction N1.
[0050] The correction processing unit 102 excludes surfaces having coordinate correction positions where this angle is equal to or greater than φ degrees. In other words, when the user views the surface of the steel frame T from an oblique direction, the correction processing unit 102 excludes coordinate correction positions on the surface of the steel frame T (S405).
[0051] The correction processing unit 102 judges whether the distance between the nearest coordinate correction position S and the second nearest coordinate correction position S is less than a predetermined distance (S406). If it is less than the predetermined distance, it is judged that there is no candidate (S408), and if it is the predetermined distance or more, the nearest coordinate correction position S is selected (S407).
[0052] Through this process, the correction processing unit 102 selects one coordinate correction position S.
[0053] Next, a detailed process of the process S304 in Fig. 7 will be described. Fig. 9 is a flowchart showing a calculation process of a deviation of a coordinate correction position in the correction processing unit 102. Fig. 9 shows a process using a reference marker for each portion of the steel frame T.
[0054] When a reference marker appears in an image captured by the camera 101, the correction processing unit 102 calculates the position P1 of the reference marker based on the self-position estimated by the self-position and orientation estimation unit 107 (S501, S502). The correction processing unit 102 has information such as the shape and size of the reference marker in advance, and can determine from what direction and at what distance the image of the reference marker was captured based on the image of the reference marker captured by the camera 101.
[0055] The correction processing unit 102 also acquires position P2, which is a correction-corresponding portion in the image data Gn of the decorative material currently being displayed that corresponds to the coordinate correction position of the steel frame T (S503). The correction processing unit 102 is aware of the correction-corresponding portion and its position that corresponds to the coordinate correction position of the steel frame T in each image data Gn, and acquires position information of the correction-corresponding portion that corresponds to the coordinate correction position in the image data Gn that is to be synthesized near position P1.
[0056] The correction processing unit 102 calculates the difference between the position P1 and the position P2 as an estimated error (S504).
[0057] As the user moves, a deviation occurs between the image data G and the steel frame T. This is because, although the image data G and the steel frame T are initially aligned using initial markers, it is difficult to perfectly align the positions in detail, and because SLAM position and orientation estimation errors accumulate as the user moves. Therefore, the deviation may become larger as the user moves. The above process is for calculating the deviation.
[0058] 9 shows a process using a reference marker, but this need not be used. FIG 10 is a flow chart showing another process for calculating the positional deviation between the image data G and the steel frame T.
[0059] When the depth sensor 105 receives coordinate input of the coordinate correction position (S601: YES), it calculates the distance to the coordinate correction position on the steel frame T in the shooting direction of the camera 101 (the user's line of sight direction) and calculates a position P1 (S602). In addition, the correction processing unit 102 acquires a position P2, which is a correction corresponding to the coordinate correction position of the steel frame T in the currently displayed image data G (S603). The correction processing unit 102 calculates the difference between the positions P1 and P2 as an estimated error (S604).
[0060] This makes it possible to calculate the position P1 without using a reference marker, and to calculate the positional deviation based on the position P1.
[0061] Next, the effects of one aspect of this embodiment will be described. The image display device 100 in one aspect of this embodiment is a device that synthesizes and displays predetermined image data G (including image data G1-Gn) for a steel frame T, which is a synthesis object having a plurality of partial members of steel frames T1-Tn in the user's field of view photographed by a camera 101. This image display device 100 includes a storage unit 103 that stores design data of the steel frame T, which is the synthesis object, a display unit 104 that generates and displays a synthetic image by synthesizing image data G showing a predetermined decorative material based on a reference position (reference marker or coordinate correction position) in one part of the steel frame T (for example, the steel frame T1, etc.), and a correction processing unit 102 that corrects a deviation between the image data G and the steel frame T in the user's field of view caused by the user U moving.
[0062] This correction processing unit 102 uses the design data stored in the memory unit 103 and the user U's own position relative to the steel frame T to obtain a coordinate correction position S, which serves as a reference position for the correction processing, from a partial component such as the steel frame T1.
[0063] The correction processing unit 102 performs correction by aligning the image data G based on the one coordinate correction position S.
[0064] This process makes it possible to identify which steel frame Tn it is without attaching markers corresponding to each part of the steel frame T (steel frames T1 to Tn), and to synthesize image data Gn showing the decorative material corresponding to that steel frame Tn.
[0065] In this image display device 100, when the correction processing unit 102 acquires a coordinate correction position that serves as a reference position for the correction processing, it acquires multiple coordinate correction positions and selects from the multiple coordinate correction positions one coordinate correction position that satisfies a predetermined condition (e.g., is closest) to the user U's own position.
[0066] This makes it possible to select a more appropriate coordinate correction position.
[0067] In this embodiment, the predetermined image data G indicates a decorative material (covering member) configured to cover the steel frame T, which is the object to be synthesized.
[0068] This makes it possible to align the steel frame T with the image data G, allowing the user to visually recognize it as a building including decorative materials.
[0069] In this embodiment, the steel frame T is composed of multiple surfaces. For example, its cross section is a rectangle. The coordinate correction positions are defined on the multiple surfaces. When selecting one coordinate correction position S, the correction processing unit 102 selects, from among the multiple surfaces of the steel frame T, a coordinate correction position defined on a surface that has a predetermined angle with respect to the direction from the user U's own position to each surface of the steel frame T.
[0070] This allows a more appropriate coordinate correction position to be selected, and therefore allows image data to be combined accurately.
[0071] Furthermore, in this image display device 100, the correction processing unit 102 selects, as one coordinate correction position, the coordinate correction position that is closest to the user U. This makes it possible to select an appropriate coordinate correction position.
[0072] Moreover, this image display device 100 further includes a depth sensor 105 that measures the distance to the steel frame T. The correction processing unit 102 calculates an actual coordinate correction position using the distance measured by the depth sensor 105. Then, the correction processing unit 102 calculates the deviation between the actual coordinate correction position and a position corresponding to the coordinate correction position S in the image data G synthesized before correction, and performs correction based on the deviation.
[0073] The image display device 100 further includes a recognition unit (not shown) that recognizes a marker attached to the object to be synthesized. The correction processing unit 102 calculates an actual coordinate correction position based on the position of the recognized marker. Then, the correction processing unit 102 calculates the deviation between the actual coordinate correction position and a position corresponding to the coordinate correction position in the synthesized image data G before correction, and performs correction based on this deviation.
[0074] It becomes possible to perform correction based on these deviations.
[0075] The block diagrams used in the description of the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by wire, wirelessly, etc.). The functional blocks may be realized by combining the one device or the multiple devices with software.
[0076] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocation, mapping, assignment, etc. For example, a functional block (component) that performs the function of transmission is called a transmitting unit or a transmitter. In either case, as described above, there is no particular limitation on the method of realization.
[0077] For example, the image display device 100 according to an embodiment of the present disclosure may function as a computer that performs processing of the image processing method of the present disclosure. Fig. 11 is a diagram showing an example of a hardware configuration of the image display device 100 according to an embodiment of the present disclosure. The image display device 100 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.
[0078] In the following description, the term "apparatus" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the image display device 100 may be configured to include one or more of the devices shown in the drawings, or may be configured to exclude some of the devices.
[0079] Each function in the image display device 100 is realized by loading a specific software (program) onto hardware such as a processor 1001 and a memory 1002, causing the processor 1001 to perform calculations, control communication via a communication device 1004, and control at least one of the reading and writing of data in the memory 1002 and the storage 1003.
[0080] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured with a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the above-mentioned correction processing unit 102, calculation processing unit 106, self-position and orientation estimation unit 107, etc. may be realized by the processor 1001.
[0081] Moreover, the processor 1001 reads out a program (program code), a software module, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to the read out programs. As the program, a program that causes a computer to execute at least a part of the operations described in the above-mentioned embodiment is used. For example, the correction processing unit 102 of the image display device 100 may be realized by a control program stored in the memory 1002 and operated in the processor 1001, and other functional blocks may be similarly realized. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may be transmitted from a network via a telecommunication line.
[0082] The memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store a program (program code), software modules, etc. that are executable to implement an image processing method according to an embodiment of the present disclosure.
[0083] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. Storage 1003 may be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of memory 1002 and storage 1003.
[0084] The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 1004 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc., in order to realize, for example, at least one of Frequency Division Duplex (FDD) and Time Division Duplex (TDD).
[0085] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).
[0086] In addition, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.
[0087] Furthermore, the image display device 100 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.
[0088] The order of the steps, sequences, flow charts, etc. of each aspect / embodiment described in this disclosure may be changed unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.
[0089] The input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be transmitted to another device.
[0090] The determination may be based on a value represented by a single bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).
[0091] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).
[0092] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.
[0093] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.
[0094] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired and / or wireless technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.
[0095] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.
[0096] In addition, the terms described in this disclosure and the terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of a channel and a symbol may be a signal (signaling). Also, a signal may be a message. Also, a component carrier (CC) may be called a carrier frequency, a cell, a frequency carrier, etc.
[0097] As used in this disclosure, the terms "system" and "network" are used interchangeably.
[0098] In addition, the information, parameters, etc. described in the present disclosure may be represented using absolute values, may be represented using relative values from a predetermined value, or may be represented using other corresponding information. For example, a radio resource may be indicated by an index.
[0099] The names used for the parameters above are not limiting in any way.
[0100] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment" can be interpreted as "assuming," "expecting," "considering," etc.
[0101] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.
[0102] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."
[0103] The "means" in the configuration of each of the above devices may be replaced with "part," "circuit," "device," etc.
[0104] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Further, when used in this disclosure, the term "or" is not intended to be an exclusive or.
[0105] In this disclosure, where articles have been added due to translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.
[0106] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different." [Explanation of symbols]
[0107] 100: image display device, 101: camera, 102: correction processing unit, 103: storage unit, 104: display unit, 105: depth sensor, 106: calculation processing unit, 107: self-position and orientation estimation unit.
Claims
1. An image display device that displays a predetermined image by synthesizing it with a synthesis target object in real space, a storage unit that stores design data indicating a positional relationship of each part in the synthesis object; a display unit that generates and displays a composite image by combining a predetermined image with a portion of the composite object as a reference position; a correction unit that corrects a misalignment between the predetermined image and the synthesis object caused by a user moving; Equipped with The correction unit is acquiring a coordinate correction position serving as a reference position for a correction process of the one portion of the composite object using the design data and the user's own position with respect to the composite object; Correcting the synthesis position of the predetermined image based on the coordinate correction position; When acquiring a coordinate correction position that is a reference position for the correction process, a plurality of coordinate correction positions are acquired; selecting one coordinate correction position that satisfies a predetermined condition with respect to the user's own position from the plurality of coordinate correction positions; Image display device.
2. the predetermined image showing a covering member configured to cover the composite object; The image display device according to claim 1 .
3. The correction unit is When selecting the one coordinate correction position, the one coordinate correction position is selected from the plurality of coordinate correction positions defined for a synthesis object having a predetermined angle with respect to an orientation of each of the plurality of synthesis objects from the user's own position, among the plurality of synthesis objects. The image display device according to claim 1 .
4. The correction unit is selecting a coordinate correction position that is closest to the user from among the plurality of coordinate correction positions as the one coordinate correction position; The image display device according to claim 3 .
5. Further comprising a sensor unit for measuring a distance to the synthesis object, The correction unit is Calculating an actual coordinate correction position using the distance measured by the sensor unit; Calculating a deviation between the actual coordinate correction position and a portion of the image synthesized before correction that corresponds to the coordinate correction position; performing a correction based on the deviation; The image display device according to any one of claims 1 to 4.
6. A recognition unit that recognizes a marker attached to the synthesis object, The correction unit is Calculating an actual coordinate correction position based on the recognized marker position; Calculating a deviation between the actual coordinate correction position and a portion of the image synthesized before correction that corresponds to the coordinate correction position; performing a correction based on the deviation; The image display device according to any one of claims 1 to 4.
Citation Information
Patent Citations
Compound reality presentation device, its control method and computer program
JP2008293209A
Image forming apparatus, image forming system, and image forming program
JP2016009432A
Video display system
JP2019066293A
Image processing apparatus and method, and program
US20160320863A1
