Image processing system and image processing method

Abstract

To provide an image processing device that can preferably share a virtual model among a plurality of devices.SOLUTION: According to one embodiment, an image processing device comprises: a positioning unit which performs positioning of a coordinate system of a three-dimensional model space in which a virtual object is placed and a coordinate system of a real space; an image processing unit which generates an image of the virtual object observed from a user's viewpoint position based on the positioning; and a superimposed display unit which superimposes and displays a real space image and the virtual object. The positioning unit performs the positioning based on virtual object data and relative position information between a marker and the virtual object. Furthermore, the positioning unit obtains the virtual object data and the relative position information common to a first device corresponding to the image processing device and a second device corresponding to an image processing device different from the first device, and performs the positioning based on the obtained data and information.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] Embodiments of the present invention relate to an image processing apparatus, an image processing method, and an image processing system.

Background Art

[0002] Techniques for expressing a fusion of the real space and the virtual space using stereoscopic vision technology (Mixed Reality) are known. For example, a method of generating a stereoscopic image using a left-eye image and a right-eye image captured by a stereo camera and a virtual model (virtual object) has been proposed. In addition, a method of superimposing and displaying a virtual model on a real-space image while taking into account the viewpoint position of the user has been proposed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] With the progress of the information society, in the manufacturing site and the like, a virtual model of a product is created by 3D CAD (Computer Aided Design), and the shape and design of the product are confirmed and reviewed by a PC (Personal Computer) or a goggle-type display device at the time of design. Also, in the construction work of a plant and the like, the state of plant equipment is confirmed and reviewed using a virtual model on a PC at the time of design.

[0005] In the past, construction supervisors and contractors would make construction plans while mentally considering the design drawings and the on-site conditions during construction. However, nowadays, goggle-type holographic display devices can be used to display a virtual model of the completed state superimposed on a real-space image.

[0006] However, existing goggle-type holographic display devices do not have a means for multiple display devices to share and check the same virtual model. Therefore, in order to share and check the same virtual model among multiple display devices, each display device must hold virtual model data, and the multiple display devices must place the virtual model at the same position on the site space image to check or review it.

[0007] According to this technique, the shape of a virtual model seen by a user of one display device and the shape of a virtual model seen by a user of the other display device are the same model shape. However, the data used for display on the former display device and the data used for display on the latter display device are independent data. Therefore, even if one user changes the display content of a virtual model, this change cannot be reflected in the display content of the virtual model of the other user. For example, even if the former user rotates the virtual model, the latter user cannot see the rotation.

[0008] Therefore, the embodiments of the present invention provide an image processing device, an image processing method, and an image processing system that enable a virtual model to be suitably shared among a plurality of devices. [Means for solving the problem]

[0009] According to one embodiment, the image processing device includes an alignment unit that aligns a coordinate system of a three-dimensional model space in which a virtual object generated based on a design member designed by computer-aided design is placed, with a coordinate system of a real space. The image processing device further includes an image processing unit that generates an image of the virtual object observed from a user's viewpoint position, corresponding to a real space image observed from the user's viewpoint position, based on the alignment. The image processing device further includes a superimposition display unit that superimposes and displays the real space image and the virtual object. Furthermore, the alignment unit performs the alignment based on virtual object data, which is data of the virtual object, and relative position information representing the relative position between a marker provided in the real space and the virtual object. Furthermore, the alignment unit acquires the virtual object data and the relative position information common to a first device corresponding to the image processing device and a second device corresponding to an image processing device different from the first device, and performs the alignment based on the acquired virtual object data and the relative position information. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of an image processing device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram for explaining a model space in the first embodiment. [Diagram 3] FIG. 1 is a schematic diagram showing a configuration of an image processing system according to a first embodiment. [Figure 4] FIG. 11 is a schematic diagram showing the configuration of an image processing system according to a second embodiment. [Diagram 5] FIG. 11 is a block diagram for explaining the operation of an image processing system according to a second embodiment. [Figure 6] FIG. 13 is a schematic diagram showing the configuration of an image processing system according to a third embodiment. [Figure 7] FIG. 13 is a schematic diagram showing the configuration of an image processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a block diagram for explaining the operation of an image processing system according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In Fig. 1 to Fig. 8, the same components are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0012] (First embodiment) FIG. 1 is a block diagram showing the configuration of an image processing device 1 according to the first embodiment.

[0013] The image processing device 1 of this embodiment is a device that aligns a real space captured by a camera with a three-dimensional model space, and displays a real space image and an image of a virtual object (virtual model) in a superimposed manner. As shown in Fig. 1, the image processing device 1 of this embodiment includes a head-mounted video unit 10, a design unit 20, and a display unit 30. Fig. 1 further shows a marker M provided in the real space for alignment.

[0014] The head-mounted image unit 10 is a device worn on the head of an operator (user), and is, for example, a wearable computer equipped with a CPU (Central Processing Unit). In this embodiment, the image processing device 1 is a goggle-type holographic display device, and the head-mounted image unit 10 is goggles that function as a wearable computer. In the following description, the image processing device 1 is also referred to as "display device 1" as appropriate. A detailed configuration of the head-mounted image unit 10 will be described later.

[0015] The design unit 20 is a functional block that places design members in the coordinate system of the design space, and is, for example, a PC equipped with a CPU. The design unit 20 of this embodiment is a CAD device, generates design members using a design support tool having a 3D CAD model, and places the design members in a 3D model space for design. The detailed configuration of the design unit 20 will be described later.

[0016] The display unit 30 is a functional block that displays various types of information on a screen, and is, for example, a display device. The display unit 30 in this embodiment is a liquid crystal display or a CRT (Cathode Ray Tube) display. The display unit 30 displays various types of information generated by the design unit 20. The display unit 30 also displays a real space image and a virtual object in a superimposed manner.

[0017] The marker M is a mark used to associate a coordinate system of the real space with a coordinate system of a three-dimensional model space for design. The marker M in this embodiment is, for example, a two-dimensional geometric pattern printed on a sheet such as paper.

[0018] Next, a detailed configuration of the design unit 20 will be described. The design unit 20 includes an attribute information adding unit 202, a design processing unit 204, and an input operation unit 206.

[0019] The attribute information adding unit 202 adds attribute information to the designed members generated by the design processing unit 204. For example, the attribute information adding unit 202 adds three-dimensional CAD model attribute information for selecting a three-dimensional CAD model to the designed members.

[0020] The design processing unit 204 generates design members using a design support tool and places the design members in the three-dimensional model space for design. The design processing unit 204 generates design members based on the attribute information assigned by the attribute information assigning unit 202 and places the design members in the three-dimensional model space for design. For example, the design processing unit 204 selects design members from the three-dimensional CAD model for design based on the three-dimensional CAD model attribute information assigned by the attribute information assigning unit 202.

[0021] The input operation unit 206 is used to input information required for operating the head-mounted imaging unit 10 and the design unit 20 to the image processing device 1 of this embodiment. The input operation unit 206 is configured with, for example, a keyboard, a mouse, a pointing device, and the like.

[0022] Next, we will explain the detailed configuration of the head-mounted video unit 10. The head-mounted video unit 10 includes a plurality of cameras 102, a microphone 104, an input processing unit 106, a data storage unit 107, a recognition processing unit 108, an image control processing unit 110, a liquid crystal screen 112, and a speaker 114.

[0023] The camera 102 outputs an image of the real space as image data. The camera 102 is, for example, a color digital camera such as a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera. The image data captured by the camera 102 is output to the image control processing unit 110 via the input processing unit 106.

[0024] The microphone 104 outputs the sound collected near the head-mounted image unit 10 as audio data. For example, the microphone 104 collects the sound as a gesture made by the operator, and outputs audio data about the gesture. This audio data is output to the recognition processing unit 108 via the input processing unit 106. Note that the image processing device 1 of this embodiment may further capture the gesture of the operator with the camera 102, and output image data about the gesture from the camera 102 to the recognition processing unit 108 via the input processing unit 106.

[0025] The input processing unit 106 performs processing for inputting image data and voice data to the recognition processing unit 108 and the image processing control unit 110. The input processing unit 106 includes a position adjustment unit 106a and a conversion unit 106b.

[0026] The alignment unit 106a aligns the coordinate system of the three-dimensional model space in which the virtual object generated based on the design member is arranged with the coordinate system of the real space. The alignment unit 106a of this embodiment recognizes the position and orientation of the head-mounted image unit 10 using information of the marker M captured by the camera 102, and aligns the coordinate system of the three-dimensional model space with the coordinate system of the real space. For example, the alignment unit 106a sets the position of the marker M to the coordinate origin of the coordinate system of the three-dimensional model space. In this case, the marker M is placed in advance at a position in the real space corresponding to the coordinate origin of the coordinate system of the three-dimensional model space, and the position and orientation of the head-mounted image unit 10 is recognized based on the image data of the captured marker M.

[0027] The conversion unit 106b converts information about the design members generated by the design processing unit 204 into virtual objects. The conversion unit 106b of this embodiment converts the design members into virtual objects arranged in the three-dimensional model space using information about the design members arranged in the three-dimensional model space, for example, information about a three-dimensional CAD model.

[0028] The data storage unit 107 stores various types of data. The data storage unit 107 is configured, for example, with a magnetic recording device such as a hard disk drive (HDD) and / or a semiconductor memory such as a random access memory (RAM), a read only memory (ROM), or a solid state drive (SSD). The data storage unit 107 of this embodiment is used, for example, to store data and information used for alignment by the alignment unit 106a.

[0029] The recognition processing unit 108 performs various types of recognition processing and includes a voice recognition processing unit 108a, an image recognition processing unit 108b, and a three-dimensional space recognition processing unit 108c.

[0030] The voice recognition processor 108a processes voice data acquired from the microphone 104 via the input processor 106. For example, the voice recognition processor 108a converts the voice data collected by the microphone 104 into an instruction code. In this case, the head-mounted video unit 10 performs a control operation according to the instruction code.

[0031] The image recognition processing unit 108b processes image data acquired from the camera 102 via the input processing unit 106. For example, the image recognition processing unit 108b acquires distance information between the coordinates in the real space image in the superimposed image and the coordinates on the image of the virtual object.

[0032] The three-dimensional space recognition processing unit 108c also processes image data acquired from the camera 102 via the input processing unit 106. For example, the three-dimensional space recognition processing unit 108c calculates three-dimensional coordinates of natural feature points extracted from an image in real space based on the image data by SfM (Structure from Motion) and stores them as landmark data. In this case, the three-dimensional space recognition processing unit 108c estimates the position and orientation of the head-mounted image unit 10 based on the correspondence between the landmark data and the natural feature points. This makes it possible to estimate the position and orientation of the head-mounted image unit 10 in the coordinate system in real space even when the marker M is not captured.

[0033] The image processing control unit 110 controls image processing and other processing, and includes a control unit 110a, an image processing unit 110b, and an output processing unit 110c.

[0034] The control unit 110a controls various operations of the image processing device 1 of this embodiment. For example, the control unit 110a controls the operation of the head-mounted video unit 10, and controls the operations of the design unit 20 and the display unit 30 through the control of the head-mounted video unit 10.

[0035] Based on the alignment by the alignment unit 106a, the image processing unit 110b synthesizes a real space image observed from the user's viewpoint position with an image of a virtual object observed from the user's viewpoint position. After the alignment by the alignment unit 106a is completed, the image processing unit 110b changes the position and shape of a virtual object generated based on a design member based on the position and orientation of the head-mounted image unit 10 estimated by the three-dimensional space recognition processing unit 108c, and superimposes the virtual object on the real space image. The image processing unit 110b also generates a movable object of the virtual object and superimposes it on the real space image. In this way, the virtual object generated by the image processing unit 110b is superimposed on the real space image.

[0036] The output processing unit 110c converts the data format of data input from the image processing unit 110b and the like, and outputs the converted data. The converted data is output to, for example, the display unit 30, the liquid crystal screen 112, the speaker 114, and the like. The real space image and the virtual object superimposed by the image processing unit 110b are output from the output processing unit 110c, and are displayed in a superimposed manner on the display unit 30 and the liquid crystal screen 112. The output processing unit 110c is an example of a superimposed display unit.

[0037] The liquid crystal screen 112 displays various information at a position where the user wearing the head mounted image unit 10 can see. For example, the liquid crystal screen 112 displays a real space image observed from the user's viewpoint position and an image of a virtual object observed from the user's viewpoint position in a superimposed manner. The liquid crystal screen 112 of this embodiment is a head mounted display. The liquid crystal screen 112 may realize display by an optical see-through method. Such display is realized, for example, by displaying an image of a virtual object generated according to the position and posture of the observer's viewpoint on an optical see-through type liquid crystal display. In this case, the image of the virtual object is superimposed so as to be superimposed on the actual real space, not on the image of the real space. In this way, the superimposed display of this embodiment may be performed on an image of the real space, or may be performed on the actual real space.

[0038] The speaker 114 outputs audio to the ears of a user wearing the head-mounted video unit 10. For example, the speaker 114 outputs audio based on the recognition result by the recognition processing unit 108, such as audio based on an instruction code output by the voice recognition processing unit 108a. This audio may be used as an instruction for a design support tool.

[0039] The communication between the head mounted video unit 10 and the design unit 20 may be wireless or wired. Similarly, the communication between the head mounted video unit 10 and the display unit 30 may be wireless or wired. The design unit 20 may be disposed inside the head mounted video unit 10 instead of being disposed outside the head mounted video unit 10. An example of the design unit 20 disposed outside the head mounted video unit 10 is a PC, and an example of the design unit 20 disposed inside the head mounted video unit 10 is an MC (Micro Controller). When the design unit 20 is disposed inside the head mounted video unit 10, the image processing device 1 does not need to include the display unit 30.

[0040] Furthermore, the image processing device 1 of this embodiment may generate design members using a design support tool and place the design members in a three-dimensional model space for design while the operator wears the head-mounted image unit 10. This allows the design members designed on-site to be observed as images of virtual objects. In this case, a UI (User Interface) for operating the design support tool may be realized by displaying a virtual keyboard or virtual numeric keypad on the liquid crystal screen 112. This makes it possible to confirm, verify, and correct the virtual objects of the design members that overlap the actual real space while performing the design activity, thereby making it possible to further improve design efficiency. As a result, it becomes possible to eliminate the need for a 3D laser scanning process, which can lead to a significant shortening of the process.

[0041] FIG. 2 is a diagram for explaining the model space of the first embodiment.

[0042] Fig. 2 shows a lattice model MDL indicating a model space and the arrangement of design members A1 to A4. Fig. 2 shows a part of one plane (horizontal plane) in the three-dimensional lattice point model MDL. The attribute information assignment unit 202 (Fig. 1) generates a three-dimensional lattice point model MDL as a three-dimensional model space corresponding to the real space to be constructed based on the actual dimensions of the real space to be constructed.

[0043] The lines X1 to X4 and Y1 to Y3 shown in Fig. 2 indicate the X and Y coordinates in the lattice point model MDL. For example, the lines X1 to X4 and Y1 to Y3 correspond to the positions of the grid lines in real space. Furthermore, the marker position M' indicates the position where the marker M (Fig. 1) is placed. For example, the design members A1 to A3 are one flange pipe and two pipes on both ends of the flange pipe, and the design member A4 is one straight pipe.

[0044] The attribute information assigning unit 202 assigns the position where the designed member is to be placed in the three-dimensional model space to the designed member as attribute information. For example, the attribute information assigning unit 202 assigns the position where the designed member is to be placed to the designed member as attribute information based on the coordinates on the lattice point model MDL specified by the input processing unit 106 (FIG. 1). The attribute information assigning unit 202 can also assign the position where the designed member is to be placed to the designed member as attribute information, corresponding to the position of the center line on the lattice point model MDL specified by the input processing unit 106. In addition, the attribute information assigning unit 202 may place the designed member in the three-dimensional model space using distance information acquired by the image recognition processing unit 108b (FIG. 1).

[0045] FIG. 3 is a schematic diagram showing the configuration of the image processing system according to the first embodiment.

[0046] The image processing system of this embodiment is composed of two display devices (image processing devices) 1a and 1b. Each of the display devices 1a and 1b has the same configuration as the display device (image processing device) 1 shown in FIG. 1. Each of the display devices 1a and 1b of this embodiment is a goggle-type holographic display device. FIG. 3 shows a user Pa wearing the goggles (head-mounted image unit 10) of the display device 1a and a user Pb wearing the goggles (head-mounted image unit 10) of the display device 1b. One of the display devices 1a and 1b is an example of a first device, and the other of the display devices 1a and 1b is an example of a second device. The image processing system of this embodiment may be composed of three or more display devices 1.

[0047] Fig. 3 shows a real space 301 where users Pa and Pb are located, and a virtual model (virtual object) 302 superimposed on a real space image on the liquid crystal screen 112 (Fig. 1) of the display devices 1a and 1b. Fig. 3 shows a cube as an example of the three-dimensional virtual model 302. Users Pa and Pb can see the virtual model 302 projected onto the real space 301. Fig. 3 further shows a marker M placed in the real space 301.

[0048] The alignment unit 106a (FIG. 1) provided in each of the display devices 1a and 1b will be described below.

[0049] In each of the display devices 1a and 1b, the alignment unit 106a aligns the coordinate system of a three-dimensional model space in which a virtual model 302 generated based on a design member is arranged, with the coordinate system of the real space 301. For example, the alignment unit 106a recognizes the position and orientation of the head-mounted image unit 10 using information on a marker M captured by a camera 102 (FIG. 1), and aligns the coordinate system of the three-dimensional model space with the coordinate system of the real space 301.

[0050] In each of the display devices 1a and 1b of this embodiment, the alignment unit 106a performs the above alignment based on virtual model data (virtual object data) which is data of the virtual model 302, and relative position information which indicates the relative position between the marker M and the virtual model 302. Fig. 3 shows a schematic diagram of the virtual model data Da used in the display device 1a and the virtual model data Db used in the display device 1b. On the other hand, an example of the relative position information is information on the position of the origin of the coordinate system of the three-dimensional model space in which the virtual model 302 is placed, relative to the position of the marker M.

[0051] In this embodiment, the alignment unit 106a of the display device 1a and the alignment unit 106a of the display device 1b acquire the same virtual model data and relative position information, i.e., virtual model data and relative position information common to the display device 1a and the display device 1b. Therefore, the virtual model data Da and the virtual model data Db are data with the same content. Then, the alignment unit 106a of the display device 1a performs the above alignment based on the acquired virtual model data and relative position information. Similarly, the alignment unit 106a of the display device 1b also performs the above alignment based on the acquired virtual model data and relative position information.

[0052] The alignment unit 106a of the display device 1a and the alignment unit 106a of the display device 1b can acquire the same virtual model data and relative position information by acquiring the virtual model data and the relative position information from the same storage, for example. This storage may be provided in the display device 1a, may be provided in the display device 1b, or may be provided outside the display devices 1a and 1b. In addition, the display devices 1a and 1b may store the virtual model data and the relative position information acquired from the same storage in the data storage unit 107 (FIG. 1) of the display devices 1a and 1b in advance, and read out the virtual model data and the relative position information from the data storage unit 107 of the display devices 1a and 1b when displaying the virtual model 302. Examples of such storage will be described later in the second to fourth embodiments.

[0053] As described above, the display devices 1a and 1b display the virtual model 302 based on the same virtual model data. This makes it possible for the display devices 1a and 1b to display the virtual model 302 of the same shape. In addition, the display devices 1a and 1b perform the above-mentioned alignment based on the same virtual model data and relative position information. This makes it possible for the display devices 1a and 1b to arrange the virtual models 302 of the same shape at the same position in the real space image in the same orientation. This is because the reference position for arranging the virtual model 302 can be made common between the display devices 1a and 1b. Therefore, according to this embodiment, it becomes possible for the display devices 1a and 1b to share the same display content of the same virtual model 302.

[0054] In Fig. 3, user Pa is located on the right side of virtual model 302, and user Pb is located on the left side of virtual model 302. Therefore, user Pa observes virtual model 302 from the right side of virtual model 302 on liquid crystal screen 112, and user Pb observes virtual model 302 from the left side of virtual model 302 on liquid crystal screen 112. According to this embodiment, users Pa and Pb can simultaneously observe the same virtual model 302 from any position. In Fig. 3, users Pa and Pb observe the same virtual model 302 from different angles.

[0055] Second embodiment FIG. 4 is a schematic diagram showing the configuration of an image processing system according to the second embodiment.

[0056] As shown in Fig. 4, the image processing system of this embodiment includes similar components to those of the image processing system of the first embodiment. Fig. 4 further shows a network connector 401, a network 402, a server 403, a storage 404, a cloud 405, and a storage 406 that can be used by the image processing system of this embodiment. The storages 404 and 406 are examples of storages outside the display devices 1a and 1b.

[0057] The display devices 1a and 1b can be connected to a network 402 via a network connector 401. The network connector 401 is, for example, a communication router or a PC. The network 402 is, for example, an on-premise network such as a VPN (Virtual Private Network). The display devices 1a and 1b can access a storage 404 of a server 403 via the network 402. The storage 404 is, for example, configured with an HDD and / or an SSD. FIG. 4 diagrammatically shows virtual model data D1 stored in the storage 404. The virtual model data D1 is data of the virtual model 302. In this embodiment, relative position information indicating the relative position between the marker M and the virtual model 302 is also stored in the storage 404.

[0058] Furthermore, the display devices 1a and 1b can access a storage 406 on a cloud 405 via a network connector 401. The storage 406 is configured by, for example, an HDD and / or an SSD. FIG. 4 diagrammatically shows virtual model data D2 stored in the storage 406. Like the virtual model data D1, the virtual model data D2 is also data of the virtual model 302. In this embodiment, relative position information indicating the relative position between the marker M and the virtual model 302 is also stored in the storage 406.

[0059] In this embodiment, the alignment unit 106a of the display device 1a and the alignment unit 106a of the display device 1b acquire the virtual model data D1 and the relative position information from the storage 404, and thereby it is possible to acquire the same virtual model data D1 and relative position information. This makes it possible for the display device 1a and the display device 1b to perform the above-mentioned alignment based on the same virtual model data D1 and relative position information. In this case, the virtual model data D1 acquired by the display device 1a corresponds to the virtual model data Da in FIG. 3, and the virtual model data D1 acquired by the display device 1b corresponds to the virtual model data Db in FIG. 3.

[0060] In this embodiment, the alignment unit 106a of the display device 1a and the alignment unit 106a of the display device 1b may acquire the same virtual model data D2 and relative position information by acquiring the virtual model data D2 and the relative position information from the storage 406. In this case, the display device 1a and the display device 1b can perform the above alignment based on the same virtual model data D2 and relative position information. In this case, the virtual model data D2 acquired by the display device 1a corresponds to the virtual model data Da in FIG. 3, and the virtual model data D2 acquired by the display device 1b corresponds to the virtual model data Db in FIG. 3.

[0061] For example, when a company uses the display devices 1a and 1b, the storages 404 and 406 may be arranged at the company's headquarters, and the display devices 1a and 1b may be used on-site. As a result, after transporting the display devices 1a and 1b to the site, it becomes possible to provide the virtual model data and the relative position information to the display devices 1a and 1b. However, the arrangement location of the storages 404 and 406 may be other than the headquarters, and the usage location of the display devices 1a and 1b may be other than the site.

[0062] The display devices 1a and 1b can further communicate with each other via the network connector 401. An example of such communication will be described below.

[0063] In the image processing system of this embodiment, the display device 1a functions as a parent device, and the display device 1b functions as a child device. A user Pa of the display device 1a, which is a parent device, can input an operation related to the virtual model 302 to the display device 1a. For example, the user Pa can perform an operation to measure the dimensions of the virtual model 302 displayed on the liquid crystal screen 112, and thereby the measurement result of the dimensions can be displayed on the liquid crystal screen 112. FIG. 4 shows a measurement value L of the length of one side of a cube. The user Pa can input an operation related to the virtual model 302 to the display device 1a by moving a hand-shaped cursor H on the liquid crystal screen 112. For example, when measuring the length of one side of a cube, the start point and the end point of the side can be specified with the cursor H to measure the length of the side. In addition, the user Pa can input an operation to move or rotate the virtual model 302 in the real space image to the display device 1a, and thereby the virtual model 302 can be moved or rotated on the liquid crystal screen 112.

[0064] In this way, the display device 1a, which is the parent device, can change the display content of the virtual model 302 in various ways. The image processing unit 110b of the display device 1a changes the display content of the virtual model 302 by accepting an operation performed by the user Pa. This function of the image processing unit 110b is an example of a display control unit. In addition, the output processing unit 110c of the display device 1a transmits operation information related to the operation performed by the user Pa to the display device 1b, which is the child device, via the network connector 401. This function of the output processing unit 110c is an example of an information providing unit.

[0065] When the image processor 110b of the display device 1b receives the operation information from the display device 1a, the image processor 110b changes the display content of the virtual model 302 on the liquid crystal screen 112 of the display device 1b based on the operation information. As a result, when the user Pa measures the length of one side of the cube, the measurement value L is displayed not only on the liquid crystal screen 112 of the display device 1a but also on the liquid crystal screen 112 of the display device 1b. Also, when the user Pa performs an operation to rotate the virtual model 302, the virtual model 302 rotates not only on the liquid crystal screen 112 of the display device 1a but also on the liquid crystal screen 112 of the display device 1b. Therefore, according to this embodiment, it is possible to share the change in the display content of the virtual model 302 between the display device 1a and the display device 1b, and further, it is possible to share the change in the display content in real time. The user Pa becomes the operator of the virtual model 302, and the user Pb becomes the viewer of the operation result. The above operation information is an example of predetermined information provided from the parent device to the child device.

[0066] The operation information may be in any format as long as it is information that can identify the content of the operation performed by the user Pa and the change in the display content associated with the operation. For example, the operation information regarding the operation of measuring the length of one side of a cube can be configured using information regarding the start point of the side, the end point of the side, and the measured value L.

[0067] FIG. 5 is a block diagram for explaining the operation of the image processing system according to the second embodiment.

[0068] As shown in Fig. 5, the display devices 1a and 1b can acquire virtual model data D1 and relative position information from the storage 404 via the network connector 401 (Fig. 4). In addition, the display device 1a, which is the parent device, can transmit data D to the display device 1b, which is the child device, via the network connector 401. An example of the data D is the above-mentioned operation information.

[0069] In this case, the data D may be transmitted from the display device 1a to the storage 404 and stored in the storage 404, and the data D stored in the storage 404 may be transmitted to the display device 1b. This makes it possible to store the data D such as operation information in the storage 404 together with the virtual model data D1 and the relative position information. In this case, the storage 406 (FIG. 4) may be used instead of the storage 404.

[0070] According to this embodiment, it is possible to share changes to the display contents of the virtual model 302 by transmitting operation information, which generally has a small amount of data, from the display device 1a to the display device 1b, instead of transmitting virtual model data, which generally has a large amount of data, from the display device 1a to the display device 1b. This makes it possible to reduce the amount of communication between the display devices 1a and 1b. For example, it is considered that the amount of communication when transmitting operation information from the display device 1a to the display device 1b is significantly reduced compared to the amount of communication when transmitting video data of the virtual model 302 from the display device 1a to the display device 1b. Furthermore, according to this embodiment, it is possible to improve security by limiting the data to be communicated between the display devices 1a and 1b.

[0071] In the image processing system of this embodiment, the display device 1a may always function as a parent device and the display device 1b may always function as a child device, or the display device 1 functioning as a parent device and the display device 1 functioning as a child device may be switchable. On the other hand, the display device 1a may always function as a child device and the display device 1b may always function as a parent device. In this way, the parent-child relationship between the display devices 1a and 1b may be fixed or variable. Further details of the parent device and the child device will be described later in the third and fourth embodiments.

[0072] Third embodiment FIG. 6 is a schematic diagram showing the configuration of an image processing system according to the third embodiment.

[0073] As shown in Fig. 6, the image processing system of this embodiment has the same components as the image processing system of the first embodiment. Similarly to Fig. 4, Fig. 6 further shows a network connector 401 that can be used by the image processing system of this embodiment.

[0074] In this embodiment, the parent-child relationship between the display devices 1a and 1b is fixed. Therefore, only the display device 1a of the display devices 1a and 1b functions as a parent device, and only the display device 1b of the display devices 1a and 1b functions as a child device. A user Pa of the display device 1a becomes an operator of the virtual model 302, and a user Pb of the display device 1b becomes a viewer of the operation result. According to this embodiment, as in the second embodiment, it becomes possible for the display devices 1a and 1b to share changes to the display contents of the virtual model 302.

[0075] In this embodiment, the virtual model data Da and the relative position information are stored in the data storage unit 107 (FIG. 1) of the display device 1a. Therefore, the positioning unit 106a of the display device 1a acquires the virtual model data Da and the relative position information from the data storage unit 107 of the display device 1a. On the other hand, the virtual model data and the relative position information are not stored in the data storage unit 107 of the display device 1b. Therefore, the positioning unit 106a of the display device 1b acquires the virtual model data Da and the relative position information from the data storage unit 107 of the display device 1a via the network connector 401. This makes it possible to perform positioning on the display device 1a and the display device 1b based on the same virtual model data Da and relative position information.

[0076] According to this embodiment, even when the network 402 or the cloud 405 cannot be used, or when the storages 404 and 406 cannot be used, the display devices 1a and 1b can share the display contents and changes thereto. On the other hand, according to the second embodiment, it is possible to reduce the amount of communication between the display devices 1a and 1b compared to this embodiment.

[0077] In this embodiment, the display device 1a includes a data storage unit 107 that stores the virtual model data Da and the relative position information, and the display device 1b does not include a data storage unit 107 that stores the virtual model data and the relative position information. To realize such a configuration, the display device 1b of this embodiment may not include the data storage unit 107 itself, or may include a data storage unit 107 that is not used to store the virtual model data and the relative position information.

[0078] (Fourth embodiment) FIG. 7 is a schematic diagram showing the configuration of an image processing system according to the fourth embodiment.

[0079] As shown in Fig. 7, the image processing system of this embodiment has the same components as the image processing system of the first embodiment. Fig. 7 further shows a network connector 401 that can be used by the image processing system of this embodiment, similar to Figs. 4 and 6.

[0080] In this embodiment, not only can the user Pa of the display device 1a be the operator of the virtual model 302 and the user Pb of the display device 1b be the viewer of the operation results, but also the user Pb of the display device 1b can be the operator of the virtual model 302 and the user Pa of the display device 1a can be the viewer of the operation results. This makes it possible to change the display contents of the virtual model 302 more freely.

[0081] In this embodiment, the virtual model data Da and the relative position information are stored in the data storage unit 107 (FIG. 1) of the display device 1a. Therefore, the alignment unit 106a of the display device 1a acquires the virtual model data Da and the relative position information from the data storage unit 107 of the display device 1a. Also, in this embodiment, the virtual model data Db and the relative position information are stored in the data storage unit 107 of the display device 1b. Therefore, the alignment unit 106a of the display device 1b acquires the virtual model data Da and the relative position information from the data storage unit 107 of the display device 1b. In this embodiment, the same virtual model data and relative position information are stored in advance in the data storage unit 107 of the display device 1a and in the data storage unit 107 of the display device 1b. This makes it possible to perform alignment based on the same virtual model data and relative position information in the display device 1a and the display device 1b.

[0082] According to this embodiment, even when the network 402 or the cloud 405 cannot be used or when the storages 404 and 406 cannot be used, the display device 1a and the display device 1b can share the display contents and changes thereto. Furthermore, according to this embodiment, it is possible to reduce the amount of communication between the display devices 1a and 1b compared to the third embodiment. This is because, in this embodiment, it is not necessary to provide the virtual model data Da and the relative position information from the display device 1a to the display device 1b. In general, when the amount of virtual model data Da is large, it is desirable to adopt this embodiment, and when the amount of virtual model data Da is small, it is desirable to adopt the third embodiment.

[0083] In this embodiment, the parent-child relationship between the display devices 1a and 1b is variable. That is, the display device 1 functioning as a parent device and the display device 1 functioning as a child device can be switched. This will be described in detail with reference to FIG. 8.

[0084] FIG. 8 is a block diagram for explaining the operation of the image processing system according to the fourth embodiment.

[0085] FIG. 8(a) shows an image processing system in which the display device 1a functions as a parent device and the display device 1b functions as a child device. In this embodiment, each of the display devices 1a and 1b includes a switching unit 501 for switching the parent-child relationship. For example, the switching unit 501 of the display device 1a sets the display device 1a as a parent device, and the switching unit 501 of the display device 1b sets the display device 1b as a child device in response to a request from the switching unit 501 of the display device 1a, so that the display devices 1a and 1b can become a parent device and a child device, respectively. The parent-child relationship may be switched in response to an operation by the user Pa or the user Pb, or may be automatically switched by the display device 1a or the display device 1b. The switching unit 501 may be provided in only one of the display devices 1a and 1b.

[0086] In Fig. 8(a), display device 1a, which is a parent device, transmits data D such as operation information to display device 1b, which is a child device, via network connector 401. This makes it possible for changes in the display content made by operation by user Pa to be reflected not only on liquid crystal screen 112 of display device 1a, but also on liquid crystal screen 112 of display device 1b.

[0087] 8(b) shows an image processing system in which the display device 1b functions as a parent device and the display device 1a functions as a child device. The function of the switching unit 501 of the display devices 1a and 1b is as described above.

[0088] In Fig. 8(b), display device 1b, which is a parent device, transmits data D such as operation information to display device 1a, which is a child device, via network connector 401. This makes it possible for changes in the display content made by the operation of user Pb to be reflected not only on liquid crystal screen 112 of display device 1b, but also on liquid crystal screen 112 of display device 1a.

[0089] In each of the second to fourth embodiments, the image processing system may be configured with N display devices 1 (N is an integer equal to or greater than 3). In this case, one display device 1 may function as a parent device, and the remaining N-1 display devices 1 may function as child devices.

[0090] Although some embodiments have been described above, these embodiments are presented only as examples and are not intended to limit the scope of the invention. The novel device, method, and system described in this specification can be implemented in various other forms. In addition, various omissions, substitutions, and modifications can be made to the forms of the device, method, and system described in this specification without departing from the spirit of the invention. The appended claims and their equivalents are intended to include such forms and modifications that fall within the scope and spirit of the invention. [Explanation of symbols]

[0091] 1, 1a, 1b: image processing device (display device), 10: Head-mounted video department, 20: Design department, 30: Display department, 102: camera, 104: microphone, 106: input processing unit, 106a: alignment unit, 106b: conversion unit, 107: data storage unit, 108: Recognition processing unit, 108a: Voice recognition processing unit, 108b: image recognition processing unit, 108c: three-dimensional space recognition processing unit, 110: image processing control unit, 110a: control unit, 110b: image processing unit, 110c: output processing unit, 112: liquid crystal screen, 114: speaker, 202: attribute information assigning unit, 204: design processing unit, 206: input operation unit, 301: Real space, 302: Virtual model, 401: network connector, 402: network, 403: server, 404: Storage, 405: Cloud, 406: Storage, 501: Switching unit

Claims

1. a positioning unit that performs positioning between a coordinate system of a three-dimensional model space in which a virtual object generated based on a design member designed by computer-aided design is placed and a coordinate system of a real space; an image processing unit that generates an image of the virtual object observed from a viewpoint position of the user in correspondence with a real space image observed from a viewpoint position of the user based on the alignment; and a superimposition display unit that displays the real space image and the virtual object in a superimposed manner; An image processing device comprising: the alignment unit performs the alignment based on virtual object data that is data of the virtual object and relative position information that represents a relative position between a marker provided in the real space and the virtual object; the position alignment unit acquires the virtual object data and the relative position information common to a first device corresponding to the image processing device and a second device corresponding to an image processing device different from the first device, and performs the position alignment based on the acquired virtual object data and the acquired relative position information. Image processing device.

2. The image processing apparatus of claim 1 , wherein each of the first and second devices is a goggle-type holographic display device.

3. The image processing device according to claim 1 , wherein the first and second devices acquire the virtual object data and the relative position information stored in a storage outside the first and second devices.

4. The image processing device according to claim 3 , wherein the first and second devices acquire the virtual object data and the relative position information from the storage via a network, or acquire the virtual object data and the relative position information from the storage on a cloud.

5. At least the first device of the first and second devices includes a data storage unit that stores the virtual object data and the relative position information, The image processing device according to claim 1 , wherein the position adjustment unit acquires the virtual object data and the relative position information stored in the data storage unit of the first device.

6. The image processing device according to claim 5 , wherein only the first device of the first and second devices includes the data storage unit that stores the virtual object data and the relative position information.

7. The image processing device according to claim 6 , wherein the second device acquires the virtual object data and the relative position information stored in the data storage unit of the first device.

8. The image processing device according to claim 5 , wherein each of the first and second devices includes the data storage unit that stores the virtual object data and the relative position information.

9. a display control unit that changes the display content of the virtual object by the superimposed display unit by receiving an operation performed by the user; an information providing unit that provides operation information regarding the operation performed by the user from the first device to the second device; The image processing device according to claim 1 , further comprising:

10. The image processing device according to claim 9 , wherein the second device changes a display content of the virtual object by the second device based on the operation information provided from the first device to the second device.

11. one of the first and second devices functions as a parent device that outputs predetermined information about the virtual object; The other of the first and second devices functions as a child device that acquires the predetermined information output by the parent device. The image processing device according to claim 1 .

12. The image processing device according to claim 11 , wherein the predetermined information includes operation information related to an operation performed by the user on the display content of the virtual object.

13. Only the first device of the first and second devices functions as the parent device; Of the first and second devices, only the second device functions as the child device. The image processing device according to claim 11.

14. The image processing device according to claim 11 , wherein the first and second devices are switchable between functioning as the parent device and functioning as the child device.

15. a position alignment unit of the image processing device aligns a coordinate system of a three-dimensional model space in which a virtual object generated based on a design member designed by computer-aided design is placed with a coordinate system of a real space; an image processing unit of the image processing device generates an image of the virtual object observed from a viewpoint position of the user, corresponding to a real space image observed from a viewpoint position of the user, based on the alignment; a superimposed display unit of the image processing device superimposing the real space image and the virtual object; Including, the alignment unit performs the alignment based on virtual object data that is data of the virtual object and relative position information that represents a relative position between a marker provided in the real space and the virtual object; the position alignment unit acquires the virtual object data and the relative position information common to a first device corresponding to the image processing device and a second device corresponding to an image processing device different from the first device, and performs the position alignment based on the acquired virtual object data and the acquired relative position information. Image processing methods.

16. An image processing system including a first device corresponding to an image processing device and a second device corresponding to an image processing device different from the first device, Each of the first and second devices comprises: a positioning unit that performs positioning between a coordinate system of a three-dimensional model space in which a virtual object generated based on a design member designed by computer-aided design is placed and a coordinate system of a real space; an image processing unit that generates an image of the virtual object observed from a viewpoint position of the user in correspondence with a real space image observed from a viewpoint position of the user based on the alignment; and a superimposition display unit that displays the real space image and the virtual object in a superimposed manner; Equipped with the alignment unit performs the alignment based on virtual object data that is data of the virtual object and relative position information that represents a relative position between a marker provided in the real space and the virtual object; the position alignment unit acquires the virtual object data and the relative position information common to a first device corresponding to the image processing device and a second device corresponding to an image processing device different from the first device, and performs the position alignment based on the acquired virtual object data and the acquired relative position information. Image processing system.

Images