Image processing system and image processing method

Abstract

To provide an image processing device that can preferably share a virtual model among a plurality of devices.SOLUTION: According to one embodiment, an image processing device comprises: a positioning unit which performs positioning of a coordinate system of a three-dimensional model space in which a virtual object is placed and a coordinate system of a real space; an image processing unit which generates an image of the virtual object observed from a user's viewpoint position based on the positioning; and a superimposed display unit which superimposes and displays a real space image and the virtual object. The positioning unit performs the positioning based on virtual object data and relative position information between a marker and the virtual object. Furthermore, the positioning unit obtains the virtual object data and the relative position information common to a first device corresponding to the image processing device and a second device corresponding to an image processing device different from the first device, and performs the positioning based on the obtained data and information.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to an image processing device, an image processing method, and an image processing system. [Background technology]

[0002] Mixed reality (MR) is a technology that uses stereoscopic technology to combine real and virtual spaces. For example, a method has been proposed for generating a stereoscopic image using left-eye and right-eye images captured by a stereo camera and a virtual model (virtual object). Another method has been proposed for superimposing a virtual model on a real-space image, taking into account the user's viewpoint. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-94102 [Patent Document 2] Japanese Patent Publication No. 2020-24497 Summary of the Invention [Problem to be solved by the invention]

[0004] With the advancement of the information society, in manufacturing sites, virtual models of products are created using 3D Computer Aided Design (CAD), and the shape and design of the products are checked and reviewed during the design phase using a personal computer (PC) or a goggle-type display device.In addition, in plant construction work, the status of plant equipment is checked and reviewed using virtual models on a PC during the design phase.

[0005] In the past, construction supervisors and contractors would mentally compare the design drawings with the actual site conditions during on-site construction to create construction plans. However, nowadays, goggle-type holographic display devices can be used to display a virtual model of the completed state superimposed on a real-world image.

[0006] However, existing goggle-type holographic display devices do not have a means for multiple display devices to share and review the same virtual model. Therefore, in order to share and review the same virtual model on multiple display devices, each display device must store virtual model data and the multiple display devices must place the virtual model at the same position on the on-site space image for confirmation and review.

[0007] According to this technique, the shape of the virtual model viewed by a user of one display device and the shape of the virtual model viewed by a user of the other display device are the same model shape. However, the data used for display on the former display device and the data used for display on the latter display device are independent data. Therefore, even if one user changes the display content of the virtual model, this change cannot be reflected in the display content of the virtual model of the other user. For example, even if the former user rotates the virtual model, the latter user cannot see the rotation.

[0008] Therefore, the embodiments of the present invention provide an image processing device, an image processing method, and an image processing system that enable a virtual model to be suitably shared among a plurality of devices. [Means for solving the problem]

[0009] According to one embodiment, image processing system teeth, The image processing device includes a plurality of image processing devices, each of which includes: Place virtual objects generated based on design elements designed by computer-aided design. The marker position coordinates were set along with Alignment of the coordinate system of the 3D model space with the coordinate system of the real space based on position information of the marker obtained by capturing an image of the marker placed at a position in the real space corresponding to the marker position coordinates and the marker position coordinates in the three-dimensional model space. It is equipped with an alignment unit that performs alignment. Each of the plurality of image processing devicesThe image processing unit further includes an image processing unit that generates an image of the virtual object observed from the user's viewpoint position, corresponding to a real space image observed from the user's viewpoint position, based on the alignment. Each of the plurality of image processing devices The present invention further includes a superimposition display unit that displays the real space image and the virtual object in a superimposed manner. .difference Furthermore, Among the plurality of image processing devices No. 1 Image Processing device, and the first Image Processing Different from the device other No. 2 Image Processing Device What is that? Common The data of the virtual object Virtual object data and A coordinate system representing a relative position of the virtual object relative to the marker in a three-dimensional model space. Obtain relative position information, Furthermore, based on the position information of the marker obtained by each alignment unit capturing an image of the common marker at each position and the marker position coordinates in the three-dimensional model space, a coordinate system of the three-dimensional model space and a coordinate system of the real space are aligned. Perform alignment. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of an image processing device according to a first embodiment. [Figure 2] FIG. 2 is a diagram for explaining a model space in the first embodiment. [Figure 3] FIG. 1 is a schematic diagram illustrating a configuration of an image processing system according to a first embodiment. [Figure 4] FIG. 10 is a schematic diagram showing the configuration of an image processing system according to a second embodiment. [Figure 5] FIG. 10 is a block diagram for explaining the operation of an image processing system according to a second embodiment. [Figure 6] FIG. 10 is a schematic diagram showing the configuration of an image processing system according to a third embodiment. [Figure 7] FIG. 10 is a schematic diagram showing the configuration of an image processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a block diagram for explaining the operation of an image processing system according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings. In Figures 1 to 8, the same components are denoted by the same reference numerals, and redundant description will be omitted.

[0012] (First embodiment) FIG. 1 is a block diagram showing the configuration of an image processing device 1 according to the first embodiment.

[0013] The image processing device 1 of this embodiment is a device that aligns a real space captured by a camera with a three-dimensional model space and superimposes an image of a virtual object (virtual model) on an image of the real space. As shown in Fig. 1, the image processing device 1 of this embodiment includes a head-mounted video unit 10, a design unit 20, and a display unit 30. Fig. 1 also shows a marker M provided in the real space for alignment purposes.

[0014] The head-mounted image unit 10 is a device worn on the head of an operator (user), and is, for example, a wearable computer equipped with a CPU (Central Processing Unit). In this embodiment, the image processing device 1 is a goggle-type holographic display device, and the head-mounted image unit 10 is goggles that function as a wearable computer. In the following description, the image processing device 1 will also be referred to as the "display device 1" as appropriate. The detailed configuration of the head-mounted image unit 10 will be described later.

[0015] The design unit 20 is a functional block that places design components in a coordinate system of a design space, and is, for example, a PC equipped with a CPU. The design unit 20 of this embodiment is a CAD device that generates design components using a design support tool having a 3D CAD model and places the design components in a 3D model space for design. The detailed configuration of the design unit 20 will be described later.

[0016] The display unit 30 is a functional block that displays various types of information on a screen, and is, for example, a display device. In this embodiment, the display unit 30 is a liquid crystal display or a CRT (Cathode Ray Tube) display. The display unit 30 displays various types of information generated by the design unit 20. The display unit 30 also displays a real space image and a virtual object in a superimposed manner.

[0017] The marker M is a mark used to associate a coordinate system in the real space with a coordinate system in the three-dimensional model space for design. The marker M in this embodiment is, for example, a two-dimensional geometric pattern printed on a sheet of paper or the like.

[0018] Next, a detailed description will be given of the configuration of the design unit 20. The design unit 20 includes an attribute information assigning unit 202, a design processing unit 204, and an input operation unit 206.

[0019] The attribute information assigning unit 202 assigns attribute information to the design components generated by the design processing unit 204. For example, the attribute information assigning unit 202 assigns three-dimensional CAD model attribute information for selecting a three-dimensional CAD model to the design components.

[0020] The design processing unit 204 generates design members using a design support tool and places the design members in the 3D model space for design. The design processing unit 204 generates design members and places them in the 3D model space for design based on the attribute information assigned by the attribute information assigning unit 202. For example, the design processing unit 204 selects design members from the 3D CAD model for design based on the 3D CAD model attribute information assigned by the attribute information assigning unit 202.

[0021] The input operation unit 206 is used to input information required for operating the head-mounted video unit 10 and the design unit 20 to the image processing device 1 of this embodiment. The input operation unit 206 is configured by, for example, a keyboard, a mouse, a pointing device, etc.

[0022] Next, we will explain the detailed configuration of head-mounted video unit 10. Head-mounted video unit 10 includes multiple cameras 102, a microphone 104, an input processing unit 106, a data storage unit 107, a recognition processing unit 108, an image control processing unit 110, a liquid crystal screen 112, and a speaker 114.

[0023] The camera 102 outputs an image of real space as image data. The camera 102 is, for example, a color digital camera such as a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera. The image data captured by the camera 102 is output to the image control processing unit 110 via the input processing unit 106.

[0024] The microphone 104 outputs audio collected near the head-mounted video unit 10 as audio data. For example, the microphone 104 collects audio as gestures made by the operator and outputs audio data about the gestures. This audio data is output to the recognition processing unit 108 via the input processing unit 106. Note that the image processing device 1 of this embodiment may further capture an image of the operator's gestures with the camera 102 and output image data about the gestures from the camera 102 to the recognition processing unit 108 via the input processing unit 106.

[0025] The input processing unit 106 performs processing for inputting image data and audio data to the recognition processing unit 108 and the image processing control unit 110. The input processing unit 106 includes a position adjustment unit 106a and a conversion unit 106b.

[0026] The alignment unit 106a aligns the coordinate system of the three-dimensional model space, in which a virtual object generated based on the design components is placed, with the coordinate system of the real space. In this embodiment, the alignment unit 106a recognizes the position and orientation of the head-mounted image unit 10 using information about the marker M captured by the camera 102, and aligns the coordinate system of the three-dimensional model space with the coordinate system of the real space. For example, the alignment unit 106a sets the position of the marker M to the coordinate origin of the coordinate system of the three-dimensional model space. In this case, the marker M is placed in advance in a position in the real space corresponding to the coordinate origin of the coordinate system of the three-dimensional model space, and the position and orientation of the head-mounted image unit 10 is recognized based on image data capturing the marker M.

[0027] The conversion unit 106b converts the information of the design member generated by the design processing unit 204 into a virtual object. arrangement Using information about the designed components, for example, information about a three-dimensional CAD model, the designed components are converted into virtual objects arranged in a three-dimensional model space.

[0028] The data storage unit 107 stores various types of data. The data storage unit 107 is configured, for example, by a magnetic recording device such as a hard disk drive (HDD) and / or a semiconductor memory such as a random access memory (RAM), a read only memory (ROM), or a solid state drive (SSD). The data storage unit 107 of this embodiment is used, for example, to store data and information used for alignment by the alignment unit 106a.

[0029] The recognition processing unit 108 performs various types of recognition processing and includes a voice recognition processing unit 108a, an image recognition processing unit 108b, and a three-dimensional space recognition processing unit 108c.

[0030] The voice recognition processing unit 108a processes voice data acquired from the microphone 104 via the input processing unit 106. For example, the voice recognition processing unit 108a converts the voice data collected by the microphone 104 into an instruction code. In this case, the head-mounted video unit 10 performs a control operation in accordance with this instruction code.

[0031] The image recognition processing unit 108b processes image data acquired from the camera 102 via the input processing unit 106. For example, the image recognition processing unit 108b acquires distance information between the coordinates in the real space image in the superimposed image and the coordinates on the image of the virtual object.

[0032] The three-dimensional space recognition processing unit 108c also processes image data acquired from the camera 102 via the input processing unit 106. For example, the three-dimensional space recognition processing unit 108c calculates the three-dimensional coordinates of natural feature points extracted from an image in real space based on the image data using SfM (Structure from Motion), and stores the calculated coordinates as landmark data. In this case, the three-dimensional space recognition processing unit 108c estimates the position and orientation of the head-mounted image unit 10 based on the correspondence between the landmark data and the natural feature points. This makes it possible to estimate the position and orientation of the head-mounted image unit 10 in the coordinate system in real space even when an image of the marker M is not captured.

[0033] The image processing control unit 110 controls image processing and other processes, and includes a control unit 110a, an image processing unit 110b, and an output processing unit 110c.

[0034] The control unit 110a controls various operations of the image processing device 1 of this embodiment. For example, the control unit 110a controls the operation of the head-mounted video unit 10, and controls the operations of the design unit 20 and the display unit 30 through the control of the head-mounted video unit 10.

[0035] Based on the alignment performed by the alignment unit 106a, the image processing unit 110b combines a real-space image observed from the user's viewpoint with an image of a virtual object observed from the user's viewpoint. After the alignment performed by the alignment unit 106a is complete, the image processing unit 110b changes the position and shape of the virtual object generated based on the design components based on the position and orientation of the head-mounted video unit 10 estimated by the 3D space recognition processing unit 108c, and superimposes the virtual object on the real-space image. The image processing unit 110b also generates a movable object of the virtual object and superimposes it on the real-space image. In this way, the virtual object generated by the image processing unit 110b is superimposed on the real-space image.

[0036] The output processing unit 110c converts the data format of data input from the image processing unit 110b and the like, and outputs the converted data. The converted data is output to, for example, the display unit 30, the liquid crystal screen 112, the speaker 114, and the like. The real space image and virtual object superimposed by the image processing unit 110b are output from the output processing unit 110c and displayed in a superimposed manner on the display unit 30 and the liquid crystal screen 112. The output processing unit 110c is an example of a superimposed display unit.

[0037] The liquid crystal screen 112 displays various information at a position where it can be seen by a user wearing the head-mounted video unit 10. For example, the liquid crystal screen 112 displays a real space image observed from the user's viewpoint and an image of a virtual object observed from the user's viewpoint, superimposed on each other. The liquid crystal screen 112 of this embodiment is a head-mounted display. The liquid crystal screen 112 may realize display using an optical see-through method. Such display is realized, for example, by displaying an image of a virtual object generated according to the position and posture of the observer's viewpoint on an optical see-through liquid crystal display. In this case, the image of the virtual object is superimposed so as to be superimposed on the actual real space, rather than on the image of real space. In this way, the superimposed display of this embodiment may be performed on an image of real space, or on the actual real space.

[0038] The speaker 114 outputs audio to the ears of a user wearing the head-mounted video unit 10. For example, the speaker 114 outputs audio based on the recognition result by the recognition processing unit 108, such as audio based on an instruction code output by the audio recognition processing unit 108a. This audio may be used as an instruction for a design support tool.

[0039] Note that communication between the head-mounted video unit 10 and the design unit 20 may be wireless or wired communication. Similarly, communication between the head-mounted video unit 10 and the display unit 30 may be wireless or wired communication. Furthermore, the design unit 20 may be arranged inside the head-mounted video unit 10 instead of being arranged outside the head-mounted video unit 10. An example of a design unit 20 arranged outside the head-mounted video unit 10 is a PC, and an example of a design unit 20 arranged inside the head-mounted video unit 10 is an MC (Micro Controller). When the design unit 20 is arranged inside the head-mounted video unit 10, the image processing device 1 does not need to include the display unit 30.

[0040] Furthermore, the image processing device 1 of this embodiment may generate design components using a design support tool and place the design components in a 3D design model space while the operator is wearing the head-mounted video unit 10. This allows the design components designed on-site to be observed as images of virtual objects. In this case, the UI (User Interface) for operating the design support tool may be realized by displaying a virtual keyboard or virtual numeric keypad on the LCD screen 112. This allows the operator to confirm, verify, and modify the virtual objects of the design components that overlap with the actual real space while performing the design process, further improving design efficiency. As a result, it is possible to eliminate the need for 3D laser scanning, leading to a significant reduction in the process time.

[0041] FIG. 2 is a diagram for explaining the model space of the first embodiment.

[0042] Fig. 2 shows a lattice model MDL representing a model space and the arrangement of design members A1 to A4. Fig. 2 shows a portion of one plane (horizontal plane) in the three-dimensional lattice model MDL. The attribute information assigning unit 202 (Fig. 1) generates a three-dimensional lattice model MDL as a three-dimensional model space corresponding to the real space to be constructed, based on the actual dimensions of the real space to be constructed.

[0043] The lines X1 to X4 and lines Y1 to Y3 shown in Figure 2 indicate the X and Y coordinates within the lattice point model MDL. For example, the lines X1 to X4 and lines Y1 to Y3 correspond to the positions of the center lines in real space. Furthermore, the marker position M' indicates the position where the marker M (Figure 1) is placed. For example, the design members A1 to A3 are one flange pipe and two pipes at both ends of it, and the design member A4 is one straight pipe.

[0044] The attribute information assigning unit 202 assigns the position where the designed member is to be placed in the three-dimensional model space to the designed member as attribute information. For example, the attribute information assigning unit 202 assigns the position where the designed member is to be placed to the designed member as attribute information based on coordinates on the lattice point model MDL specified by the input processing unit 106 (FIG. 1). The attribute information assigning unit 202 can also assign the position where the designed member is to be placed to the designed member as attribute information, corresponding to the position of the center line on the lattice point model MDL specified by the input processing unit 106. Furthermore, the attribute information assigning unit 202 may place the designed member in the three-dimensional model space using distance information acquired by the image recognition processing unit 108b (FIG. 1).

[0045] FIG. 3 is a schematic diagram showing the configuration of the image processing system of the first embodiment.

[0046] The image processing system of this embodiment is composed of two display devices (image processing devices) 1a and 1b. Each of the display devices 1a and 1b has the same configuration as the display device (image processing device) 1 shown in FIG. 1. Each of the display devices 1a and 1b of this embodiment is a goggle-type holographic display device. FIG. 3 shows a user Pa wearing the goggles (head-mounted image unit 10) of the display device 1a, and a user Pb wearing the goggles (head-mounted image unit 10) of the display device 1b. One of the display devices 1a and 1b is an example of a first device, and the other of the display devices 1a and 1b is an example of a second device. Note that the image processing system of this embodiment may be composed of three or more display devices 1.

[0047] 3 shows a real space 301 where users Pa and Pb are located, and a virtual model (virtual object) 302 superimposed on the real space image on the liquid crystal screen 112 (FIG. 1) of the display devices 1a and 1b. FIG. 3 shows a cube as an example of the three-dimensional virtual model 302. Users Pa and Pb can see the virtual model 302 projected onto the real space 301. FIG. 3 also shows a marker M placed in the real space 301.

[0048] The alignment unit 106a (FIG. 1) provided in each of the display devices 1a and 1b will be described below.

[0049] In each of the display devices 1a and 1b, the alignment unit 106a aligns the coordinate system of the three-dimensional model space in which the virtual model 302 generated based on the design components is placed with the coordinate system of the real space 301. For example, the alignment unit 106a recognizes the position and orientation of the head-mounted image unit 10 using information on the marker M captured by the camera 102 (FIG. 1), and aligns the coordinate system of the three-dimensional model space with the coordinate system of the real space 301.

[0050] In each of the display devices 1a and 1b of this embodiment, the alignment unit 106a performs the above alignment based on virtual model data (virtual object data) that is data on the virtual model 302 and relative position information that indicates the relative position between the marker M and the virtual model 302. Fig. 3 schematically shows virtual model data Da used in the display device 1a and virtual model data Db used in the display device 1b. On the other hand, an example of the relative position information is information about the position of the origin of the coordinate system of the three-dimensional model space in which the virtual model 302 is placed, relative to the position of the marker M.

[0051] In this embodiment, the alignment unit 106a of the display device 1a and the alignment unit 106a of the display device 1b acquire the same virtual model data and relative position information, i.e., virtual model data and relative position information common to the display device 1a and the display device 1b. Therefore, the virtual model data Da and the virtual model data Db are data with the same content. The alignment unit 106a of the display device 1a then performs the above alignment based on the acquired virtual model data and relative position information. Similarly, the alignment unit 106a of the display device 1b also performs the above alignment based on the acquired virtual model data and relative position information.

[0052] The alignment unit 106a of the display device 1a and the alignment unit 106a of the display device 1b can acquire the same virtual model data and relative position information by acquiring the virtual model data and relative position information from the same storage, for example. This storage may be provided within the display device 1a, within the display device 1b, or external to the display devices 1a and 1b. Furthermore, the display devices 1a and 1b may each store the virtual model data and relative position information acquired from the same storage in the data storage unit 107 (FIG. 1) of the display devices 1a and 1b in advance, and read the virtual model data and relative position information from the data storage unit 107 of the display devices 1a and 1b when displaying the virtual model 302. Examples of such storage will be described later in the second to fourth embodiments.

[0053] As described above, the display devices 1a and 1b display the virtual model 302 based on the same virtual model data. This makes it possible for the display devices 1a and 1b to display the virtual model 302 of the same shape. In addition, the display devices 1a and 1b perform the above-described alignment based on the same virtual model data and relative position information. This makes it possible for the display devices 1a and 1b to place the virtual model 302 of the same shape at the same position and in the same orientation in the real space image. This is because the reference position for placing the virtual model 302 can be made common between the display devices 1a and 1b. Therefore, according to this embodiment, it is possible for the display devices 1a and 1b to share the same display content of the same virtual model 302.

[0054] In FIG. 3, user Pa is located on the right side of virtual model 302, and user Pb is located on the left side of virtual model 302. Therefore, user Pa observes virtual model 302 from the right side of virtual model 302 on liquid crystal screen 112, and user Pb observes virtual model 302 from the left side of virtual model 302 on liquid crystal screen 112. According to this embodiment, users Pa and Pb can simultaneously observe the same virtual model 302 from any position. In FIG. 3, users Pa and Pb observe the same virtual model 302 from different angles.

[0055] (Second embodiment) FIG. 4 is a schematic diagram showing the configuration of an image processing system according to the second embodiment.

[0056] As shown in Fig. 4, the image processing system of this embodiment has the same components as the image processing system of the first embodiment. Fig. 4 further shows a network connector 401, a network 402, a server 403, a storage 404, a cloud 405, and a storage 406 that can be used by the image processing system of this embodiment. The storages 404 and 406 are examples of storages external to the display devices 1a and 1b.

[0057] The display devices 1a and 1b can be connected to a network 402 via a network connector 401. The network connector 401 is, for example, a communication router or a PC. The network 402 is, for example, an on-premise network such as a VPN (Virtual Private Network). The display devices 1a and 1b can access a storage 404 of a server 403 via the network 402. The storage 404 is, for example, configured with an HDD and / or an SSD. FIG. 4 schematically shows virtual model data D1 stored in the storage 404. The virtual model data D1 is data of the virtual model 302. In this embodiment, relative position information indicating the relative position between the marker M and the virtual model 302 is also stored in the storage 404.

[0058] Furthermore, the display devices 1a and 1b can access a storage 406 on a cloud 405 via a network connector 401. The storage 406 is configured, for example, with an HDD and / or an SSD. FIG. 4 schematically shows virtual model data D2 stored in the storage 406. Like the virtual model data D1, the virtual model data D2 is also data of the virtual model 302. In this embodiment, relative position information indicating the relative position between the marker M and the virtual model 302 is also stored in the storage 406.

[0059] In this embodiment, the alignment unit 106a of the display device 1a and the alignment unit 106a of the display device 1b acquire the virtual model data D1 and relative position information from the storage 404, thereby enabling them to acquire the same virtual model data D1 and relative position information. This enables the display device 1a and the display device 1b to perform the above-mentioned alignment based on the same virtual model data D1 and relative position information. In this case, the virtual model data D1 acquired by the display device 1a corresponds to the virtual model data Da in FIG. 3, and the virtual model data D1 acquired by the display device 1b corresponds to the virtual model data Db in FIG. 3.

[0060] In this embodiment, the alignment unit 106a of the display device 1a and the alignment unit 106a of the display device 1b may acquire the same virtual model data D2 and relative position information by acquiring the virtual model data D2 and relative position information from the storage 406. In this case, the display device 1a and the display device 1b can perform the above alignment based on the same virtual model data D2 and relative position information. In this case, the virtual model data D2 acquired by the display device 1a corresponds to the virtual model data Da in FIG. 3, and the virtual model data D2 acquired by the display device 1b corresponds to the virtual model data Db in FIG. 3.

[0061] For example, when a company uses the display devices 1a and 1b, the storages 404 and 406 may be located at the company's headquarters, and the display devices 1a and 1b may be used on-site. This makes it possible to provide the display devices 1a and 1b with virtual model data and relative position information after the display devices 1a and 1b are transported to the site. However, the storages 404 and 406 may be located at a location other than the headquarters, and the display devices 1a and 1b may be used at a location other than the site.

[0062] The display devices 1a and 1b can also communicate with each other via a network connector 401. An example of such communication will be described below.

[0063] In the image processing system of this embodiment, the display device 1a functions as a parent device, and the display device 1b functions as a child device. A user Pa of the parent display device 1a can input operations related to the virtual model 302 to the display device 1a. For example, the user Pa can measure the dimensions of the virtual model 302 displayed on the LCD screen 112, thereby displaying the measurement results on the LCD screen 112. FIG. 4 shows the measured value L of the length of one side of a cube. The user Pa can input operations related to the virtual model 302 to the display device 1a by moving a hand-shaped cursor H on the LCD screen 112. For example, when measuring the length of one side of a cube, the user Pa can measure the length of the side by specifying the start and end points of the side with the cursor H. The user Pa can also input operations to the display device 1a to move or rotate the virtual model 302 within the real space image, thereby moving or rotating the virtual model 302 on the LCD screen 112.

[0064] In this way, the display device 1a, which is the parent device, can change the display content of the virtual model 302 in various ways. The image processing unit 110b of the display device 1a changes the display content of the virtual model 302 by accepting an operation performed by the user Pa. This function of the image processing unit 110b is an example of a display control unit. Furthermore, the output processing unit 110c of the display device 1a transmits operation information related to the operation performed by the user Pa to the display device 1b, which is the child device, via the network connector 401. This function of the output processing unit 110c is an example of an information providing unit.

[0065] Upon receiving operation information from the display device 1a, the image processing unit 110b of the display device 1b changes the display content of the virtual model 302 on the liquid crystal screen 112 of the display device 1b based on the operation information. As a result, when the user Pa measures the length of one side of a cube, the measurement value L is displayed not only on the liquid crystal screen 112 of the display device 1a but also on the liquid crystal screen 112 of the display device 1b. Furthermore, when the user Pa performs an operation to rotate the virtual model 302, the virtual model 302 rotates not only on the liquid crystal screen 112 of the display device 1a but also on the liquid crystal screen 112 of the display device 1b. Therefore, according to this embodiment, changes in the display content of the virtual model 302 can be shared between the display device 1a and the display device 1b, and further, the changes in the display content can be shared in real time. The user Pa is the operator of the virtual model 302, and the user Pb is the viewer of the operation results. The above operation information is an example of predetermined information provided from the parent device to the child device.

[0066] The operation information may be in any format as long as it is information that can identify the content of the operation performed by user Pa and the change in display content that accompanies the operation. For example, operation information regarding an operation to measure the length of one side of a cube can be configured using information regarding the start point of the side, the end point of the side, and the measurement value L.

[0067] FIG. 5 is a block diagram for explaining the operation of the image processing system of the second embodiment.

[0068] As shown in Fig. 5, the display devices 1a and 1b can acquire virtual model data D1 and relative position information from the storage 404 via the network connector 401 (Fig. 4). Furthermore, the display device 1a, which is the parent device, can transmit data D to the display device 1b, which is the child device, via the network connector 401. An example of the data D is the above-mentioned operation information.

[0069] In this case, the data D may be transmitted from the display device 1a to the storage 404 and stored in the storage 404, or the data D stored in the storage 404 may be transmitted to the display device 1b. This makes it possible to store the data D such as operation information together with the virtual model data D1 and the relative position information in the storage 404. In this case, the storage 406 (FIG. 4) may be used instead of the storage 404.

[0070] According to this embodiment, it is possible to share changes to the display content of the virtual model 302 by transmitting operation information, which generally has a small amount of data, from the display device 1a to the display device 1b instead of transmitting virtual model data, which generally has a large amount of data, from the display device 1a to the display device 1b. This makes it possible to reduce the amount of communication between the display devices 1a and 1b. For example, it is considered that the amount of communication when transmitting operation information from the display device 1a to the display device 1b is significantly reduced compared to the amount of communication when transmitting video data of the virtual model 302 from the display device 1a to the display device 1b. Furthermore, according to this embodiment, it is possible to improve security by limiting the data to be communicated between the display devices 1a and 1b.

[0071] In the image processing system of this embodiment, the display device 1a may always function as a parent device, and the display device 1b may always function as a child device, or the display device 1 functioning as a parent device and the display device 1 functioning as a child device may be switchable. Alternatively, the display device 1a may always function as a child device, and the display device 1b may always function as a parent device. In this way, the parent-child relationship between the display devices 1a and 1b may be fixed or variable. Further details of the parent device and the child device will be described later in the third and fourth embodiments.

[0072] (Third embodiment) FIG. 6 is a schematic diagram showing the configuration of an image processing system according to the third embodiment.

[0073] As shown in Fig. 6, the image processing system of this embodiment has the same components as the image processing system of Embodiment 1. Similarly to Fig. 4, Fig. 6 also shows a network connector 401 that can be used by the image processing system of this embodiment.

[0074] In this embodiment, the parent-child relationship between the display devices 1a and 1b is fixed. Therefore, of the display devices 1a and 1b, only the display device 1a functions as a parent device, and of the display devices 1a and 1b, only the display device 1b functions as a child device. A user Pa of the display device 1a operates the virtual model 302, and a user Pb of the display device 1b views the operation results. According to this embodiment, similar to the second embodiment, it is possible for the display devices 1a and 1b to share changes to the display content of the virtual model 302.

[0075] In this embodiment, the virtual model data Da and the relative position information are stored in the data storage unit 107 (FIG. 1) of the display device 1a. Therefore, the alignment unit 106a of the display device 1a acquires the virtual model data Da and the relative position information from the data storage unit 107 of the display device 1a. On the other hand, the virtual model data and the relative position information are not stored in the data storage unit 107 of the display device 1b. Therefore, the alignment unit 106a of the display device 1b acquires the virtual model data Da and the relative position information from the data storage unit 107 of the display device 1a via the network connector 401. This makes it possible to align the display devices 1a and 1b based on the same virtual model data Da and relative position information.

[0076] According to this embodiment, it is possible for the display devices 1a and 1b to share display contents and changes thereto even when the network 402 or the cloud 405 cannot be used or when the storages 404 and 406 cannot be used. On the other hand, according to the second embodiment, it is possible to reduce the amount of communication between the display devices 1a and 1b compared to this embodiment.

[0077] In this embodiment, the display device 1a includes a data storage unit 107 that stores the virtual model data Da and the relative position information, while the display device 1b does not include a data storage unit 107 that stores the virtual model data and the relative position information. To achieve this configuration, the display device 1b of this embodiment may not include the data storage unit 107 itself, or may include a data storage unit 107 that is not used to store the virtual model data and the relative position information.

[0078] (Fourth embodiment) FIG. 7 is a schematic diagram showing the configuration of an image processing system according to the fourth embodiment.

[0079] As shown in Fig. 7, the image processing system of this embodiment has the same components as the image processing system of Embodiment 1. Fig. 7 further shows a network connector 401 that can be used by the image processing system of this embodiment, similar to Figs. 4 and 6.

[0080] In this embodiment, not only can the user Pa of the display device 1a be the operator of the virtual model 302 and the user Pb of the display device 1b be the viewer of the operation results, but also the user Pb of the display device 1b can be the operator of the virtual model 302 and the user Pa of the display device 1a be the viewer of the operation results. This makes it possible to change the display content of the virtual model 302 more freely.

[0081] In this embodiment, the virtual model data Da and the relative position information are stored in the data storage unit 107 (FIG. 1) of the display device 1a. Therefore, the positioning unit 106a of the display device 1a acquires the virtual model data Da and the relative position information from the data storage unit 107 of the display device 1a. Also, in this embodiment, the virtual model data Db and the relative position information are stored in the data storage unit 107 of the display device 1b. Therefore, the positioning unit 106a of the display device 1b acquires the virtual model data Da and the relative position information from the data storage unit 107 of the display device 1b. Virtual model data DBIn this embodiment, the same virtual model data and relative position information are stored in advance in the data storage unit 107 of the display device 1a and the data storage unit 107 of the display device 1b. This makes it possible to align the display devices 1a and 1b based on the same virtual model data and relative position information.

[0082] According to this embodiment, even when the network 402 or the cloud 405 cannot be used or when the storages 404 and 406 cannot be used, it is possible for the display devices 1a and 1b to share display contents and changes thereto. Furthermore, according to this embodiment, it is possible to reduce the amount of communication between the display devices 1a and 1b compared to the third embodiment. This is because, in this embodiment, it is not necessary to provide the virtual model data Da and relative position information from the display device 1a to the display device 1b. In general, when the amount of virtual model data Da is large, it is desirable to adopt this embodiment, and when the amount of virtual model data Da is small, it is desirable to adopt the third embodiment.

[0083] In this embodiment, the parent-child relationship between the display devices 1a and 1b is variable. That is, it is possible to switch between the display device 1 functioning as a parent device and the display device 1 functioning as a child device. This will be described in detail with reference to FIG. 8.

[0084] FIG. 8 is a block diagram for explaining the operation of the image processing system of the fourth embodiment.

[0085] FIG. 8(a) shows an image processing system in which the display device 1a functions as a parent device and the display device 1b functions as a child device. In this embodiment, each of the display devices 1a and 1b includes a switching unit 501 for switching the parent-child relationship. For example, the switching unit 501 of the display device 1a sets the display device 1a as a parent device, and the switching unit 501 of the display device 1b sets the display device 1b as a child device in response to a request from the switching unit 501 of the display device 1a, thereby enabling the display devices 1a and 1b to function as a parent device and a child device, respectively. The parent-child relationship may be switched in response to an operation by the user Pa or the user Pb, or may be automatically switched by the display device 1a or the display device 1b. Alternatively, the switching unit 501 may be provided in only one of the display devices 1a and 1b.

[0086] In Fig. 8(a), display device 1a, which is a parent device, transmits data D such as operation information to display device 1b, which is a child device, via network connector 401. This allows changes in the display content made by user Pa to be reflected not only on liquid crystal screen 112 of display device 1a, but also on liquid crystal screen 112 of display device 1b.

[0087] 8(b) shows an image processing system in which the display device 1b functions as a parent device and the display device 1a functions as a child device. The function of the switching unit 501 of the display devices 1a and 1b is as described above.

[0088] In Fig. 8(b), display device 1b, which is a parent device, transmits data D such as operation information to display device 1a, which is a child device, via network connector 401. This allows changes in the display content made by the operation of user Pb to be reflected not only on liquid crystal screen 112 of display device 1b, but also on liquid crystal screen 112 of display device 1a.

[0089] In each of the second to fourth embodiments, the image processing system may be configured with N display devices 1 (N is an integer equal to or greater than 3). In this case, one display device 1 may function as a parent device, and the remaining N-1 display devices 1 may function as child devices.

[0090] Although several embodiments have been described above, these embodiments are presented by way of example only and are not intended to limit the scope of the invention. The novel devices, methods, and systems described herein may be embodied in various other forms. Furthermore, various omissions, substitutions, and modifications may be made to the forms of the devices, methods, and systems described herein without departing from the spirit of the invention. The appended claims and their equivalents are intended to cover such forms and modifications that fall within the scope and spirit of the invention. [Explanation of symbols]

[0091] 1, 1a, 1b: image processing device (display device), 10: Head mounted video department, 20: Design department, 30: Display department, 102: camera, 104: microphone, 106: input processing unit, 106a: alignment unit, 106b: conversion unit, 107: data storage unit, 108: Recognition processing unit, 108a: Voice recognition processing unit, 108b: image recognition processing unit, 108c: three-dimensional space recognition processing unit, 110: image processing control unit, 110a: control unit, 110b: image processing unit, 110c: output processing unit, 112: liquid crystal screen, 114: speaker, 202: attribute information assigning unit, 204: design processing unit, 206: input operation unit, 301: Real space, 302: Virtual model, 401: Network connector, 402: Network, 403: Server, 404: Storage, 405: Cloud, 406: Storage, 501: Switching unit

Claims

1. An image processing system comprising a plurality of image processing devices, Each of the plurality of image processing devices a positioning unit that aligns a coordinate system of a three-dimensional model space in which virtual objects generated based on design members designed by computer-aided design are arranged and in which marker position coordinates are set, with a coordinate system of a real space, based on position information of the marker obtained by capturing an image of a marker arranged at a position in the real space corresponding to the marker position coordinates, and the marker position coordinates in the three-dimensional model space; an image processing unit that generates an image of the virtual object observed from a viewpoint position of the user in correspondence with a real space image observed from a viewpoint position of the user based on the alignment; a superimposition display unit that displays the real space image and the virtual object in a superimposed manner; Equipped with a first image processing device among the plurality of image processing devices and a second image processing device different from the first image processing device acquire virtual object data that is data of the common virtual object and relative position information that indicates a relative position in a coordinate system of the three-dimensional model space in which the virtual object is to be placed with respect to the marker, and align the coordinate system of the three-dimensional model space with the coordinate system of the real space based on the position information of the marker obtained by each alignment unit capturing an image of the common marker at each position and the marker position coordinates in the three-dimensional model space; At least one of the first image processing device and the second image processing device further a display control unit that changes the display content of the virtual object by the superimposed display unit by receiving an operation performed by the user; an information providing unit that provides operation information relating to the operation performed by the user to other image processing devices; Equipped with the other image processing device changes the display content of the virtual object by its own image processing device based on the provided operation information; Image processing system.

2. The image processing system of claim 1 , wherein each of the first image processing device and the second image processing device is a goggle-type holographic display device.

3. The image processing system according to claim 1 , wherein the first image processing device and the second image processing device acquire the virtual object data and the relative position information stored in a storage outside the first image processing device and the second image processing device.

4. 4. The image processing system according to claim 3, wherein the first image processing device and the second image processing device acquire the virtual object data and the relative position information from the storage via a network, or acquire the virtual object data and the relative position information from the storage on a cloud.

5. The image processing system according to claim 1 , wherein each of the first image processing device and the second image processing device includes a data storage unit that stores the virtual object data and the relative position information.

6. one of the first image processing device and the second image processing device functions as a parent device that outputs the operation information regarding the virtual object; the other of the first image processing device and the second image processing device functions as a slave device that acquires the operation information output by the master device. The image processing system according to claim 1 .

7. only the first image processing device of the first image processing device and the second image processing device functions as the parent device; of the first image processing device and the second image processing device, only the second image processing device functions as the child device; The image processing system according to claim 6 .

8. The image processing system according to claim 6 , wherein the first image processing device and the second image processing device are switchable between functioning as the parent device and functioning as the child device.

9. In each of a plurality of image processing devices that perform image processing, an alignment unit aligns a coordinate system of a three-dimensional model space in which a virtual object generated based on a design member designed by computer-aided design is arranged and in which marker position coordinates are set, with a coordinate system of a real space, based on position information of the marker obtained by capturing an image of a marker arranged at a position in the real space corresponding to the marker position coordinates and the marker position coordinates in the three-dimensional model space; an image processing unit generates an image of the virtual object observed from the viewpoint position of the user, corresponding to a real space image observed from the viewpoint position of the user, based on the alignment; a superimposed display unit that superimposes the real space image and the virtual object; a first image processing device among the plurality of image processing devices and a second image processing device different from the first image processing device acquire virtual object data that is data of the common virtual object and relative position information that indicates a relative position in a coordinate system of the three-dimensional model space in which the virtual object is to be placed with respect to the marker, and align the coordinate system of the three-dimensional model space with the coordinate system of the real space based on the position information of the marker obtained by each alignment unit capturing an image of the common marker at each position and the marker position coordinates in the three-dimensional model space; At least one of the first image processing device and the second image processing device further a display control unit that changes the display content of the virtual object by the superimposed display unit by accepting an operation performed by the user; an information providing unit providing operation information relating to the operation performed by the user to another image processing device; the other image processing device changes the display content of the virtual object by its own image processing device based on the provided operation information; Image processing methods.

Images