Image processing device, image processing method, and program

Abstract

To enable a further efficient relocation of a CG model to a position where a tester can work by using the CG model.SOLUTION: A recommended region deciding unit 250 calculates, in each key frame stored in map data possessed by a storing unit 210, a reliability based on the number of feature points present in each single picked-up image, the dispersion value, and the number of markers, and identifies the key frame that has the highest reliability. Next, a layout correcting unit 260 sets a bounding box based on the shape of a CG model, and updates location information in such a way that the gravity center position of the CG model is projected around the image of the selected key frame.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] In particular, the present invention relates to an image processing apparatus, an image processing method and a program suitable for placing an initially placed CG model in an appropriate location. [Background technology]

[0002] Conventionally, systems that present virtual reality or mixed reality that render a CG model according to the movement of a display device are known. In such systems, a CG model is initially displayed on a reference coordinate system with the settings of the CG model's unique coordinate system that was determined when the CG model was created. However, since the placement position in the unique coordinate system (hereinafter referred to as a local coordinate system) at the time of creating the CG model is determined for convenience at the time of creating the CG model, the CG model is often placed in a location that is different from a location suitable for presentation in virtual reality or mixed reality. Therefore, depending on the position at which the CG model is initially placed, it may take a lot of effort to reposition the CG model.

[0003] To address the issue of the effort required to rearrange a CG model, a function has been provided to adjust the position, orientation, and scale of a CG model so that the entire CG model is displayed within the field of view of a display device at a specific moment. Patent Document 1 discloses a method of determining the position of an object having a flat or curved surface from an environment map obtained by observing real space, and setting and displaying a CG model at that position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2011-203824 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the method described in Patent Document 1, although it is possible to place a CG model at a predetermined location on a plane or curve, there are cases where that location is not suitable for estimating the position and orientation of the camera. In such cases, a phenomenon occurs in which the CG model shakes due to an estimation error in the position and orientation after the CG model is placed. When this phenomenon occurs, the user cannot perform task verification using the CG model, and it takes a lot of effort, such as repositioning the CG model, until the user can stably observe the CG model or perform task verification using the CG model.

[0006] In view of the above-mentioned problems, an object of the present invention is to enable a CG model to be more efficiently relocated to a position where an experiencer can work with the CG model. [Means for solving the problem]

[0007] The image processing device of the present invention is characterized in that it comprises a management means for managing placement information of a CG model to be placed and displayed in a real space captured by an imaging means, an identification means for identifying an area in which an index used to estimate the position and orientation of the imaging means satisfies predetermined conditions, and a correction means for correcting the placement information of the CG model managed by the management means so that the CG model is displayed in the area identified by the identification means. Effect of the Invention

[0008] According to the present invention, it is possible to more efficiently rearrange a CG model to a position where a user can work using the CG model. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of a functional configuration of an image processing apparatus according to a first embodiment; [Diagram 2] 1 is a block diagram illustrating an example of a hardware configuration of an image processing device according to a first embodiment. [Diagram 3]13 is a flowchart showing an example of a processing procedure for displaying an image on which a CG model is superimposed. [Figure 4] 13 is a flowchart showing an example of a detailed processing procedure for correcting the position at which a CG model is arranged. [Diagram 5] FIG. 1 is a schematic diagram showing the relationship between the experiencer and the CG model. [Figure 6] FIG. 13 is a diagram showing an example of a room in which markers and patterns are arranged. [Figure 7] FIG. 13 is a diagram for explaining a state in which feature points are extracted from a room in which markers and patterns are arranged. [Figure 8] FIG. 13 is a diagram for explaining a trajectory along which a camera is moved. [Figure 9] FIG. 13 is a diagram for explaining the positions of key frames with high reliability. [Figure 10] 11A and 11B are diagrams for explaining the degree of dispersion, the number of feature points, and the number of markers for each key frame. [Figure 11] FIG. 13 is a diagram showing an example of a result of correcting the layout of a CG model. [Figure 12] 11 is a diagram for explaining how the distance between a camera and a CG model is calculated. FIG. [Figure 13] FIG. 13 is a diagram showing an example of a confirmation screen displayed on a display unit of the HMD. [Figure 14] FIG. 11 is a block diagram illustrating an example of a functional configuration of an image processing apparatus according to a second embodiment. [Figure 15] FIG. 13 is a diagram showing an example of the result of correcting the layout of a CG model so that the contact mark is at the center. [Figure 16] FIG. 13 is a diagram for explaining the position at which the CG model is initially placed. [Figure 17] 13A to 13C are schematic diagrams for explaining a method of correcting the arrangement of a CG model in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] (First embodiment) Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. In this embodiment, an image processing device that estimates the position and orientation of an image capturing unit based on an image input from the image capturing unit and optimizes the placement of a CG model based on the obtained map information will be described in detail.

[0011] FIG. 2 is a block diagram showing an example of the hardware configuration of the image processing device 100 according to this embodiment. 2, a CPU 901 reads out a control program stored in a ROM 902 and executes various processes. A RAM 907 is used as a temporary storage area such as the main memory and work area of ​​the CPU 901. An external storage device 906 stores various data and programs. A storage medium I / F 905 is an interface for connecting to a removable storage medium such as an SD card. An operation unit 908 accepts various operations by a user.

[0012] The communication I / F 903 is an interface for performing communication processing with an external device via wired or wireless communication. The HMD (Head Mounted Display) 180 displays a composite image, which is a composite image of a captured image and a CG model, on a display unit built into the HMD 180. The imaging unit 103 is configured integrally with the HMD 180 and acquires the captured image. The bus 910 is a bus for connecting these components.

[0013] Note that the functions and processes of the image processing device 100 described below are realized by the CPU 901 reading and executing a program stored in the ROM 902 or the external storage device 906. As another example, the CPU 901 may read a program stored in a storage medium such as an SD card instead of the ROM 902 or the like.

[0014] Fig. 1 is a block diagram showing an example of the functional configuration of the image processing device 100 of this embodiment. Hereinafter, an example will be described in which an experiencer 590 views a composite image of a door part CG model 500 and a tool CG model 510 projected onto an image captured by the imaging unit 103 through a display unit built into the HMD 180, as shown in Fig. 5.

[0015] 1(a), the imaging unit 103 is, for example, a camera that captures a scene, and may be a color camera or a monochrome camera. The imaging unit 103 may also be a stereo camera for presenting a stereo image to the HMD 180. The imaging unit 103 outputs the captured image captured by the imaging unit 103 to the image acquisition unit 105. The image acquisition unit 105 acquires the captured image captured by the imaging unit 103, stores it in the RAM 907, and outputs the acquired captured image to the HMD data management unit 130.

[0016] The HMD data management unit 130 is a module that manages information acquired from the HMD 180 and information to be presented to the HMD 180. Here, the information held by the HMD data management unit 130 includes the captured image acquired from the imaging unit 103, the camera position and orientation of the imaging unit 103, and internal parameters of the imaging unit 103 such as the focal length and the principal point. The HMD data management unit 130 also manages map data required for estimating the position and orientation, and the placement information of markers placed in the scene. Furthermore, the HMD data management unit 130 also manages information such as the position and orientation of a handheld object, geometric information of a CG model (including the vertices, edge information, and texture images of a three-dimensional polygon), and placement information of a CG model (including the position and orientation of each CG model in a reference coordinate system). Note that, if the imaging unit 103 is a stereo camera, the information on the stereo camera may be held as information for each camera.

[0017] Here, map data necessary for estimating the position and orientation will be described. The map data includes, for example, information on data based on past captured images (hereinafter referred to as key frames). Specifically, the map data includes the ID of the key frame, the camera position and orientation associated with the key frame, and information on the captured image associated with the key frame. Furthermore, the map data includes, as information on the captured image of the key frame, information such as the number of feature points extracted from the captured image of the key frame and their coordinates, and the variance value of the feature points on the image of the key frame. Note that, for example, the variance value of the feature points on the image can be calculated by dividing the captured image of the key frame into 10×10 image blocks, dividing the number of areas in the image blocks that have feature points by the total number, and multiplying the result by 100.

[0018] The map data also includes information such as the number of markers captured in the keyframe and the reliability of the keyframe. To detect the markers, a rectangular marker detection process with an identification ID such as ArUco can be used. In other words, this method detects a rectangular area from an image, identifies the bit pattern placed in the rectangular area by homography transformation, and outputs the ID and the image coordinates of the four vertices of the rectangle. The reliability of the keyframe is used as an index that indicates the possibility of a registration error calculated from the number of feature points, variance value, and number of markers. Details of the reliability of the keyframe will be described later.

[0019] Furthermore, the map data contains unique feature point IDs that are assigned by matching extracted feature points between captured images of multiple key frames. The map data also contains the three-dimensional coordinates (X, Y, Z) of feature points that are obtained by triangulating the matched feature points with the same ID based on the camera position and orientation of each key frame, as well as the location information of markers placed in the scene.

[0020] The HMD data management unit 130 extracts and saves feature points at a predetermined distance interval based on the estimated position information from the captured images continuously output from the imaging unit 103, and accumulates them as map data. For example, in a scene in which markers 630 and 635 are placed on the floor of a room, a pattern 610 used for alignment is placed on the wall, and furniture 620 is placed at the back of the room, as shown in Fig. 6, feature points 600 as shown in Fig. 7 are extracted from the captured images. In this case, the Shi-Tomasi method is used, for example, to extract the feature points. That is, the feature points are extracted by searching for intersections where the vertical and horizontal luminance variations are large within a predetermined block of the image.

[0021] Moreover, the reliability of a key frame is an index representing the smallness of error of the estimated value of the camera position and orientation estimated by the position and orientation estimation unit 140 (described later) based on the number of feature points, variance value, and number of markers of the same key frame. For example, if the number of feature points in one captured image is considered to be 200, the reliability of the feature points is calculated by dividing the number of feature points of the key frame by 200. Similarly, a threshold value (e.g., 70) is set for the variance value, and the reliability of the variance value is calculated by dividing the variance value of the key frame by the threshold value. Similarly, the reliability of the marker number is calculated by dividing the number of markers of the key frame by a threshold value (e.g., 1). The reliability of the key frame uses a numerical value obtained by adding up the individual reliability of the feature points, variance value, and number of markers.

[0022] The position and orientation estimation unit 140 estimates the current camera position and orientation of the imaging unit 103 based on the map data stored in the HMD data management unit 130, the acquired captured image, and the arrangement information of the markers. As a method for estimating the camera position and orientation, the position and orientation estimation unit 140 may use, for example, ORBSLAM, which is one of the self-position and orientation estimation methods. Specifically, first, a past captured image (reference image) that contains the most image feature block features contained in the input current captured image is determined. Then, the feature points of the reference image are associated with the feature points of the current captured image, and the current camera position and orientation are estimated from the correspondence between the three-dimensional feature points of the camera position and orientation of the reference image and the feature points of the current captured image.

[0023] The method for estimating the camera position and orientation is not limited to ORBSLAM, and any method may be applied as long as it is a method for estimating the camera position and orientation by creating and referring to map data.

[0024] Here, in a situation where there are few feature points in the captured image or the feature points are biased to only a part of the image, an error may occur in the estimation of the position and orientation, or the position and orientation of the camera may not be able to be estimated. If an error occurs in the estimation of the camera position and orientation, or if the camera position and orientation itself cannot be estimated, the CG drawing unit 170 may shift the position at which the door part CG model 500 is drawn, or the door part CG model 500 may not be able to be drawn. This results in a situation where the user 590 cannot perform the task he or she wants to verify. Therefore, the position and orientation estimation unit 140 has a function of detecting a marker. As a result, even in a situation where there are few feature points or a scene where the feature points are biased, the error in the camera position and orientation can be reduced by integrating the estimated information of the camera position and orientation estimated from the marker.

[0025] The CG drawing unit 170 generates an image by rendering a CG model against a background of a captured image based on the items managed by the HMD data management unit 130. Specifically, the CG drawing unit 170 generates an image by combining a captured image with a CG model based on the captured image, the camera position and orientation, the internal parameters of the imaging unit 103, the geometric information of the CG model, and the arrangement information of the CG model, and performs display control to output the combined image to the HMD 180. Note that if the display unit of the HMD 180 is a stereo display, left and right images may be generated separately and displayed on the corresponding displays.

[0026] The HMD 180 is a display that presents the image generated by the CG rendering unit 170 to the experiencer 590. In this embodiment, an HMD that is worn on the head of the experiencer 590 is illustrated, but the present invention is not limited to displaying on an HMD, and may also be applied to displays of smartphones with cameras, tablets, laptops with cameras, and the like.

[0027] The data management unit 195 performs layout correction and contact determination for the CG model by referring to the arrangement information of the CG model, the geometric information of the CG model, the position and orientation of the hand-held object, and the map data managed by the HMD data management unit 130. Note that the data management unit 195 may also handle other operations such as control of the transformation, display, and erasure of the CG model.

[0028] Next, a detailed configuration of the data management unit 195 will be described. The storage unit 210 holds the placement information of the CG model managed by the HMD data management unit 130, the geometric information of the CG model, the position and orientation of the hand-held object, and map data, and sequentially synchronizes the information updated by the HMD data management unit 130.

[0029] The contact determination unit 230 determines whether the tool CG model 510 held by the experiencer 590 contacts the door section CG model 500 based on the geometric information of the CG model, the arrangement information of the CG model, and the position and orientation of the hand-held object held by the storage unit 210. If contact occurs, the contact determination unit 230 generates a contact mark 520 as shown in FIG. 5 as a new CG model to notify the experiencer 590 of the contact, and updates the arrangement information of the CG model and the geometric information of the CG model held by the storage unit 210. The data updated by the storage unit 210 is sequentially transmitted to the HMD data management unit 130, and the contact mark 520 is displayed on the HMD 180 based on the updated information. Note that, in the present embodiment, an example of performing contact determination has been described, but other determinations may be performed as long as the experiencer 590 visually recognizes the door section CG model 500 and performs verification.

[0030] The layout auditing unit 240 uses the map data, the arrangement information of the CG model, and the geometric information of the CG model stored in the storage unit 210 to determine whether the door CG model 500 is placed at an appropriate position with respect to the user 590. For example, as shown in FIG. 16, in a scene where mixed reality is experienced, the door CG model 500 may not be displayed at a place expected by the user 590, and may be placed near a fixture 620 that the user 590 does not expect. In such a case, the location where the door CG model 500 is placed is not within the field of view of the user 590, and it takes time to search for it. Next, when the user approaches the door CG model 500 placed at the position shown in FIG. 16 to work while visually recognizing the door CG model 500, a captured image with few feature points as shown in FIG. 10(a) is input. This increases the error in the camera position and orientation output by the position and orientation estimation unit 140. Furthermore, even if an attempt is made to verify the contact determination between door section CG model 500 and tool CG model 510 held by participant 590, participant 590's hand collides with fixture 620, making it impossible to verify correctly.

[0031] In order to solve these problems, the layout auditing unit 240 sets conditions for correcting the layout and judges whether or not the layout correction is necessary. Here, if the door part CG model 500 is not reflected in the field of view of the experiencer 590 for one minute after the door part CG model 500 is read out, the condition for correcting the layout is met. In addition, the layout auditing unit 240 identifies a key frame in which the door part CG model 500 is reflected from the position and orientation of each past image held by the map data held by the storage unit 210 and the internal parameters of the imaging unit 103. Then, if the key frame reliability does not exceed 2.00, the condition for correcting the layout is met. Furthermore, if a key frame in which the door part CG model 500 is reflected is identified and a certain number or more of three-dimensional feature points exist in front of the camera viewpoint from the door part CG model 500, the condition for correcting the layout is met.

[0032] The layout audit unit 240 may determine whether correction is necessary by setting conditions individually, or may determine whether correction is necessary by combining the above three judgment conditions. Furthermore, other than the above three judgment conditions, the layout audit unit 240 may determine whether correction is necessary by applying another judgment method for detecting factors that hinder the work of the experiencer 590. When the layout audit unit 240 determines that layout correction is necessary, the layout audit unit 240 specifies the location where the door part CG model 500 should be placed by the recommended area determination unit 250.

[0033] The recommended region determining unit 250 calculates the reliability of the alignment from the map data held by the storage unit 210, selects the key frame with the highest reliability at the current time, and sends the key frame information to the layout correcting unit 260. Here, a process when the imaging unit 103 is moved as shown in FIG. 8 in a scene as shown in FIG. 6 will be described. FIG. 8 is a schematic diagram showing an overview of the scene in FIG. 6 as viewed from above. In FIG. 8, a trajectory 850 indicates the trajectory of the movement of the imaging unit 103. Moreover, positions 811, 813, 815, 817, and 819 on the trajectory 850 of the imaging unit 103 indicate the positions of the key frames.

[0034] The recommended area determining unit 250 calculates the reliability of each key frame in the map data. Here, an example of calculating the reliability of key frames at positions 815, 817, and 819 in Fig. 8 will be described. Fig. 10(a) to Fig. 10(c) show images captured at positions 815, 817, and 819, respectively, and show diagrams in which feature points are superimposed on the captured images in each key frame.

[0035] As described above, the appropriate number of feature points in one captured image is 200 points, the threshold value of the variance is set to 70, and the threshold value for the number of markers is set to 1. In this case, for example, in the case of a key frame at position 815, the reliability of the variance is 30 / 70=0.42, the reliability of the feature points is 21 / 200=0.11, and the reliability of the number of detected markers is 1 / 1=1.00. Therefore, the reliability of the key frame at position 815 is calculated as 0.42+0.11+1.00=1.53. By a similar calculation method, the reliability of the key frame at position 817 is 1.59 (1.14+0.45+0), and the reliability of the key frame at position 819 is 2.33 (0.93+0.40+1.00). In this case, as shown in FIG. 9, the key frame at position 819 is determined to be an appropriate key frame for alignment.

[0036] The recommended region determination unit 250 selects the most reliable keyframe through the above procedure, and sends information on the appropriate keyframe to the layout correction unit 260. Note that the method for calculating the reliability of the keyframe is not particularly limited, and any method may be used to calculate the reliability as long as it is calculated as a measure representing the alignment error of the door section CG model 500. In this embodiment, since the reliability tends to increase as the number of feature points and the variance value increase, information on keyframes with clearly few feature points may be deleted from the map data.

[0037] Layout correction unit 260 arranges door part CG model 500 so that its representative point (for example, the center of gravity position representing the average position of the three-dimensional vertices of the three-dimensional model) overlaps with the center of the image. For example, when the key frame at position 819 is selected by recommended area determination unit 250 as the most reliable key frame, layout correction unit 260 arranges door part CG model 500 in the image shown in Fig. 10(c). At this time, layout correction unit 260 arranges door part CG model 500 so that its representative point overlaps with image center 1100, as shown in Fig. 11.

[0038] However, at this stage, since there is no distance information for the image center 1100 relative to the key frame, the placement location in a three-dimensional position cannot be determined. Therefore, in order to determine the three-dimensional position, the layout correction unit 260 calculates a vector connecting the center position of the image of the appropriate key frame and the camera origin of the key frame. Fig. 12 is a schematic diagram for explaining a method of correcting the placement of the door section CG model 500 from the map point related to the position 819 and the shape information of the door section CG model 500. Fig. 12(a) shows a view of the position 819 from above, and Fig. 12(b) shows a view of the position 819 from the left side.

[0039] The layout correction unit 260 first places the door part CG model 500 with a distance of 30 cm from the position 819 as an initial value. Then, at a position 30 cm away, it is determined whether or not the three-dimensional feature point at the position 819 and the area of ​​the door part CG model 500 intersect. The area of ​​the door part CG model 500 is defined by a bounding box 1220 surrounding the door part CG model 500 as shown in, for example, FIG. 12(a) and FIG. 12(b). If they do not intersect, the distance from the position 819 is set to be farther in stages at intervals of 1 cm, and verification of whether or not the area of ​​the door part CG model 500 intersects with the three-dimensional feature point is repeated. If the door part CG model 500 intersects with the three-dimensional feature point, the previous distance is determined as the appropriate distance 1210, and the corrected position of the three-dimensional position of the door part CG model 500 is determined. The layout correction unit 260 updates the placement information of the door part CG model 500 held in the storage unit 210 with respect to the corrected position of the door part CG model 500, and synchronizes it with the HMD data management unit .

[0040] Next, the details of the processing of this embodiment will be described with reference to the flowchart in Fig. 3. Fig. 3 is a flowchart showing an example of a processing procedure for displaying an image on which a CG model is superimposed in this embodiment. Hereinafter, an example will be described in which a user 590 views a composite image of a door part CG model 500 and a tool CG model 510 projected on an image captured by the imaging unit 103, as shown in Fig. 5.

[0041] First, in step S300, the HMD data management unit reads out the geometric information of the door part CG model 500 and the tool CG model 510 from the storage unit 210, and stores it in the HMD data management unit . Next, in step S320, the image acquisition unit 105 receives the captured image from the imaging unit 103, and the HMD data management unit 130 holds it as the captured image.

[0042] Next, in step S330, the position and orientation estimation unit 140 estimates the camera position and orientation of the imaging unit 103 using the map data and the input captured image. In step S340, CG rendering section 170 renders door section CG model 500 and tool CG model 510 based on the data held in HMD data management section .

[0043] In step S350, the CG rendering unit 170 combines the input photographed image with the door part CG model 500 and the tool CG model 510 rendered in step S340 based on the data held in the HMD data management unit 130. Then, the CG rendering unit 170 outputs the combined image to the HMD 180.

[0044] In step S360, the HMD data management unit 130 determines whether the camera position estimated by the position and orientation estimation unit 140 is a certain distance or more away from the camera position of the key frame stored in the map data. If the camera position is a certain distance or more away, the HMD data management unit 130 adds information to the map data as a new key frame and records it. In addition, the data management unit 195 updates the information stored in the storage unit 210 in accordance with the update of the map data.

[0045] In step S365, the contact determination unit 230 of the data management unit 195 performs contact determination between the door part CG model 500 and the tool CG model 510 based on the data held in the storage unit 210.

[0046] In step S370, data management unit 195 detects whether or not the layout of door-part CG model 500 has been corrected based on the information in storage unit 210, and performs processing to correct the layout of door-part CG model 500 as necessary. The details of the processing will be described later. In step S380, the HMD data management unit 130 determines whether or not there is an end instruction from the experiencer 590. If there is an end instruction, the process ends. If there is no end instruction, the process proceeds to step S320, and the process continues using a new captured image.

[0047] FIG. 4 is a flowchart showing an example of a detailed process procedure of step S370. In step S410, the layout audit unit 240 identifies, from the map data held in the storage unit 210, a key frame in which the door part CG model 500 is reflected.

[0048] In step S415, the layout audit unit 240 judges whether the door part CG model 500 is placed in an appropriate position with respect to the experiencer 590. In this process, as described above, if the reliability of the key frame in which the door part CG model 500 is reflected does not exceed 2.00, it is judged that the door part CG model 500 is not placed in an appropriate position. Similarly, if a certain number or more of three-dimensional feature points exist in front of the camera viewpoint from the door part CG model 500, it is judged that the door part CG model 500 is not placed in an appropriate position. If the result of this judgment is that the door part CG model 500 is placed in an appropriate position, no correction of the layout is required, and the process is terminated. On the other hand, if the door part CG model 500 is not placed in an appropriate position, the process is moved to step S420, since a correction of the layout is required.

[0049] In step S 420 , the recommended region determining unit 250 calculates the reliability of each key frame from the map data held by the storage unit 210 . In step S430, the recommended region determining unit 250 selects the key frame with the highest reliability calculated in step S420.

[0050] In step S440, layout correction unit 260 sets bounding box 1220 based on the shape of door section CG model 500, and adjusts the placement so that the center of gravity of door section CG model 500 is projected onto image center 1100 of the selected keyframe. In step S450, layout correction unit 260 determines the distance from the camera position of the selected key frame to door section CG model 500. Then, layout correction unit 260 updates the placement information of door section CG model 500 held in storage unit 210 with the position of door section CG model 500 after the correction, and synchronizes it with HMD data management unit 130.

[0051] As described above, according to this embodiment, the layout of the CG model is corrected based on a key frame with high reliability suitable for estimating the position and orientation. In this way, the CG model is placed in a place suitable for estimating the position and orientation, so the layout work of the CG model can be completed in a shorter time than before. This allows even an experiencer with little knowledge to perform the verification work in an area with good alignment accuracy, reducing the time required for system construction.

[0052] In this embodiment, the layout correction unit 260 corrects the layout of the door part CG model 500 so that the center of gravity of the door part CG model 500 overlaps with the image center of the key frame, but the location of the layout is not limited to the center of gravity. The location of the door part CG model 500 may be changed according to the workability of the experiencer 590. For example, it is preferable that the location of the contact mark 520 generated by the contact determination unit 230 contacts the tool CG model 510 with a small error when estimating the position and orientation of the door part CG model 500. Also, when dealing with a CG model larger in size than the door part CG model 500, the location to be verified may be significantly shifted from the center of gravity. For these reasons, the location of the CG model may be changed according to the situation so that the experiencer can work at a place with a small alignment error. For example, as shown in FIG. 15, the layout may be corrected so that the contact mark 520 when the contact determination is performed is located at the image center of the key frame.

[0053] Furthermore, in this embodiment, based on the result of correction by the layout correction unit 260, the HMD data management unit 130 immediately synchronizes the position information of the CG model, and the CG drawing unit 170 draws the door section CG model 500 at the corrected three-dimensional position. On the other hand, if the position of the door section CG model 500 is corrected while the experiencer 590 is performing a verification work on the door section CG model 500, the experiencer 590 may be forced to interrupt the work. Therefore, the HMD data management unit 130 may not immediately synchronize the position information of the CG model, and the CG drawing unit 170 may present the experiencer with the timing to correct the position of the door section CG model 500.

[0054] Fig. 13 is a diagram showing an example of a confirmation screen displayed on the display unit of the HMD 180. In the example of Fig. 13, the position of the door part CG model 500 before correction and the position of a second CG model 1310 representing the position after correction are shown. In addition, in the example of Fig. 13, a thumbnail image 1320 in which the corrected door part CG model 500 is composited onto the photographed image of the selected key frame is also presented. Note that the thumbnail image 1320 is placed in the three-dimensional space with reference to the camera position and orientation of the selected key frame.

[0055] Furthermore, a message window 1330 is displayed at the bottom of the screen, presenting the experiencer 590 with the option of selecting whether or not to make correction. The message window 1330 also displays an option inquiring whether or not to make correction at the position of the currently displayed correction candidate. When the experiencer 590 selects the option "Yes" from the operation unit 908, the HMD data management unit 130 synchronizes the positioning information of the CG model, enabling the CG drawing unit 170 to draw the door part CG model 500 at the corrected three-dimensional position.

[0056] 13, an option "Show next candidate" may be presented to indicate that there are other correction candidates. In this case, in step S430, the recommended region determination unit 250 selects multiple keyframes in descending order of the reliability of the keyframes calculated in step S420. Then, in steps S440 and S450, the layout correction unit 260 adjusts the placement of the door part CG model 500 for each keyframe, and the storage unit 210 holds the placement information of each CG model. When the experiencer 590 selects the option "Show next candidate" from the operation unit 908, the screen is switched to one in which the placement position of the CG model is different.

[0057] Second embodiment In the first embodiment, the data management unit 195 is configured to be included in one physical image processing device 100, but it is not necessary to consolidate them into one physical device. For example, the function of displaying the CG model and the function of correcting the layout of the CG model may be separated onto two physically different systems. In this case, the function of correcting the layout of the CG model can be shared when there are multiple users, which is effective.

[0058] In this embodiment, a system for displaying a CG model and a system for performing calculations for correcting the layout of the CG model are constructed separately and connected via a network, and the layout correction calculation process is shared by a plurality of display devices. Fig. 14 is a block diagram showing an example of a functional configuration in this embodiment. Note that blocks with the same numbers as those in Fig. 1 have the same functions, so their explanations will be omitted. Also, in the example shown in Fig. 14, there is one CG model placement processing unit 197 and one client transmission / reception unit 150, but the present invention can also be applied to a case where there are a plurality of these.

[0059] 14, the client transmitting / receiving unit 150 is a block for transmitting data managed by the HMD data management unit 130 to the integrated management unit 190 and for receiving data from the integrated management unit 190 to a CG model placement processing unit 197. The client transmitting / receiving unit 150 also stores the IP address and port number of the integrated management unit 190, and communicates with a connection control unit 193 to transmit successively updated data.

[0060] The connection control unit 193 checks whether the data sent from the client transmitting / receiving unit 150 is acceptable as data to be reflected in the data management unit 195. For example, if there is a connection history within the past 30 minutes, the connection process is omitted and the storage unit 210 of the data management unit 195 is updated, but if this does not apply, the experiencer 590 may be instructed to perform a process to re-establish a connection. Furthermore, the data management unit 195 may perform exclusive control so that different data sent from multiple client transmitting / receiving units 150 are not processed simultaneously in the storage unit 210, and the processing may be performed based on the order registered in the control queue.

[0061] 14, the CG model placement processing unit 197 and the client transmitting / receiving unit 150 are configured within a device carried by the experiencer 590, similar to the image processing device 100 shown in Fig. 5, but the client transmitting / receiving unit 150 may be a separate device. On the other hand, the integrated management unit 190 can be configured within, for example, a general personal computer (PC).

[0062] As described above, according to this embodiment, by controlling the connections of multiple experiencers, the integrated management unit 190 can centrally manage correction information for the placement of CG models presented to multiple experiencers. By centrally managing the correction information for the placement of CG models in this manner, the processing load on the device on the experiencer side can be reduced. In other words, by reducing the processing load, it is possible to suppress a decrease in frame rate even when a complex CG model is rendered by the CG rendering unit 170.

[0063] (Third embodiment) In the first embodiment, a method was exemplified in which map data is generated from an image captured by the imaging unit 103 fixed to the HMD 180 worn by the user 590, and the position of the CG model is corrected based on the map data and marker information arranged in space. On the other hand, if there is a method for identifying an area with high position and orientation estimation accuracy, other methods may be applied. Therefore, in this embodiment, a method for correcting the position of the CG model using a positioning method using an optical sensor will be described. Note that the configuration of the image processing device 100 in this embodiment is basically the same as that of the first embodiment, and the following describes the differences from the first embodiment.

[0064] FIG. 17 is a schematic diagram for explaining a method for correcting the arrangement of a CG model in this embodiment. As shown in FIG. 17, a plurality of camera devices equipped with optical sensors are fixedly arranged near the wall of a room 1730, and these camera devices are connected to a controller device (not shown). Then, the plurality of camera devices can simultaneously capture a ball marker 1720 to estimate the three-dimensional position of the ball marker 1720. Furthermore, the position and orientation of an object to which three or more ball markers 1720 are fixed can be estimated. In this embodiment, three ball markers 1720 are fixed to the imaging unit 103 for synthesizing the door part CG model 500 on the captured image, and the camera position and orientation of the imaging unit 103 are estimated using an optical sensor.

[0065] In this embodiment, the position and orientation estimation unit 140 receives the results of the camera position and orientation of the imaging unit 103 obtained from an optical sensor (multiple camera devices 1710) as a method for estimating the camera position and orientation. Furthermore, the recommended area determination unit 250 sets an area 1780 in which the viewing areas of the multiple camera devices 1710 shown in Fig. 17 overlap as an area with high estimation accuracy of the position and orientation. Furthermore, the layout correction unit 260 arranges the door part CG model 500 so that the center of gravity of the area 1780 overlaps with the center of gravity 1760 of the door part CG model 500.

[0066] (Other embodiments) Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0067] The disclosure of this embodiment includes the following configuration, method, and program.

[0068] (Configuration 1) a management means for managing arrangement information of a CG model to be arranged and displayed in a real space captured by the imaging means; a specifying means for specifying an area in which an index used to estimate the position and orientation of the imaging means satisfies a predetermined condition; a correcting means for correcting placement information of the CG model managed by the managing means so that the CG model is displayed in the area specified by the specifying means; An image processing device comprising: (Configuration 2) An estimation means for estimating a position and orientation of the imaging means; a recording means for recording map data of the real space obtained based on the position and orientation estimated by the estimation means; Equipped with 2. The image processing apparatus according to claim 1, wherein the specifying means specifies an area in which an index based on the map data recorded by the recording means satisfies the predetermined condition.

[0069] (Configuration 3) 3. The image processing device according to configuration 2, wherein the map data includes data based on two or more captured images having different positions and orientations of the imaging means. (Configuration 4) 4. The image processing device according to configuration 2 or 3, wherein the map data includes three-dimensional coordinates of feature points calculated from a plurality of positions and orientations of the imaging means. (Configuration 5) 5. The image processing device according to any one of configurations 2 to 4, wherein the estimation means estimates the position and orientation by combining information on markers arranged in the real space.

[0070] (Configuration 6) The image processing device according to any one of configurations 1 to 5, characterized in that the index is an index based on the number of feature points in an image previously captured by the imaging means, a variance value of the feature points, and the number of markers arranged in the real space. (Configuration 7) The image processing device according to configuration 6, wherein the identification means identifies an area included in a previously captured image captured by the imaging means in the past that has the highest reliability according to the number of feature points, the variance value of the feature points, and the number of markers arranged in the real space. (Configuration 8) 8. The image processing device according to claim 7, wherein the correction means corrects position information of the CG model so that the center of gravity of the CG model coincides with the center of the captured image in which the reliability is highest. (Configuration 9) 9. The image processing device according to any one of configurations 1 to 8, wherein the correction means further corrects position information of the CG model so that an area of ​​the CG model does not overlap a feature point.

[0071] (Configuration 10) The image processing device according to any one of configurations 1 to 9, further comprising a display control means for causing a display means to display an image obtained by combining the CG model whose placement information has been corrected by the correction means with the image captured by the imaging means. (Configuration 11) 11. The image processing device according to configuration 10, wherein the display control means causes the display means to display a confirmation screen for notifying the user that the position information of the CG model is to be changed by the correction means. (Configuration 12) 3. The image processing device according to claim 1, wherein the identifying means identifies an area where the viewing areas of images captured by a plurality of cameras fixed in the real space overlap as an area that satisfies the predetermined condition.

[0072] (method) a management step of managing arrangement information of the CG model to be arranged and displayed in the real space captured by the imaging means; a step of identifying an area in which an index used to estimate the position and orientation of the imaging means satisfies a predetermined condition; a correction step of correcting placement information of the CG model managed in the management step so that the CG model is displayed in the area specified in the specification step; An image processing method comprising:

[0073] (program) a management step of managing arrangement information of the CG model to be arranged and displayed in the real space captured by the imaging means; a step of identifying an area in which an index used to estimate the position and orientation of the imaging means satisfies a predetermined condition; a correction step of correcting placement information of the CG model managed in the management step so that the CG model is displayed in the area specified in the specification step; A program for causing a computer to execute the following. [Explanation of symbols]

[0074] 195 data management unit, 250 recommended area determination unit, 260 layout correction unit

Claims

1. a management means for managing placement information of a CG model to be placed and displayed in a real space captured by the imaging means; a specifying means for specifying an area in which an index used to estimate the position and orientation of the imaging means satisfies a predetermined condition; a correcting means for correcting the placement information of the CG model managed by the managing means so that the CG model is displayed in the area specified by the specifying means; An image processing device comprising:

2. an estimation means for estimating the position and orientation of the imaging means; a recording means for recording map data of the real space obtained based on the position and orientation estimated by the estimation means; Equipped with 2. The image processing apparatus according to claim 1, wherein said specifying means specifies an area in which an index based on the map data recorded by said recording means satisfies said predetermined condition.

3. 3. The image processing apparatus according to claim 2, wherein the map data includes data based on two or more captured images having different positions and orientations of the imaging means.

4. 4. The image processing apparatus according to claim 2, wherein the map data includes three-dimensional coordinates of feature points calculated from a plurality of positions and orientations of the image capturing means.

5. The image processing apparatus according to claim 2 , wherein the estimation means estimates the position and orientation by combining information on markers arranged in the real space.

6. 3. The image processing device according to claim 1, wherein the index is based on the number of feature points in images previously captured by the imaging means, the variance value of the feature points, and the number of markers placed in the real space.

7. The image processing device according to claim 6, characterized in that the identification means identifies an area included in a previously captured image that has the highest reliability according to the number of feature points in the image captured by the imaging means in the past, the variance value of the feature points, and the number of markers arranged in the real space.

8. 8. The image processing apparatus according to claim 7, wherein the correction means corrects the position information of the CG model so that the center of gravity of the CG model coincides with the center of the photographed image that has the highest reliability.

9. 3. The image processing apparatus according to claim 1, wherein the correction means further corrects the layout information of the CG model so that the area of ​​the CG model does not overlap with the feature points.

10. 3. The image processing device according to claim 1, further comprising a display control means for displaying on a display means an image obtained by combining the CG model, the placement information of which has been corrected by the correction means, with the image captured by the imaging means.

11. 11. The image processing apparatus according to claim 10, wherein the display control means causes the display means to display a confirmation screen to inform the user that the correction means will change the position information of the CG model.

12. 2. The image processing apparatus according to claim 1, wherein the specifying means specifies an area where viewable areas of images captured by a plurality of cameras fixed in the real space overlap as the area that satisfies the predetermined condition.

13. a management step of managing placement information of a CG model to be placed and displayed in a real space captured by an imaging means; a specifying step of specifying an area in which an index used to estimate the position and orientation of the imaging means satisfies a predetermined condition; a correcting step of correcting the placement information of the CG model managed in the managing step so that the CG model is displayed in the area identified in the identifying step; An image processing method comprising:

14. a management step of managing placement information of a CG model to be placed and displayed in a real space captured by an imaging means; a specifying step of specifying an area in which an index used to estimate the position and orientation of the imaging means satisfies a predetermined condition; a correcting step of correcting the placement information of the CG model managed in the managing step so that the CG model is displayed in the area identified in the identifying step; A program that causes a computer to execute the following.

Images