Information processing device for processing 3D model, and information processing method

Abstract

To reduce the time and effort of browsing a plurality of 2.5D models while maintaining visibility.SOLUTION: According to the present invention, an information processing device includes acquisition means for acquiring a plurality of 3D models each corresponding to a first subject and obtained by viewing the first subject from different viewpoint positions, instruction means for instructing a position attitude of a viewpoint of appreciating the 3D model, selection means for selecting at least one 3D model from the plurality of 3D models on the basis of the position attitude of the viewpoint instructed by the instruction means, and generation means for generating an image obtained by viewing the 3D model selected by the selection means with the position attitude of the viewpoint.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method for processing a 3D model. [Background technology]

[0002] A 3D model created based on information obtained by photographing an object with an RGBD camera (RGB sensor + depth sensor) such as a stereo camera has 3D information of the object in the shooting direction, but lacks 3D information from behind it. In this specification, such 3D models are called 2.5D models. One way to view a 2.5D model is to open it in a viewer application that can view 3D models, just like a general 3D model, and view the 2.5D model by moving the virtual camera provided in the viewer. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2020-525927 Summary of the Invention [Problem to be solved by the invention]

[0004] Since 2.5D models only have 3D information from the direction in which they were photographed, if you want to view a 3D model of an object from various directions, including from behind, you need to obtain 3D information from multiple directions and create multiple 2.5D models. In this case, it is a hassle to open and view each 2.5D model one by one in a viewer application.

[0005] In addition, when viewing all 2.5D models in a viewer, there is a risk that the visibility of the subject may be impaired. The position and orientation of the 2.5D models usually contain errors. In such cases, the boundaries of the 2.5D models do not match and overlap incompletely, making the visibility of the subject poor.

[0006] Patent Document 1 proposes a method of guiding the user's viewpoint away from the incomplete 3D model area, including the back side of the 2.5D model, so as not to draw the user's attention to the incomplete 3D model area. However, in the above case, good-looking areas and bad-looking areas are mixed in the same field of view, making it difficult to improve visibility by guiding the viewpoint.

[0007] The present invention aims to solve the above problems and reduce the effort required to view multiple 2.5D models while maintaining visibility. [Means for solving the problem]

[0008] In order to solve the above problems, the information processing device of the present invention is characterized by having an acquisition means for acquiring a plurality of 3D models, each corresponding to a first subject and viewing the first subject from different viewpoint positions, an instruction means for indicating a viewpoint position and orientation for viewing the 3D models, a selection means for selecting at least one 3D model from the plurality of 3D models based on the viewpoint position and orientation indicated by the instruction means, and a generation means for generating an image of the 3D model selected by the selection means viewed from the viewpoint position and orientation. Effect of the Invention

[0009] This reduces the hassle of viewing multiple 2.5D models while maintaining visibility of the subject. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 shows a hardware configuration of an information processing device 100 according to an embodiment and a modification of the present invention. [Diagram 2] FIG. 1 is a flowchart showing a 2.5D model display according to a first embodiment. [Diagram 3] A diagram showing the subject's (person's) head [Figure 4] Diagram showing 2.5D model [Diagram 5] FIG. 1 is an image diagram illustrating each model arranged in a 3D space according to the first embodiment; [Figure 6] FIG. 1 shows a viewpoint position and a selected 2.5D model according to a first embodiment; [Figure 7] FIG. 1 shows another viewpoint position and a selected 2.5D model according to the first embodiment. [Figure 8] FIG. 11 is a diagram showing yet another viewpoint position and a selected 2.5D model according to the first embodiment. [Figure 9] FIG. 1 is a diagram showing a limited range of a viewpoint position according to a first embodiment; [Figure 10] A diagram showing the range of possible viewpoint movements for each of the two 2.5D models. [Figure 11] Graph showing the relationship between the position and orientation of the viewpoint and the reliability of the field of view according to the first embodiment. [Figure 12] FIG. 11 is a flowchart showing a 2.5D model display according to the second embodiment. [Figure 13] FIG. 13 is a diagram showing 3D primitives and a viewpoint path according to a second embodiment. [Figure 14] FIG. 11 is a flowchart showing a 2.5D model display according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] In addition, in this embodiment, a 2.5D model refers to a 3D model obtained by integrating the color of a subject seen from a certain viewpoint and the distance from the viewpoint corresponding to that color to the subject, or a video or multiple images generated using the 3D model.

[0013] [First embodiment] The first embodiment described below is an example of an apparatus that executes a 2.5D model processing method on a computer. In this embodiment, a 3D model of a subject (e.g., a human head) obtained by capturing images of the subject from multiple directions, but in which information from some directions is lacking, is called a 2.5D model. In this embodiment, a use case in which a user views a subject from various directions using multiple 2.5D models is described. This embodiment can be executed on a desktop computer, a laptop computer, a portable computer, and the like.

[0014] The configuration of an information processing device 100 for executing a 2.5D model processing method will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the minimum functional configuration for executing the processing of the present invention, and other components are not shown.

[0015] The input unit 10 in FIG. 1 is an interface that receives actions from a user, such as a mouse, keyboard, or joystick, and obtains information necessary for processing a 2.5D model. The information necessary for processing a 2.5D model is information about the position and orientation of a plurality of 2.5D models and a viewpoint. That is, the input unit 10 includes a model obtaining means and a viewpoint position and orientation obtaining means. Here, the viewpoint is a standpoint for viewing a 2.5D model arranged in a 3D space, and has parameters such as position and orientation, angle of view, and resolution, similar to a virtual camera in a viewer application for a 3D model. In addition to these, in this embodiment, information about the position and orientation of the 2.5D model is obtained from the input unit 10 (the input unit 10 includes a 2.5D model position and orientation obtaining means).

[0016] The control unit 11 controls data in the input unit 10, the calculation unit 12, and the storage unit 13, and controls each unit of the entire computer 100. The control unit 11 has a function of acquiring images at the above viewpoints and selecting an appropriate 2.5D model from multiple 2.5D models.

[0017] The calculation unit 12 has a calculation processing device and executes a calculation processing program stored in the storage unit 13 .

[0018] The storage unit 13 is composed of a primary storage device such as a RAM or memory, and a secondary storage device such as an SSD or a hard disk. The storage unit 13 stores information about the 2.5D models and their positions and orientations input via the input unit 10, information about the position and orientation of the viewpoint, etc. The storage unit 13 also stores information about the processing of the 2.5D models processed by the calculation unit 12.

[0019] The control unit 11 has a built-in CPU as a computer, and controls each part of the entire information processing device 100 via a bus based on a computer program stored in a non-volatile memory.

[0020] How to process 2.5D models A method for processing a 2.5D model executed in the computer 100 of this embodiment will be specifically described with reference to the flowchart in Fig. 2. The process corresponding to each step of the flowchart is realized by the control unit 11 reading out a corresponding processing program stored in, for example, a non-volatile memory in the storage unit 13, expanding the program in a volatile memory in the control unit 11, and executing the program, thereby operating each unit.

[0021] In S201, the control unit 11 acquires a plurality of 2.5D models via the input unit 10 and stores them in the storage unit 13. In this embodiment, the data of the plurality of 2.5D models is obtained from the human head 300 in FIG. 3. FIG. 3(A) shows the human head as seen from the front, and FIG. 3(B) shows the human head as seen from above. The calculation unit 12 generates 2.5D models from the four directions of the front, left, right, and back of the head 300, and the control unit 11 acquires them via the input unit 10. The 2.5D models 400, 401, 402, and 403 in FIG. 4(A) show the state of the 2.5D models acquired from each direction as seen from the acquired direction. FIG. 4(B) shows the state of the 2.5D models 400, 401, 402, and 403 in FIG. 4(A) as seen from above. Since the 2.5D models only have information on the shooting direction, the information of the head in the opposite direction of these 2.5D models is missing.

[0022] In S202, the control unit 11 acquires the positions and orientations of the multiple 2.5D models acquired in S201 via the input unit 10, and stores them in the storage unit 13.

[0023] In S203, the control unit 11 arranges a plurality of 2.5D models in the 3D space based on the positions and orientations acquired in S202.

[0024] In S204, the control unit 11 acquires the position and orientation of the viewpoint via the input unit 10 and stores it in the storage unit 13. In this embodiment, when S204 is executed for the first time, the position and orientation of the viewpoint is acquired such that the viewpoint faces the head 300 from the position in front of the person's head 300. Here, the direction of the viewpoint has the same meaning as the shooting direction of the virtual camera in the viewer application, and is the direction from the viewpoint's standpoint toward the subject. This direction is uniquely determined by the orientation of the viewpoint. When S204 is executed again after S207 is executed, the position and orientation of the viewpoint is acquired according to the user's input.

[0025] Fig. 5 shows a 3D space reflecting the information of S202 to S204. 2.5D models 400, 401, 402, and 403 are arranged so as to partially overlap. A viewpoint 500 (showing a virtual camera placed in the sense of the shooting direction) is arranged in front of the 2.5D model 400 and facing the direction of the 2.5D model, as shown in Fig. 5. An arrow 501 indicates the direction of the viewpoint 500.

[0026] In S205, the control unit 11 selects a specific 2.5D model from among multiple 2.5D models present in the field of view, based on the position and orientation of the viewpoint. In this embodiment, as shown in equation (1), the 2.5D model with the smallest cosine between the shooting direction of each 2.5D model and the direction of the viewpoint is selected. The shooting direction of the 2.5D model is acquired from the coordinate system defined in the 2.5D model. The shooting directions of the 2.5D models 401, 402, 403, and 404 are arrows 502, 503, 504, and 505, respectively. Here, d v is the viewpoint direction vector, d m is the shooting direction vector of the 2.5D model m, and · represents the dot product.

[0027]

number

[0028] Fig. 6 is a diagram for explaining the viewpoint position and the 2.5D model to be selected. That is, when the viewpoint 500 is in the position and orientation shown in Fig. 6(A), the 2.5D model 400 that is facing almost directly is selected. Fig. 6(A) does not show the unselected 2.5D models 401, 402, and 403.

[0029] In S206, the control unit 11 acquires an image of the viewpoint. Here, the image is an image that shows a 3D space in which a 2.5D model acquired based on the parameters of the viewpoint is placed. However, among the multiple 2.5D models, only the specific 2.5D model selected in S205 is displayed in the image. FIG. 6(B) shows the image 600. Currently, the 2.5D model 400 is selected, so only the 2.5D model 400 is displayed in the image 600.

[0030] In S207, the control unit 11 displays the image 600 acquired in S206 on an image display unit not shown in FIG.

[0031] In S208, the user selects whether or not to end the image viewing. If the user wishes to end the image viewing, the control unit 11 ends the image display. If the user wishes to not end the image viewing, the process proceeds to S204.

[0032] In S204, control unit 11 acquires the position and orientation of the viewpoint again via input unit 10.

[0033] In S205, the control unit 11 selects a 2.5D model again. Fig. 7 is a diagram for explaining a viewpoint position different from that in Fig. 6 and a selected 2.5D model. Fig. 7(A) shows a state of a 3D space in which the viewpoint and the selected 2.5D model are arranged. Although the position and orientation of the viewpoint have changed, the 2.5D model selected based on formula (1) remains the same, 2.5D model 400.

[0034] Through S206 and S207, the image acquired at viewpoint 500 is displayed on the image display unit. Figure 7(B) shows the image 700. Since the position and orientation of the viewpoint has changed, the displayed image is different from image 600.

[0035] The state when the position and orientation of the viewpoint are acquired again is shown below. Explanations similar to those before will be omitted.

[0036] Fig. 8 is a diagram for explaining a viewpoint position and a selected 2.5D model that are further different from those in Fig. 6 and Fig. 7. Fig. 8(A) shows a viewpoint reflecting a newly acquired position and orientation, and a state of a 3D space in which a selected 2.5D model is placed. The 2.5D model selected according to formula (1) is 403. Image 800 in Fig. 8(B) is an image acquired at this time at viewpoint 500. Only 2.5D model 403 is displayed, and other 2.5D models 400, 401, 402 are not displayed.

[0037] The above describes a method for viewing 2.5D models by controlling (instructing) the position and orientation of the viewpoint when there are multiple 2.5D models of the head. The 2.5D model displayed is switched according to the specified position and orientation of the viewpoint, so that it is possible to save the trouble of opening multiple 2.5D model files one by one. In addition, the position and orientation of the 2.5D model usually includes an error. In such a case, the boundaries between the 2.5D models do not match and overlap incompletely, which reduces the visibility of the subject. To address this issue, by selecting an appropriate 2.5D model according to the viewpoint and displaying only that 2.5D model, the incomplete overlap does not occur, and the visibility of the subject is maintained. On the other hand, in order to display an image according to the position and orientation of the viewpoint specified more accurately, multiple 2.5D models may be synthesized to generate a 2.5D model corresponding to the position and orientation of the viewpoint.

[0038] When the selected 2.5D model is changed, the position and orientation of the viewpoint may be adjusted so that the size of the 2.5D model in successively displayed images is consistent (the same). Specifically, the position and orientation of the viewpoint is adjusted so that the size of the same subject in successive images matches. Existing methods such as block matching and machine learning may be used to detect the same subject. Alternatively, shape information of the 2.5D model may be used for matching, and the position and orientation of the viewpoint may be adjusted so that the outline or silhouette of the 2.5D model is consistent. By performing such an operation, the user feels less uncomfortable when switching between 2.5D models. This effect can be obtained not only when the viewpoint is adjusted, but also when the position and orientation or size of the model is adjusted when changing the 2.5D model.

[0039] In this embodiment, the position and orientation of the viewpoint is acquired from the user's input and can be set arbitrarily, but the position and orientation of the viewpoint may be restricted so that it does not move away from the shooting direction of the 2.5D model. Specifically, first, the orientation is restricted so that the angle between the viewpoint direction and the shooting direction is within a predetermined range so as to satisfy formula (2). Here, t is a threshold value, and the smaller the threshold value, the more the viewpoint direction is restricted so as not to move away from the shooting direction. d v d m <t…(2)

[0040] Next, position restrictions will be described. Fig. 9(A) is a diagram showing the restricted range of the viewpoint position. Half-lines 900 and 901 in Fig. 9(A) are half-lines extending from the center 400 of the model, and satisfy the formula (3). v The dashed line region 902 represents the region surrounded by the half lines 900 and 901, and the position of the viewpoint is restricted to this region. d v d m =t…(3)

[0041] The impact of distance error on the appearance of a 2.5D model becomes greater as the viewpoint direction moves away from the shooting direction. By restricting the viewpoint position and orientation as in formula (2) and the dashed area 902, it becomes difficult to obtain an unattractive image, and therefore it becomes easier to maintain the visibility of the subject.

[0042] When the position and orientation of the viewpoint is restricted as described above, and an instruction to change the position and orientation that exceeds the restriction is received, the control unit 11 may select a 2.5D model according to the movement direction of the viewpoint. FIG. 9(A) shows the viewpoint 500 immediately before the restriction is exceeded, and the 3D space in which the selected 2.5D model 400 is arranged, as viewed from above. Here, the dashed line area 900 indicates the movement range of the viewpoint 500, and the arrow 903 indicates the movement direction of the viewpoint 500. FIG. 9(B) shows the state of the 3D space viewed from above when a new 2.5D model is selected according to the movement direction of the viewpoint. Here, the 2.5D model 403 is selected because it is located in the direction of the movement direction 903 of the viewpoint from the 2.5D model 400. In the 2.5D model 403, restrictions are set on the orientation and movement range of the viewpoint, similar to the 2.5D model 400. The dashed line area 902 indicates the movement range. Here, the viewpoint 500 moves so as not to exceed the movement range limit set for the selected 2.5D model 403, and the orientation also changes so as to face approximately in the shooting direction of the 2.5D model.

[0043] We have explained a method for selecting a 2.5D model according to the viewpoint movement direction when the position and orientation of the viewpoint are restricted. Because the viewpoint direction is unlikely to move away from the shooting direction of each 2.5D model, it is possible to view multiple 2.5D models by manipulating the viewpoint based on user input while maintaining the visibility of each 2.5D model.

[0044] The limited range of the position and orientation of the viewpoint may be determined for each selected 2.5D. Figure 10 shows the movable range of the viewpoint in each of two 2.5D models. Figure 10(A) shows a convex 2.5D model 1000 with respect to the viewpoint, and Figure 10(B) shows a concave 2.5D model 1010.

[0045] For example, determining whether a 2.5D model is convex or concave is performed as follows. The intersection points of each 2.5D model with a line that passes through the center of the 2.5D model and extends in the shooting direction are respectively designated as points 1001 and 1011. The center of the 2.5D model is compared with the positions of points 1001 and 1011. If there is a point in front of the center (toward the bottom of the paper in Figure 10), it can be determined that the model is convex, and vice versa.

[0046] FIG. 10A shows the movable range of the viewpoint in a convex 2.5D model 1000 viewed from above. Points 1002 and 1003 indicate the end points of the 2.5D model 1000 that exist on the same plane as point 1001. A circle passing through points 1001, 1002, and 1003 is a dashed line 1004, and point 1005 indicates the center of the circle 1004. A half line is extended from the center 1005 of the circle to points 1002 and 1003, and the area surrounded by the half line is a dashed line area 1006. This is the movable range of the viewpoint. The dashed line area 1006 is defined by a plane passing through points 1002, 1003, and 1005, but if a plane passing through point 1005 and other end points of the 2.5D model is defined in a similar manner, it is possible to determine the movable range of the viewpoint position in 3D space. The range of poses may be restricted, for example, to point approximately at point 1005 .

[0047] Fig. 10(B) shows the movable range of the viewpoint in a concave 2.5D model 1010 relative to the viewpoint, as viewed from above. Points 1012 and 1013 indicate end points of the 2.5D model 1000 that exist on the same plane as point 1011. Using information around points 1012 and 1013, the slope is calculated by differentiation, and the tangents at each point are calculated. The dashed area 1014 inside each tangent is the movable range of the viewpoint. As in Fig. 10(A), if tangents are defined at the other end points, it is possible to calculate the movable range of the viewpoint position in 3D space.

[0048] The method of determining the range of the position and orientation of the viewpoint for each 2.5D model has been described above. By appropriately determining the range of the position and orientation of the viewpoint for each 2.5D model, it becomes possible to view the subject from various viewpoints while maintaining the visibility of the subject. In addition, in the case of a convex 2.5D model, if the position range is determined in the same manner as for a concave 2.5D model, the direction of the viewpoint can be significantly away from the shooting direction of the 2.5D model. As described above, this problem can be avoided by assuming a circle 1004 along the surface of the convex 2.5D model and determining the limit range based on its center 1005. Conversely, if a concave 2.5D model is limited in the same manner as for a convex 2.5D model, the movement range on the near side (below the paper) becomes narrow, so the visibility of the subject is improved by dividing the processing into concave and convex types.

[0049] In addition, at this time, the position, orientation, and size of the background model around the multiple 2.5D models may be adjusted so that the background displayed in the field of view does not change significantly, that is, the change in the background is within a predetermined range. Here, the background model refers to a 3D model arranged around the multiple 2.5D models or a 3D model arranged to surround the 2.5D models. The latter also includes spheres, cubes, hemispheres, etc. for Image Based Lighting. Since the background in the field of view does not move significantly, the user can view the 2.5D models without significantly impairing the continuity of viewing the same subject.

[0050] Also, a 2.5D model may be selected based on the position and orientation of the viewpoint and the reliability associated with the 2.5D model. The reliability here is a value indicating how reliable the shape information of the 2.5D model is. For example, in a stereo camera, shape information is obtained by a block matching method, and the correlation value obtained at that time may be used as the reliability. Since such reliability is obtained for each area of ​​the 2.5D model, a 2.5D model with a large sum of reliability in the field of view may be selected. Graph 1100 in FIG. 11 is a graph showing the relationship between the position and orientation of the viewpoint and the reliability of the field of view. Graph 1100 shows reliability 1101, 1102, and 1103 of three 2.5D models. In area A, a 2.5D model with reliability 1101 is selected, in area B, a 2.5D model with reliability 1102 is selected, and in area C, a 2.5D model with reliability 1103 is selected.

[0051] Alternatively, the average reliability of each region may be defined as the reliability of the 2.5D model, and the values ​​may be compared to select the 2.5D model with the highest reliability. The former allows the selection of a 2.5D model taking into account the number of 2.5D models in the field of view, while the latter has the advantage of being compatible with real-time processing due to the reduced amount of calculation. By selecting a 2.5D model based on reliability, when there are multiple similar 2.5D models from continuous shooting, it becomes possible to automatically select the 2.5D model with the best shape.

[0052] [Second embodiment] The embodiment described below is an example of an apparatus that executes a 2.5D model processing method on a computer. In this embodiment, a use case is described in which a video is generated using a 2.5D model of a human head obtained from multiple directions. In this embodiment, the 2.5D model means a 3D model. Note that a description of the same configuration and processing as in the first embodiment will be omitted. The present invention can actually be executed on a desktop computer, a laptop computer, a portable computer, and the like.

[0053] Hereinafter, a method for processing a 2.5D model according to this embodiment will be described, but the configuration of the computer that executes this processing method is the same as in the first embodiment, so a description thereof will be omitted.

[0054] A method of processing a 2.5D model executed by a computer in this embodiment will be specifically described with reference to the flowchart in Fig. 12(A) and Fig. 12(B). However, since steps S1201 to S1203 are similar to those in the first embodiment, the description thereof will be omitted.

[0055] In S1204, the control unit 11 acquires a path of the viewpoints and stores it in the storage unit 13. Here, the path is information representing the positions and orientations of a plurality of viewpoints in a set order. As shown in Fig. 12(B), the acquired path 1210 is set so as to surround the head 300.

[0056] In S1205, the control unit 11 acquires the position and orientation of the first viewpoint 1211 from the acquired path 1210. The position of the viewpoint 1211 is in front of the 2.5D model 400, and its orientation is determined to face the direction of the arrow 1213. After S1208 is executed, when S1205 is executed again, the position and orientation of the viewpoint next to the viewpoint previously executed is acquired.

[0057] In S1206, the control unit 11 selects a 2.5D model based on the position and orientation of the viewpoint. The selection method is the same as in the first embodiment, and therefore will not be described. In Fig. 12(B), only the 2.5D model 400 selected at the initial viewpoint 1211 is illustrated.

[0058] In S1207, the control unit 11 acquires an image at the viewpoint acquired in S1205, and stores it in the storage unit 13. Here, the image displays the 2.5D model selected in S1206, similarly to the first embodiment.

[0059] In S1208, control unit 11 determines whether or not the current position and orientation of the viewpoint is at route end point 1212. If the viewpoint is not at route end point 1212, control unit 11 executes S1205 again. If route end point 1212 has been reached, control unit 11 executes S1209.

[0060] In S1209, control unit 11 generates a moving image based on the multiple images stored in storage unit 13, and stores the moving image in storage unit 13.

[0061] The above describes a method for generating a video in which an appropriate 2.5D model is displayed according to the position and orientation of the viewpoint when multiple 2.5D models of the head and the path of the viewpoint are given. Since the 2.5D model displayed is switched according to the position and orientation of the viewpoint, there is no need to select one by one of the multiple 2.5D models to be displayed in each frame of the video. Also, as in the first embodiment, by selecting an appropriate 2.5D model according to the viewpoint and displaying only that 2.5D model in the frame of the video, incomplete overlapping between the 2.5D models does not occur, and the visibility of the subject is maintained.

[0062] In this embodiment, the method of generating the path is not particularly specified, but the order in which the faces of the 3D primitives are specified by user input may be acquired, and the path may be generated based on that order. Here, the 3D primitive means a 3D figure whose faces can be specified, such as the rectangular parallelepiped 1300, the pentagonal prism 1301, and the hexagonal prism 1302 in FIG. 13(A). Regarding the method of generating the path, for example, consider a case in which the faces of the rectangular parallelepiped 1300 are specified in the order of arrows 1303, 1304, and 1305. The center of this rectangular parallelepiped roughly coincides with the centers of multiple 2.5D models. The width and height are within a range in which multiple 2.5D models can fit, and the width and height may be automatically adjusted so that the 2.5D models are located at approximately the same distance from the faces.

[0063] The position and orientation of the first viewpoint (hereinafter, starting point) of the path is determined so that it faces the center of the surface indicated by the arrow 1303 and in the direction of the arrow 1303. However, the position of the viewpoint is set so that multiple 2.5D models arranged inside the rectangular parallelepiped 1300 are captured in the image. The distance from the surface to the viewpoint may be a fixed value, or may be specified arbitrarily by the user. In this case, the user may be able to specify the distance from the surface to the viewpoint by adjusting the length of the arrow 1303. In this manner, the user determines the position and orientation of the starting point. The starting point 1310 is shown in FIG. 13(B). The orientation of the viewpoint 1310 is determined so that it faces the direction of the 2.5D model 400.

[0064] Next, the midpoint and end point of the path are determined. The position and orientation of the viewpoint at the midpoint are determined in the same manner as the position and orientation of the viewpoint was determined for the second specified face, arrow 1304. The same is true for arrow 1305. Here, since arrow 1305 is the last face specified, the position and orientation of the viewpoint determined by arrow 1305 becomes the end point. If the number of specified faces is two or less, there will be no midpoint, and if it is four or more, there will be multiple midpoints. Midpoint 1311 and end point 1312 are shown in FIG. 13(B).

[0065] A route is generated based on the information on the start point, intermediate point, and end point obtained so far. An existing method may be used for generating the route, for example, a method of generating a route by filling each point at a constant interval by linear interpolation, or a smooth route may be generated using a spline function or the like. A route may be generated on the surface of a sphere centered on the center of the rectangular parallelepiped 1300 so that the distance from the center of the rectangular parallelepiped 1300 to each viewpoint is constant. A dashed line 1313 is a route that connects a start point 1310, an intermediate point 1311, and an end point 1312 at equal intervals on the spherical surface. Here, the orientation of each viewpoint is defined to face the center of the rectangular parallelepiped 1300.

[0066] The method of generating a route has been described above. By using 3D primitives, the user can intuitively generate a route. In addition, at this time, each surface of the 3D primitive may correspond to the shooting direction of the 2.5D model. Since the start point, intermediate point, and end point are the shooting direction of the 2.5D model, a viewpoint capable of generating a good-looking image can be included in the route. At this time, the control unit 11 may present an appropriate 3D primitive to the user as a candidate based on the position and orientation of multiple 2.5D models and their shooting directions, or may be set from the beginning as a default 3D primitive. For example, in this embodiment, since 2.5D models are acquired from four directions approximately equally spaced from the front, back, left, and right, 3D primitives such as a cube or a rectangular parallelepiped may be set in advance. This can save the user the trouble of selecting a 3D primitive. In addition, a surface for which a corresponding 2.5D model does not exist may be made unselectable. A viewpoint from which an unattractive image is generated is less likely to be included in the route. Also, at this time, similarly to the first embodiment, the image may be stored only when the relationship between the viewpoint and the 2.5D model satisfies the formula (1). A viewpoint that generates an unattractive image is less likely to be included in the route.

[0067] In this embodiment, the method of acquiring the position and orientation of the 2.5D model is not particularly specified, but the user may input the position and orientation of the 2.5D model using a 3D primitive. In this case, the specification method can be executed in the same manner as the method of specifying the position and orientation of the viewpoint. For example, as in this embodiment, if the user understands that the 2.5D model is acquired from four directions approximately equally spaced from the front, back, left, and right, the user specifies a cube as a 3D primitive and specifies a 2.5D model corresponding to each face. Here, the upper and lower faces that do not have a corresponding 2.5D model are not specified. By using a 3D primitive, the user can intuitively specify the position and orientation of the 2.5D model.

[0068] Alternatively, the position and orientation of the 2.5D model may be acquired by using a measuring device such as an IMU or GPS that acquires the position and orientation when acquiring the 2.5D model. Since it is possible to automatically acquire the position and orientation of the 2.5D model, the burden on the user can be reduced. A similar effect can be obtained when acquiring the positions and orientations of multiple 2.5D models based on data such as multiple acquired 2.5D models and images used to generate the models. This method does not require a measuring device such as an IMU or GPS. A self-location estimation method that captures a large number of images of a subject and estimates the shooting position of the images is known as an existing method, and the position and orientation of the 2.5D model may be estimated using such a method.

[0069] [Third embodiment] The embodiment described below is an example of a device that executes a 2.5D model processing method on a computer. In this embodiment, a use case is described in which an object is viewed by appropriately displaying images of arbitrary viewpoints acquired in advance using a 2.5D model of a human head obtained from multiple directions. That is, in this embodiment, a 2.5D model means multiple images that can be obtained from a 3D model. Note that multiple images may be generated based on a video obtained using a 2.5D model. Explanations of configurations and processes similar to those of the first embodiment are omitted. The present invention can actually be executed on desktop computers, laptop computers, portable computers, and the like.

[0070] Hereinafter, a method for processing a 2.5D model according to this embodiment will be described, but the configuration of the computer that executes this processing method is the same as in the first embodiment, so a description thereof will be omitted.

[0071] A method of processing a 2.5D model executed by a computer in this embodiment will be specifically described with reference to the flowchart in Fig. 14. However, since steps S1401 and S1402 are similar to those in the first embodiment, the description thereof will be omitted.

[0072] In S1403, the control unit 11 acquires images of viewpoints of various positions and orientations for each of the multiple 2.5D models, and stores information about the images and the positions and orientations of the viewpoints at those times in the storage unit 13. The information about the positions and orientations of the viewpoints refers to the positions and orientations of the viewpoints themselves, and information that allows the information to be estimated by calculation. The information that allows the information to be estimated is, for example, the positions and orientations of the viewpoints in the coordinate systems defined for each of the 2.5D models. Since the relative positions and orientations of the multiple 2.5D models are known, even if the coordinate systems are defined for each of the 2.5D models, it is possible to convert the positions and orientations of the viewpoints in the space of a common coordinate system.

[0073] The processes in S1404 and S1405 are similar to those in the first embodiment, and therefore the description thereof will be omitted.

[0074] In S1406, the control unit 11 acquires an image obtained at a position and orientation closest to the current viewpoint position and orientation from the images acquired in S1403 for the selected 2.5D model. The comparison of the positions and orientations uses an existing method such as the Nearest Neighbor method.

[0075] The processes in S1407 and S1408 are similar to those in the first embodiment, and therefore the description thereof will be omitted.

[0076] We have described above how a user can view 2.5D models by controlling the position and orientation of the viewpoint to switch between the images displayed when there is a group of images obtained using each 2.5D model of the head. Because the 2.5D model displayed switches depending on the position and orientation of the viewpoint, there is no need to go through the trouble of opening multiple 2.5D models one by one. In addition, because an appropriate 2.5D model is selected according to the viewpoint and an image generated in advance using that 2.5D model is displayed, incomplete overlap does not occur and visibility of the subject is maintained. Because images saved in advance are viewed, they can be viewed even on computers that have difficulty generating images.

[0077] In this embodiment, the method of storing information about the position and orientation of an image and a viewpoint is not specified, but information about the position and orientation may be stored in image meta information such as Exif, a file name, etc. Since information about the position and orientation is stored in association with an image or a file, information management becomes easier.

[0078] [Other embodiments] The present invention can also be realized by supplying a program for implementing the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0079] The object of the present invention can also be achieved as follows: A storage medium on which software program code describing procedures for implementing the functions of each of the above-described embodiments is recorded is supplied to a system or device. Then, a computer (or a CPU, MPU, etc.) of the system or device reads out and executes the program code stored in the storage medium.

[0080] In this case, the program code itself read out from the storage medium realizes the novel functions of the present invention, and the storage medium storing the program code and the program constitute the present invention.

[0081] Furthermore, examples of storage media for supplying the program code include flexible disks, hard disks, optical disks, magneto-optical disks, etc. Also usable are CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0082] The functions of the above-mentioned embodiments are realized by making the computer executable the program code that has been read out. Furthermore, the functions of the above-mentioned embodiments are also realized by the processing of an operating system (OS) or the like running on the computer, which performs part or all of the actual processing based on the instructions of the program code.

[0083] In addition, the following cases are also included: First, the program code is read from a storage medium and written into a memory in an expansion board inserted into a computer or in an expansion unit connected to the computer, and then, based on the instructions of the program code, a CPU or the like in the expansion board or expansion unit performs all or part of the actual processing. [Explanation of symbols]

[0084] 100 Information processing device 10 Input section 11 Control section 12 Arithmetic section 13 Storage section

Claims

1. an acquisition means for acquiring a plurality of 3D models each corresponding to a first object and obtained by viewing the first object from different viewpoint positions; An instruction means for instructing a position and orientation of a viewpoint for viewing the 3D model; a selection means for selecting at least one 3D model from the plurality of 3D models based on the position and orientation of the viewpoint designated by the designation means; a generating means for generating an image of the 3D model selected by the selecting means viewed from the position and orientation of the viewpoint; 13. An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, further comprising image display means for displaying said image.

3. 2. The information processing apparatus according to claim 1, wherein the instruction means acquires the position and orientation of the viewpoint based on an input from a user.

4. 2. The information processing apparatus according to claim 1, further comprising a generating unit that generates a moving image based on a plurality of said images that are generated based on the positions and orientations of a plurality of viewpoints designated by said designating unit.

5. The information processing apparatus according to claim 1 , wherein the selection means selects a 3D model based on the position and orientation of the viewpoint designated by the designation means and information on the positions and orientations corresponding to the plurality of 3D models.

6. 6. The information processing apparatus according to claim 5, wherein the selection means selects a 3D model having a smallest cosine between a direction of the position and orientation of the viewpoint specified by the specification means and a direction of a position and orientation corresponding to the 3D model.

7. The information processing apparatus according to claim 1 , wherein the selection means selects a 3D model from the plurality of 3D models based on the reliability of the shapes of the plurality of 3D models.

8. The information processing apparatus according to claim 5 , wherein the instruction means selects the position and orientation of the viewpoint and the 3D model by having a user select a face from a 3D primitive having a face corresponding to any one of the plurality of 3D models.

9. The information processing device according to claim 1, characterized in that when the generation means generates the multiple images based on the positions and orientations of the multiple viewpoints indicated by the indication means, the generation means generates the first and second images so that the size of at least a portion of a subject common to the first and second images generated in succession is the same.

10. The information processing device according to claim 1, characterized in that, when a 3D model corresponding to an image to be generated successively changes, the generating means determines a position and orientation of the background model arranged around the 3D model so that a change in the background model is within a predetermined range.

11. 11. The information processing apparatus according to claim 1, wherein the generating means records the position and orientation of the viewpoint designated by the designating means when the image is generated.

12. acquiring a plurality of 3D models each corresponding to a first object and obtained by viewing the first object from different viewpoint positions; An instruction step of instructing a position and orientation of a viewpoint for viewing the 3D model; a selection step of selecting at least one 3D model from the plurality of 3D models based on the position and orientation of the viewpoint designated in the designation step; a generating step of generating an image of the 3D model selected in the selecting step viewed at the position and orientation of the viewpoint; 13. A method for controlling an information processing apparatus comprising the steps of:

13. A computer-executable program in which the steps of the method for controlling an information processing device according to claim 12 are described.

14. A computer-readable storage medium storing a program for causing a computer to execute each step of the method for controlling an information processing device according to claim 12.

Images