Information processing apparatus for processing three-dimensional model, control method thereof, and storage medium

Abstract

The invention relates to an information processing apparatus for processing a three-dimensional model, a control method thereof, and a storage medium. The invention aims to reduce the time and effort to view a plurality of 2.5-dimensional (D) models while maintaining visibility. The image processing apparatus, the image processing apparatus comprising: one or more memories for storing instructions; one or more processors in communication with the one or more memories and, when executing the instructions, cooperate with the one or more memories to: acquire a plurality of three-dimensional (3D) models, each corresponding to a first subject and viewing the first subject from a different viewpoint position; specifying a position and orientation of a viewpoint for viewing the 3D model; selecting at least one 3D model from the plurality of 3D models based on the position and orientation of the viewpoint; and generating an image viewing the at least one 3D model from the position and orientation of the viewpoint.

Description

Technical Field

[0001] The present invention relates to an information processing device for processing a three-dimensional (3D) model, a control method thereof and a storage medium. Background Art

[0002] A three-dimensional (3D) model generated based on information obtained by capturing an image of a subject using a red-green-blue-depth (RGBD) camera (RGB sensor + depth sensor) such as a stereo camera includes 3D information of the subject in the shooting direction but lacks 3D information from behind the subject. In this specification, such a 3D model is referred to as a 2.5D model. As a method for viewing a 2.5D model, the 2.5D model is opened using a viewer application that can view 3D models, and the 2.5D model is viewed by moving a virtual camera provided in the viewer in the same manner as viewing a general 3D model. Since the 2.5D model only has 3D information from the shooting direction, if the user wants to view the 3D model of the subject from various directions including the back, the user needs to obtain 3D information from various directions and generate multiple 2.5D models. At this time, it is very time-consuming to open and view the 2.5D models one by one in the viewer application.

[0003] Furthermore, if all 2.5D models are opened and viewed in the viewer, there is a risk that the visibility of the subject is impaired. The position and orientation of the 2.5D models often contain errors. In this case, the boundaries between the 2.5D models do not match and do not completely overlap, resulting in poor visibility of the subject.

[0004] According to Japanese Unexamined Patent Application Publication No. 2020-525927, a method is discussed in which the user's viewpoint is guided in a direction away from an area including an incomplete 3D model behind a 2.5D model to divert the user's attention from the area. However, in the above case, good-looking areas and bad-looking areas are mixed in the same field of view, making it difficult to improve visibility by guiding the viewpoint. Summary of the invention

[0005] The present invention aims to solve the above-mentioned problems while maintaining visibility while reducing the time and effort of viewing multiple 2.5D models.

[0006] According to one aspect of the present invention, an information processing device is provided, comprising: one or more memories for storing instructions; one or more processors, the processors communicating with the one or more memories and, when executing the instructions, cooperating with the one or more memories, so as to: acquire multiple three-dimensional (3D) models, each 3D model corresponding to a first subject and viewing the first subject from a different viewpoint position; specify a position and orientation of a viewpoint for viewing the 3D model; select at least one 3D model from the multiple 3D models based on the position and orientation of the viewpoint; and generate an image of the at least one 3D model viewed from the position and orientation of the viewpoint.

[0007] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 The hardware configuration of the image processing apparatus according to the exemplary embodiment and the modification of the present invention is illustrated.

[0009] Figure 2 is a flowchart for displaying a 2.5-dimensional (D) model according to the first exemplary embodiment.

[0010] Figure 3A and Figure 3B The figure shows the head of a subject (person).

[0011] Figure 4A and Figure 4B A 2.5D model is illustrated.

[0012] Figure 5 is an image view illustrating each model arranged in a three-dimensional (3D) space according to the first exemplary embodiment.

[0013] Fig. 6A and Figure 6B The viewpoint position and the selected 2.5D model according to the first exemplary embodiment are illustrated.

[0014] Fig. 7A and Figure 7B Another viewpoint position and a selected 2.5D model according to the first exemplary embodiment are illustrated.

[0015] Fig. 8A and Figure 8B Still another viewpoint position and a selected 2.5D model according to the first exemplary embodiment are illustrated.

[0016] Fig.9A and Fig. 9B The limitation range of the viewpoint position according to the first exemplary embodiment is illustrated.

[0017] Fig. 10A and Fig. 10B The movable range of the viewpoint in each of the two 2.5D models is illustrated.

[0018] Fig.11 is a graph illustrating the relationship between the position and orientation of the viewpoint and the reliability of the field of view according to the first exemplary embodiment.

[0019] Fig. 12A and Fig. 12B are a flowchart and views for displaying a 2.5D model according to the second exemplary embodiment.

[0020] Fig.13A and Fig. 13B 3D primitives and viewpoint routes according to the second exemplary embodiment are illustrated.

[0021] Fig.14 is a flowchart for displaying a 2.5D model according to the third exemplary embodiment. DETAILED DESCRIPTION

[0022] The exemplary embodiments will be described in detail below with reference to the accompanying drawings. The following exemplary embodiments are not intended to limit the scope of the present invention as contained in the appended claims. A plurality of features are described in the exemplary embodiments, but not all of these features are essential to the present invention, and the plurality of features may be combined arbitrarily. In addition, the same or similar components in the drawings are represented by the same reference numerals, and redundant descriptions will be omitted.

[0023] According to this exemplary embodiment, a 2.5-dimensional (D) model refers to a three-dimensional (3D) model obtained by integrating the color of an object viewed from a certain viewpoint and the distance to the object from the viewpoint corresponding to the color, or a video or multiple images generated using the 3D model.

[0024] The first exemplary embodiment described below is an example of an apparatus for executing a 2.5D model processing method by a computer. According to this exemplary embodiment, a 3D model of a subject (e.g., a person's head) obtained by taking images of the subject from multiple directions but lacking information from certain directions is referred to as a 2.5D model. According to this exemplary embodiment, a use case is described in which a user uses multiple 2.5D models to view a subject from various directions. This exemplary embodiment can be executed on a desktop computer, a laptop computer, a portable computer, etc.

[0025] Will refer to Figure 1 The configuration of the information processing apparatus 100 that executes the 2.5D model processing method is described. Figure 1It is a block diagram illustrating a minimum functional configuration for executing the process of the present invention, and other components are not shown.

[0026] Figure 1 The input unit 10 in is an interface that receives actions from the user and obtains information required for 2.5D model processing, such as a mouse, keyboard, or joystick. The information required for 2.5D model processing is information related to multiple 2.5D models and the position and orientation of viewpoints. In other words, the input unit 10 includes a model acquisition unit and a viewpoint position and orientation acquisition unit. Here, the viewpoint is a foothold for viewing a 2.5D model placed in a 3D space, and is similar to a virtual camera in a 3D model viewer application, with parameters such as position, orientation, viewing angle, and resolution. According to this exemplary embodiment, in addition to the above information, information related to the position and orientation of the 2.5D model is also obtained from the input unit 10 (the input unit 10 includes a 2.5D model position and orientation acquisition unit).

[0027] The control unit 11 controls the input unit 10, the calculation unit 12 and the storage unit 13 and data of each unit in the entire information processing device 100. The control unit 11 has a function of acquiring an image at the above viewpoint and a function of selecting an appropriate 2.5D model from a plurality of 2.5D models.

[0028] The calculation unit 12 includes a calculation processing device and executes a calculation processing program stored in the storage unit 13 .

[0029] The storage unit 13 is configured with a main storage device such as a memory (e.g., a random access memory (RAM)) and a memory, and an auxiliary storage device such as a solid state drive (SSD) and a hard disk. The storage unit 13 stores the 2.5D model input via the input unit 10, information about its position and orientation, information about the position and orientation of the viewpoint, etc. The storage unit 13 also stores information related to the processing of the 2.5D model processed by the calculation unit 12.

[0030] The control unit 11 includes a built-in central processing unit (CPU) as a computer, and controls each unit in the entire information processing apparatus 100 via a bus based on a computer program stored in a nonvolatile memory.

[0031] <<2.5D model processing method>>

[0032] Will refer to Figure 2The flowchart in specifically describes the 2.5D model processing method performed in the information processing device 100 according to the present exemplary embodiment. The processing corresponding to each step in the flowchart is implemented in the following manner: for example, the control unit 11 reads out the corresponding processing program stored in the non-volatile memory in the storage unit 13, loads the read program into the volatile memory in the control unit 11, and executes the program to make each unit operate.

[0033] In step S201, the control unit 11 acquires a plurality of 2.5D models via the input unit 10, and stores the acquired 2.5D models in the storage unit 13. Figure 3A and Figure 3B The data of multiple 2.5D models of the person's head 300 are obtained. Figure 3A and Figure 3B The head 300 of a person viewed from the front and from the top is illustrated respectively. The calculation unit 12 generates a 2.5D model of the head 300 from four directions: front, left, right, and back, and the control unit 11 acquires the generated 2.5D model of the head 300 via the input unit 10 . Figure 4A A state in which 2.5D models 400 , 401 , 402 , and 403 respectively indicating the 2.5D model acquired from each direction are viewed from the acquired direction is illustrated. Figure 4B Illustration of a view from above Figure 4A Since the 2.5D models only include information related to the camera direction, these 2.5D models lack information related to the head in the opposite direction.

[0034] In step S202 , the control unit 11 acquires the positions and orientations of the plurality of 2.5D models acquired in step S201 via the input unit 10 , and stores the acquired positions and orientations in the storage unit 13 .

[0035] In step S203 , the control unit 11 arranges a plurality of 2.5D models in the 3D space based on the positions and orientations acquired in step S202 .

[0036] In step S204, the control unit 11 acquires the position and orientation of the viewpoint via the input unit 10, and stores the acquired position and orientation in the storage unit 13. According to the present exemplary embodiment, if the processing in step S204 is performed for the first time, the position and orientation of the viewpoint facing the face of the head 300 are acquired from the position in front of the head 300 of the person. Here, the viewpoint direction has a meaning similar to the imaging direction of the virtual camera in the viewer application and is a direction from the foothold of the viewpoint toward the subject. The viewpoint direction is uniquely determined based on the orientation of the viewpoint. In the case where the processing in step S207 is performed and then the processing in step S204 is performed again, the position and orientation of the viewpoint are acquired in response to the user input.

[0037] Figure 5 The 3D space reflecting the information from step S202 to step S204 is illustrated. The 2.5D models 400, 401, 402, and 403 are arranged to partially overlap each other. Figure 5 As shown in , a viewpoint 500 (illustrated as a virtual camera in the sense of a camera direction) is placed in front of the 2.5D model 400 to face the direction of the 2.5D model 400. An arrow 501 indicates the direction of the viewpoint 500.

[0038] In step S205, the control unit 11 selects a specific 2.5D model from a plurality of 2.5D models present in the field of view based on the position and orientation of the viewpoint. According to the present exemplary embodiment, as shown in expression (1), a 2.5D model having the smallest cosine between the imaging direction of each 2.5D model and the viewpoint direction is selected. The imaging direction of the 2.5D model is acquired from the coordinate system defined in the 2.5D model. The imaging directions of the 2.5D models 400, 401, 402, and 403 are indicated by arrows 502, 503, 504, and 505, respectively. In expression (1), d v is the direction vector of the viewpoint, d m is the camera direction vector of the 2.5D model m, and · denotes the inner product.

[0039]

[0040] Fig. 6A and Figure 6B The viewpoint position and the selected 2.5D model are illustrated. In other words, if the viewpoint 500 is at Fig. 6A , then select the 2.5D model 400 that is almost directly facing it. Fig. 6A In the figure, the unselected 2.5D models 401, 402 and 403 are not shown.

[0041] In step S206, the control unit 11 acquires an image from the viewpoint. The image referred to here is an image depicting a 3D space in which 2.5D models acquired based on the viewpoint parameters are arranged. However, only the specific 2.5D model selected from the plurality of 2.5D models in step S205 is displayed in the image. Figure 6B The image 600 in FIG. 6 is the image. Since the 2.5D model 400 is currently selected, only the 2.5D model 400 is displayed in the image 600 .

[0042] In step S207, the control unit 11 Figure 1 The image 600 acquired in step S206 is displayed on an image display unit not shown in the figure.

[0043] In step S208, the user determines whether to complete viewing of the image. If viewing is completed ("Yes" in step S208), the control unit 11 terminates display of the image. If viewing is not completed ("No" in step S208), the process proceeds to step S204.

[0044] In step S204 , the control unit 11 acquires the position and orientation of the viewpoint again via the input unit 10 .

[0045] In step S205 , the control unit 11 selects the 2.5D model again. Fig. 7A and Figure 7B The diagram shows Fig. 6A and Figure 6B The viewpoint position and the selected 2.5D model in different viewpoint positions and the selected 2.5D model. Fig. 7A The viewpoint and the 3D space in which the selected 2.5D model is arranged are illustrated. The position and orientation of the viewpoint are changed, but the 2.5D model selected based on the expression (1) is still the 2.5D model 400 .

[0046] In steps S206 and S207 , the image acquired at the viewpoint 500 is displayed on the image display unit. Figure 7B The image 700 in FIG. 7 is the image. The position and orientation of the viewpoint have changed, so that the displayed image 700 is different from the image 600.

[0047] The case where the position and orientation of the viewpoint are acquired again will be described below. Descriptions similar to the above description will be omitted.

[0048] Fig. 8A and Figure 8B The diagram further illustrates Fig. 6A and Figure 6B as well as Fig. 7A and Figure 7B The viewpoint position and the selected 2.5D model in different viewpoint positions and the selected 2.5D model. Fig. 8AThe viewpoint reflecting the newly acquired position and orientation and the 3D space in which the selected 2.5D model is arranged are illustrated. The 2.5D model 403 is selected according to expression (1). Figure 8B The image 800 in FIG. 8 is an image acquired at the viewpoint 500 at this time. Only the 2.5D model 403 is displayed, and the other 2.5D models 400, 401, and 402 are not displayed.

[0049] The above describes a method for allowing a user to view a 2.5D model by controlling (specifying) the position and orientation of a viewpoint in the presence of multiple head 2.5D models. Since the 2.5D model displayed is switched corresponding to the position and orientation of the specified viewpoint, the time and effort of opening the files of multiple 2.5D models one by one can be saved. The position and orientation of the 2.5D model usually include errors. In this case, the boundaries between the 2.5D models do not match and do not completely overlap, resulting in poor visibility of the subject. To solve this problem, a suitable 2.5D model is selected according to the viewpoint and only the selected 2.5D model is displayed, so that incomplete overlap is not generated and the visibility of the subject can be maintained. On the other hand, in order to more accurately display an image corresponding to the position and orientation of the specified viewpoint, multiple 2.5D models can be combined to generate a 2.5D model corresponding to the position and orientation of the viewpoint.

[0050] In the case where the selected 2.5D model changes, the position and orientation of the viewpoint can be adjusted so that the size of the 2.5D model appearing in the continuously displayed image is constant (the same). Specifically, the position and orientation of the viewpoint are adjusted so that the sizes of the same subject displayed successively in the image match each other. Existing methods such as block matching and machine learning can be used to detect the same subject. In addition, the shape information of the 2.5D model can be used for matching, and the position and orientation of the viewpoint can be adjusted so that the outline or silhouette of the 2.5D model is constant. Operating as described above reduces the user's sense of disobedience when switching 2.5D models. Similarly, this effect can be achieved not only by adjusting the viewpoint, but also by adjusting the position, orientation or size of the 2.5D model when the 2.5D model changes.

[0051] According to the present exemplary embodiment, the position and orientation of the viewpoint are obtained from the user input and can be set arbitrarily, but the position and orientation of the viewpoint can also be limited so as not to be away from the camera direction of the 2.5D model. Specifically, first, the orientation is limited so that the angle between the viewpoint direction and the camera direction is within a certain range to satisfy expression (2). In expression (2), t is a threshold value, and the smaller t is, the more the viewpoint direction is limited to be away from the camera direction.

[0052] d v ·d m < t ... (2)

[0053] Next, the restriction on the position will be described. Fig.9A The diagram shows the limits of the viewpoint's position. Fig.9A The half lines 900 and 901 in FIG. 4 extend from the center of the 2.5D model 400 and are aligned with the d satisfying equation (3). v The dotted area 902 represents the area bounded by the half lines 900 and 901, and the position of the viewpoint is limited to this area.

[0054] d v ·d m = t ... (3)

[0055] The influence of the distance error of the 2.5D model on its appearance increases as the viewpoint direction moves away from the camera direction. If the position and orientation of the viewpoint are restricted as in expression (2) and the dashed area 902, it is less likely to obtain an ugly image, and the visibility of the subject can be maintained.

[0056] In a state where the position and orientation of the viewpoint are restricted as described above, if it is indicated that the position and orientation of the viewpoint change in a manner exceeding the restriction, the control unit 11 may select a 2.5D model according to the moving direction of the viewpoint. Fig.9A A state is illustrated in which the viewpoint 500 is just before exceeding the limit and views from above the 3D space in which the selected 2.5D model 400 is arranged. Here, a dotted line area 902 indicates a moving range of the viewpoint 500 , and an arrow 903 indicates a moving direction of the viewpoint 500 . Fig. 9B The diagram shows a state in which a 3D space in which a 2.5D model 403 is newly selected according to the moving direction of the viewpoint is viewed from above. Here, since the 2.5D model 403 is located in the direction of the viewpoint 500 from the moving direction 903 of the 2.5D model 400, the 2.5D model 403 is selected. As with the 2.5D model 400, the restrictions on the orientation and the moving range of the viewpoint are set for the 2.5D model 403. The dotted line area 902 indicates the moving range of the viewpoint 500. The viewpoint 500 moves in a manner not exceeding the restrictions on the moving range set for the selected 2.5D model 403, and the orientation also changes to a camera direction that is substantially facing the 2.5D model 403.

[0057] The above describes a method for selecting a 2.5D model according to the moving direction of a viewpoint when the position and orientation of the viewpoint are restricted. Since the viewpoint direction is unlikely to be far away from the camera direction of each 2.5D model, multiple 2.5D models can be viewed by operating the viewpoint based on user input while maintaining the visibility of each 2.5D model.

[0058] A limit range of viewpoint position and orientation may be determined for each selected 2.5D model. Fig. 10A and Fig. 10B The diagram shows the movable range of the viewpoints in the two 2.5D models. Fig. 10A The 2.5D model 1000 is shown protruded relative to the viewpoint, and Fig. 10B The 2.5D model 1010 is illustrated as being recessed relative to the viewpoint.

[0059] For example, whether the 2.5D model is convex or concave is determined as follows. The intersection points of the straight line passing through the center of the 2.5D model and extending in the imaging direction and the 2.5D models 1000 and 1010 are designated as points 1001 and 1011, respectively. The centers of the 2.5D models 1000 and 1010 are compared with the positions of the points 1001 and 1011, respectively. If the point is located in front of the center ( Fig. 10A and Fig. 10B If the image is on the lower side of the page, it can be determined as convex, otherwise it can be determined as concave.

[0060] Fig. 10A The state of the movable range of the viewpoint in the 2.5D model 1000 protruding relative to the viewpoint is illustrated as viewed from above. Points 1002 and 1003 indicate end points existing on the same surface as the point 1001 in the 2.5D model 1000. Alternating long and short dashed lines indicate a circle 1004 passing through the points 1001, 1002, and 1003, and a point 1005 indicates the center of the circle 1004.

[0061] The area bounded by the half straight line extending from the center 1005 of the circle 1004 to the points 1002 and 1003 is a dotted area 1006. This is the movable range of the viewpoint. The dotted area 1006 is defined by a plane passing through the points 1002, 1003, and 1005, but the movable range of the viewpoint position can be set in the 3D space by setting a plane passing through the point 1005 and other end points of the 2.5D model using a similar method. For example, the range of the orientation can be limited to approximately facing the point 1005.

[0062] Fig. 10B The diagram illustrates a state in which the movable range of the viewpoint in the 2.5D model 1010 that is recessed relative to the viewpoint is viewed from above. Points 1012 and 1013 indicate end points in the 2.5D model 1010 that exist on the same surface as the point 1011. The slope is calculated by differentiating using information around the points 1012 and 1013, and the tangent line of each point is calculated. The dotted area 1014 inside each tangent line is the movable range of the viewpoint. Fig. 10A Similarly, the movable range of the viewpoint position in the 3D space can be obtained by setting tangent lines at other end points.

[0063] The method of setting the range of the position and orientation of the viewpoint for each 2.5D model is described above. The range of the position and orientation of the viewpoint is appropriately set for each 2.5D model so that the subject can be viewed from various viewpoints while maintaining the visibility of the subject. In the case of a protruding 2.5D model, if the range of the position is determined using a method similar to that of a recessed 2.5D model, the viewpoint direction may be far away from the camera direction of the 2.5D model. This problem can be avoided by assuming that the circle 1004 is along the surface of the protruding 2.5D model and setting a restricted range based on the center 1005 as described above. On the contrary, if the recessed 2.5D model is restricted using a method similar to that of the protruding 2.5D model, the range of movement in the front (lower side of the page) becomes narrower, and thus the visibility of the subject can be improved by separately processing the recessed 2.5D model and the protruding 2.5D model.

[0064] At this time, the position, orientation and size of the background model surrounding the multiple 2.5D models can be adjusted so that the background displayed in the field of view does not change significantly, in other words, the change of the background is within a predetermined range. The background model refers to a 3D model arranged around the multiple 2.5D models or a 3D model arranged to surround the 2.5D model. The latter includes spheres, cubes and hemispheres for image-based lighting. Since the background in the field of view does not move significantly, the user can view the 2.5D model without significantly losing the sense of continuity of viewing the same subject.

[0065] In addition, a 2.5D model can be selected based on the position and orientation of the viewpoint and the credibility associated with the 2.5D model. The credibility is a value indicating the credibility of the shape information of the 2.5D model. For example, a stereo camera uses a block matching method to obtain shape information, and the relevant value obtained at this time can be used as the credibility. The credibility is calculated for each area of ​​the 2.5D model, so that the 2.5D model with the largest total credibility in the field of view can be selected. Fig.11 The graph 1100 in FIG. 1 illustrates the relationship between the position and orientation of the viewpoint and the credibility of the field of view. The graph 1100 indicates the credibility 1101, 1102 and 1103 of three 2.5D models. The 2.5D model with the credibility 1101, the 2.5D model with the credibility 1102 and the 2.5D model with the credibility 1103 are selected in regions A, B and C respectively.

[0066] Furthermore, the average value of the credibility of each region can be defined as the credibility of the 2.5D model, and the average values ​​can be compared to select the 2.5D model with the highest credibility. The former has the advantage that the 2.5D model can be selected by considering the number of 2.5D models captured in the field of view, while the latter has the advantage that a small amount of calculation is required and it is compatible with real-time processing. The 2.5D model is selected based on the credibility, so that when multiple similar 2.5D models are captured in continuous video, the 2.5D model with the best shape can be automatically selected.

[0067] The second exemplary embodiment described below is an example of an apparatus for executing a 2.5D model processing method by a computer. According to this exemplary embodiment, a use case of generating a motion image using a 2.5D model of a person's head acquired from multiple angles is described. According to this exemplary embodiment, the 2.5D model refers to a 3D model. A description of a configuration and processing similar to that according to the first exemplary embodiment will be omitted. Some embodiments may be executed on a desktop computer, a notebook computer, a portable computer, etc.

[0068] A 2.5D model processing method according to the present exemplary embodiment will be described below, but the configuration of a computer that executes the present processing method is similar to that of the computer according to the first exemplary embodiment, and thus a description thereof will be omitted.

[0069] Will refer to Fig. 12A and Fig. 12B The flowchart in specifically describes the 2.5D model processing method executed by a computer according to the present exemplary embodiment. However, steps S1201 to S1203 are similar to those according to the first exemplary embodiment, and thus description thereof is omitted.

[0070] In step S1204, the control unit 11 acquires the route of the viewpoints and stores the acquired route in the storage unit 13. Here, the route is information indicating the positions and orientations of a plurality of viewpoints in a set order. Fig. 12B As shown in , the acquired route 1210 is set around the head 300.

[0071] In step S1205, the control unit 11 acquires the position and orientation of the first viewpoint 1211 from the acquired route 1210. The position of the viewpoint 1211 is in front of the 2.5D model 400, and the orientation is set to face the direction of the arrow 1213. In the case where the processing in step S1205 is performed again after the processing in step S1208 is performed, the position and orientation of the viewpoint next to the previously processed viewpoint are acquired.

[0072] In step S1206, the control unit 11 selects a 2.5D model based on the position and orientation of the viewpoint. The selection method is similar to the selection method according to the first exemplary embodiment, so its description is omitted. Fig. 12B , only the 2.5D model 400 selected at the first viewpoint 1211 is illustrated.

[0073] In step S1207, control unit 11 acquires an image at the viewpoint acquired in step S1205 and stores the acquired image in storage unit 13. Here, as in the first exemplary embodiment, the 2.5D model selected in step S1206 is displayed in the image.

[0074] In step S1208, the control unit 11 determines whether the position and orientation of the current viewpoint are at the end point 1212 of the route 1210. If the viewpoint is not at the end point 1212 of the route 1210 ("No" in step S1208), the control unit 11 performs the processing in step S1205 again. If the viewpoint reaches the end point 1212 of the route 1210 ("Yes" in step S1208), the control unit 11 performs the processing in step S1209.

[0075] In step S1209 , the control unit 11 generates a moving image based on the plurality of images stored in the storage unit 13 , and stores the moving image in the storage unit 13 .

[0076] The above describes a method for generating a video displaying an appropriate 2.5D model according to the position and orientation of the viewpoint given a plurality of 2.5D models of the head and the route of the viewpoint. Since the 2.5D model to be displayed is switched according to the position and orientation of the viewpoint, it is not necessary to spend time and effort to select one by one the 2.5D models that appear in each frame of the moving image from a plurality of 2.5D models. As in the first exemplary embodiment, an appropriate 2.5D model is selected according to the viewpoint, and only the selected 2.5D model is displayed in the frame of the moving image, so that incomplete overlap is not generated between the 2.5D models, and visibility of the subject can be maintained.

[0077] According to the present exemplary embodiment, although the method of generating the route is not particularly specified, the order of specifying the surface of the 3D primitive by user input can be obtained, and the route can be generated based on the order. The 3D primitive refers to a 3D graphic whose surface can be specified, such as, Fig.13A 1300, pentagonal column 1301 or hexagonal column 1302 in the figure. For example, as a method of generating a route, consider a case where the surface of the rectangular parallelepiped 1300 is specified in the order of arrows 1303, 1304 and 1305. The center of the rectangular parallelepiped 1300 roughly coincides with the center of the plurality of 2.5D models. The width and height of the rectangular parallelepiped 1300 are within the range within which the plurality of 2.5D models fall, and the width and height can be automatically adjusted so that the 2.5D models are located at approximately the same distance from the surface.

[0078] The position and orientation of the first viewpoint (hereinafter referred to as the starting point) of the route are determined to be located at the center of the surface indicated by arrow 1303 and facing the direction of arrow 1303. However, the position of the viewpoint is set so that multiple 2.5D models inside the rectangular parallelepiped 1300 are captured in the image. The distance from the surface to the viewpoint can be a fixed value or can be arbitrarily specified by the user. At this time, the user can specify the distance from the surface to the viewpoint by adjusting the length of arrow 1303. In this way, the user determines the position and orientation of the starting point. Fig. 13B In FIG. 4 , a starting point 1310 is indicated. The orientation of the starting point 1310 is set to face the direction of the 2.5D model 400.

[0079] Next, the midpoint and the end point of the route are determined. On the second surface designated by arrow 1304, the position and orientation of the viewpoint at the midpoint are determined in the same manner as the position and orientation of the start point. The same applies to arrow 1305. Here, since arrow 1305 designates the last surface, the position and orientation of the viewpoint determined by arrow 1305 become the position and orientation of the end point. If the number of designated surfaces is 2 or less, there is no midpoint, and if the number is 4 or more, there are multiple midpoints. Fig. 13B In FIG. 1 , a midpoint 1311 and an end point 1312 are indicated.

[0080] The route is generated based on the information of the start point, midpoint, and end point acquired so far. The route can be generated using existing methods, for example, a method of filling the gaps between each point at fixed intervals using linear interpolation, or a smooth route can be generated using a spline function or the like.

[0081] A route may be generated on a spherical surface centered at the center of the cuboid 1300 so that the distance from the center of the cuboid 1300 to each viewpoint is constant. A dotted line 1313 is a route connecting the start point 1310, the midpoint 1311, and the end point 1312 at equal intervals on the spherical surface. The orientation of each viewpoint is defined as facing the center of the cuboid 1300.

[0082] The method for generating a route is described above. Using 3D primitives enables the user to intuitively generate a route. At this time, each surface of the 3D primitive can correspond to the camera direction of the 2.5D model. The starting point, midpoint and end point are the camera directions of the 2.5D model, so the route may include a viewpoint that can generate a good-looking image. At this time, the control unit 11 can present a suitable 3D primitive as a candidate to the user based on the position and orientation of multiple 2.5D models and their camera directions, or it can set the suitable 3D primitive as the default 3D primitive from the beginning. For example, according to this exemplary embodiment, 2.5D models are acquired from four directions at approximately equal intervals in the front, back, left and right directions, so that 3D primitives such as cubes or cuboids can be pre-set. Therefore, the user can save time and effort in selecting 3D primitives. The surface that does not correspond to the 2.5D model can be made unselectable. Therefore, it is unlikely that the route includes a viewpoint that generates an ugly image. In addition, at this time, the image can only be stored when the relationship between the viewpoint and the 2.5D model satisfies the expression (1) shown in the first exemplary embodiment. Therefore, the route is less likely to include viewpoints that generate ugly images.

[0083] According to this exemplary embodiment, although the method for obtaining the position and orientation of the 2.5D model is not particularly specified, the user can use 3D primitives to input the position and orientation of the 2.5D model. At this time, the designation method can be performed in the same manner as the method for specifying the position and orientation of the viewpoint. For example, in the case where the user understands that the 2.5D model is acquired from four directions at approximately equal intervals in the front, back, left and right directions according to this exemplary embodiment, the user specifies a cube as a 3D primitive and specifies the 2.5D model corresponding to each surface. Here, it is not specified that the top surface and bottom surface of the corresponding 2.5D model are not included. Using 3D primitives enables the user to intuitively specify the position and orientation of the 2.5D model.

[0084] Furthermore, when acquiring the 2.5D model, a measuring device for acquiring the position and orientation, such as an inertial measurement unit (IMU) or a global positioning system (GPS), can be used to acquire the position and orientation of the 2.5D model. Therefore, the position and orientation of the 2.5D model can be automatically acquired, thereby reducing the burden on the user. A similar effect can be achieved in the case of acquiring the position and orientation of multiple 2.5D models based on multiple acquired 2.5D models and data (such as images used to generate models).

[0085] This method does not require a measuring device such as an IMU or GPS. A self-position estimation method for capturing a large number of subject images and estimating the image capturing positions is known as an existing method, and this method can be used to estimate the position and orientation of a 2.5D model.

[0086] The third exemplary embodiment described below is an example of an apparatus for executing a 2.5D model processing method by a computer. According to this exemplary embodiment, a use case is described in which a subject is viewed by appropriately displaying images of arbitrary viewpoints pre-acquired using a 2.5D model of a person's head photographed from multiple directions. In other words, according to this exemplary embodiment, a 2.5D model refers to a plurality of images that can be acquired from a 3D model. A plurality of images can be generated based on a video acquired using a 2.5D model. Descriptions of configurations and processes similar to those according to the first exemplary embodiment will be omitted. Some embodiments may be performed on a desktop computer, a laptop computer, a portable computer, and the like.

[0087] A 2.5D model processing method according to the present exemplary embodiment will be described below, but the configuration of a computer that executes the present processing method is similar to that of the computer according to the first exemplary embodiment, and thus a description thereof will be omitted.

[0088] Will refer to Fig.14 The flowchart in specifically describes the 2.5D model processing method executed by a computer according to the present exemplary embodiment. However, steps S1401 and S1402 are similar to those according to the first exemplary embodiment, and thus description thereof will be omitted.

[0089] In step S1403, the control unit 11 acquires an image from each position and orientation of the viewpoint for each of the plurality of 2.5D models, and stores the image and information related to the position and orientation of the viewpoint at this time in the storage unit 13. The information related to the position and orientation of the viewpoint refers to the position and orientation of the viewpoint itself or information based on which the information can be estimated by calculation. For example, the information based on which the information can be estimated is the position and orientation of the viewpoint in the coordinate system defined for each 2.5D model. Since the relative positions and orientations of the plurality of 2.5D models are known, even if the coordinate system defined for each 2.5D model is used, the position and orientation of the viewpoint can be converted into the position and orientation in the space of the common coordinate system.

[0090] The processing in steps S1404 and S1405 is similar to that according to the first exemplary embodiment, and thus description thereof will be omitted.

[0091] In step S1406, the control unit 11 acquires an image acquired at a position and orientation closest to that of the current viewpoint from the images acquired in step S1403 for the selected 2.5D model. The comparison of the position and orientation is performed using an existing method such as a nearest neighbor algorithm.

[0092] The processing in steps S1407 and S1408 is similar to that according to the first exemplary embodiment, and thus description thereof will be omitted.

[0093] The above describes a method for viewing a 2.5D model by the user controlling the position and orientation of the viewpoint and switching the image to be displayed in the presence of an image group acquired using each 2.5D model of the head. Since the displayed 2.5D model is switched according to the position and orientation of the viewpoint, the time and effort of opening multiple 2.5D models one by one can be saved. In addition, a suitable 2.5D model is selected according to the viewpoint and an image pre-generated using the selected 2.5D model is displayed, so that incomplete overlap is not generated and visibility of the subject can be maintained. Since the pre-stored image is viewed, the image can be viewed even on a computer that has difficulty in generating images.

[0094] According to the present exemplary embodiment, although a method for storing an image and information related to the position and orientation of a viewpoint is not specified, the information related to the position and orientation can be stored in meta information of the image such as an exchangeable image file format (Exif) and a file name. Since the information related to the position and orientation is stored in association with the image and the file, information management becomes easier.

[0095] Some embodiments of the present invention also include the following. First, the program code is read from the storage medium, and the read program code is written into a memory provided in a function expansion board inserted in the computer or a function expansion unit connected to the computer. Then, a CPU or the like provided on the function expansion board or the function expansion unit performs part or all of the actual processing based on the instructions of the program code.

[0096] The present invention can reduce the time and effort of viewing multiple 2.5D models while maintaining the visibility of the subject.

[0097] Other embodiments

[0098] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0099] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The following claims are to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An information processing device, comprising: one or more memories for storing instructions; one or more processors in communication with the one or more memories and, when executing the instructions, cooperating with the one or more memories to: Acquire a plurality of three-dimensional (3D) models, each 3D model corresponding to a first subject and viewing the first subject from a different viewpoint position; Specifying a position and orientation of a viewpoint for viewing the 3D model; selecting at least one 3D model from the plurality of 3D models based on the position and orientation of the viewpoint; and An image of the at least one 3D model viewed from the position and orientation of the viewpoint is generated. 2 . The information processing device according to claim 1 , further comprising a display configured to display the image.

3. The information processing device according to claim 1, wherein: The position and orientation of the viewpoint are obtained based on user input.

4. According to the information processing device described in claim 1, when executing the instructions, the one or more processors also cooperate with the one or more memories to generate a moving image based on multiple images of the at least one 3D model generated according to the positions and orientations of multiple specified viewpoints.

5. The information processing device according to claim 1, wherein: The at least one 3D model is selected based on information about the position and orientation of the viewpoint and positions and orientations corresponding to the plurality of 3D models.

6. The information processing device according to claim 5, wherein: Among the plurality of 3D models, the at least one 3D model has a minimum cosine between the direction of the position and orientation of the viewpoint and the direction of the corresponding position and orientation.

7. The information processing device according to claim 1, wherein: At least one 3D model is selected from the plurality of 3D models based on the credibility of the shapes of the plurality of 3D models.

8. The information processing device according to claim 5, wherein: When executing the instructions, the one or more processors also cooperate with the one or more memories to receive a user selection of a surface from 3D primitives including a surface corresponding to any of the multiple 3D models, and select the position and orientation of the viewpoint and the 3D model.

9. The information processing device according to claim 1, wherein: When executing the instructions, the one or more processors also cooperate with the one or more memories to generate a first image and a second image while generating multiple images based on the positions and orientations of the multiple viewpoints, so that the size of at least a portion of the subject shared by the first and second images generated continuously is the same.

10. The information processing device according to claim 1, wherein: When executing the instructions, the one or more processors also cooperate with the one or more memories to determine the position and orientation of the background model arranged around the 3D model when the 3D model corresponding to the continuously generated images in the multiple 3D models changes, so that the change of the background model is within a predetermined range.

11. The information processing device according to any one of claims 1 to 10, wherein: In executing the instructions, the one or more processors further cooperate with the one or more memories to record in the image the position and orientation of the viewpoint when the image was generated.

12. A method for controlling an information processing device, the method comprising: an acquisition step for acquiring a plurality of three-dimensional (3D) models, each 3D model corresponding to a first object and viewing the first object from a different viewpoint position; A specifying step for specifying a position and orientation of a viewpoint for viewing the 3D model; A selecting step for selecting at least one 3D model from the plurality of 3D models based on the position and orientation of the viewpoint; and A generating step is used to generate an image of the at least one 3D model viewed from the position and orientation of the viewpoint.

13. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by at least one computer, cause the at least one computer to: Acquire a plurality of three-dimensional (3D) models, each 3D model corresponding to a first subject and viewing the first subject from a different viewpoint position; Specifying a position and orientation of a viewpoint for viewing the 3D model; selecting at least one 3D model from the plurality of 3D models based on the position and orientation of the viewpoint; and An image of the at least one 3D model viewed from the position and orientation of the viewpoint is generated.

14. A computer program product storing a program for causing a computer to execute the method according to claim 12.

Images