Information processing device, information processing method, and program

Abstract

This technology provides support for more effective interaction between 2D and 3D objects. [Solution] An information processing device is provided, comprising: a two-dimensional object placement unit that places two-dimensional objects, which are planar objects, in a three-dimensional digital space; and a two-dimensional object processing unit that, based on input of a predetermined operation relating to the two-dimensional object, performs a predetermined process relating to the two-dimensional object in a direction different from the direction parallel to the plane of the two-dimensional object.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, with the spread of remote conferencing systems in which multiple users conduct voice calls while sharing images via a network, the quality of communication between users located at remote locations has been greatly improved. In such a remote conferencing system, each of a plurality of users can add static information (e.g., characters, figures, and lines) or dynamic information (e.g., movement of a cursor) to an image. As a result, the variations in expressions in remote communication have increased.

[0003] Furthermore, the information added to an image is not limited to two-dimensional information, but also extends to three-dimensional information. That is, by enabling the placement of three-dimensional (hereinafter, the dimension is also simply denoted as "D") objects in a 3D digital space or the movement of 3D objects in a 3D digital space, three-dimensional expressions in remote communication have become possible.

[0004] As techniques for operating 3D objects, various techniques are known. For example, Patent Document 1 discloses a technique for detecting a user's viewpoint and gesture and moving a 3D object hit by the user's viewpoint according to the user's gesture. A light source and a camera are used to detect the user's viewpoint, and an optical sensor is used to detect the user's gesture.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

[0006] The image above is applied as a texture to a plane that constitutes a 2D object placed in a 3D digital space. In this case, since the 2D object has a lower dimension than the 3D object, it is desirable to improve the expressiveness of the 2D object to support more effective interaction between 2D and 3D objects. Improving the expressiveness of the 2D object can also be rephrased as extending the functionality of the 2D object.

[0007] Therefore, the present invention aims to solve the above problems and to provide a technology that enables more effective interaction between 2D objects and 3D objects. [Means for solving the problem]

[0008] To solve the above problems, according to one aspect of the present invention, an information processing device is provided, comprising: a two-dimensional object placement unit that places a two-dimensional object, which is a planar object, in a three-dimensional digital space; and a two-dimensional object processing unit that, based on input of a predetermined operation relating to the two-dimensional object, performs a predetermined process relating to the two-dimensional object in a direction different from the direction parallel to the plane of the two-dimensional object.

[0009] The predetermined operation may include a region designation operation that designates a portion of the two-dimensional object as a designated region, and the predetermined process may include a process that deforms the two-dimensional object by moving the designated region in the three-dimensional digital space in the different directions.

[0010] The aforementioned different directions may be the normal directions of the two-dimensional object with respect to the plane in the three-dimensional digital space.

[0011] The predetermined operation includes a movement amount specification operation that specifies the amount of movement of the designated area, and the two-dimensional object processing unit may move the designated area by the specified amount of movement.

[0012] The information processing device may include a three-dimensional object processing unit that, based on movement operations of the three-dimensional object existing in the three-dimensional digital space, moves the three-dimensional object such that part or all of the area of ​​the three-dimensional object is located behind the designated area with respect to the position of the virtual camera.

[0013] The predetermined operation includes specifying the axis direction after projection, where two mutually orthogonal axes in the three-dimensional digital space are projected onto a plane parallel to the layer for generating visual information. The predetermined process may also include calculating the two axes in the three-dimensional digital space based on the projected axis direction, calculating a three-dimensional orthogonal coordinate system based on the two axes, and associating the three-dimensional orthogonal coordinate system with the two-dimensional object.

[0014] The two-dimensional object processing unit may, when the angle formed by the projected axes is of a predetermined magnitude, obtain a length specification operation that specifies the length after projection by projecting one of the two axes onto the layer, and calculate the three-dimensional Cartesian coordinate system based on the projected axis and the projected length.

[0015] The predetermined operation includes a coordinate specification operation in which the coordinates inside the two-dimensional object are specified as specified coordinates, and the two-dimensional object processing unit may calculate the three-dimensional Cartesian coordinate system with the specified coordinates as the origin.

[0016] The information processing device may include a three-dimensional object processing unit capable of moving the three-dimensional object in the axis direction of the three-dimensional Cartesian coordinate system based on the movement operation of the three-dimensional object existing in the three-dimensional digital space.

[0017] According to another aspect of the present invention to solve the above problems, arranging a two-dimensional object, which is a planar object, in a three-dimensional digital space, and based on the input of a predetermined operation related to the two-dimensional object, executing a predetermined process related to the two-dimensional object in a direction different from the direction parallel to the plane of the two-dimensional object, and an information processing method executed by a computer is provided.

[0018] According to another aspect of the present invention to solve the above problems, a program is provided that causes a computer to function as a two-dimensional object arranging unit that arranges a two-dimensional object, which is a planar object, in a three-dimensional digital space, and a two-dimensional object processing unit that executes a predetermined process related to the two-dimensional object in a direction different from the direction parallel to the plane of the two-dimensional object based on the input of a predetermined operation related to the two-dimensional object.

Effect of the Invention

[0019] As described above, according to the present invention, a technology is provided that enables more effective interaction between 2D objects and 3D objects.

Brief Description of the Drawings

[0020] [Figure 1] It is a diagram for explaining an outline of an embodiment of the present invention. [Figure 2] It is a diagram showing a functional configuration example of a shared image object function expansion system 20 according to a first embodiment of the present invention. [Figure 3] It is a diagram showing a configuration example of a shared image object related database 224. [Figure 4] It is a diagram for explaining a shared image object in a default state. [Figure 5] It is a first diagram for explaining a deformation of a shared image object. [Figure 6]This is a second figure for explaining the deformation of the shared image object. [Figure 7] This is a figure showing how various objects are arranged in the 3D digital space 100. [Figure 8] This is a figure showing an example of visual information generated based on various objects arranged in the 3D digital space 100 shown in FIG. 7. [Figure 9] This is a flowchart showing an example of an operation executed by the shared image object function extension system 20 according to the first embodiment of the present invention. [Figure 10] This is a first figure for explaining the outline of the second embodiment of the present invention. [Figure 11] This is a second figure for explaining the outline of the second embodiment of the present invention. [Figure 12] This is a third figure for explaining the outline of the second embodiment of the present invention. [Figure 13] This is a figure showing a functional configuration example of the shared image object function extension system 120 according to the second embodiment of the present invention. [Figure 14] This is a figure showing a configuration example of the shared image object related database 1224. [Figure 15] This is a figure for explaining the detailed functions of the shared image object coordinate system generation unit 1228. [Figure 16] This is a flowchart showing an example of an operation executed by the shared image object function extension system 120 according to the second embodiment of the present invention. [Figure 17] This is a flowchart showing an example of an operation executed by the shared image object function extension system 120 according to the second embodiment of the present invention. [Figure 18] This is a flowchart showing an example of an operation executed by the shared image object function extension system 120 according to the second embodiment of the present invention. [Figure 19] This is a figure showing the hardware configuration of the information processing apparatus 900 as an example of the shared image object function extension system 20 according to an embodiment of the present invention. [Modes for carrying out the invention]

[0021] Preferred embodiments of the present invention will be described in detail below with reference to the attached drawings. In this specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0022] (0. Overview) First, an overview of the embodiments of the present invention will be described.

[0023] Figure 1 is a diagram illustrating an overview of an embodiment of the present invention. Referring to Figure 1, a 3D digital space 100 is constructed. Also referring to Figure 1, the world coordinate system XYZ is shown. For example, a position in the 3D digital space 100 can be represented by the X, Y, and Z coordinates in the world coordinate system XYZ. The 3D digital space 100 can be constructed by a system. For example, such a system is the shared image object function extension system 20 shown in Figure 2.

[0024] A shared image object 101 is placed in the 3D digital space 100. The shared image object 101 is a 2D object having a planar shape. Images shared by multiple users via a network may be pasted onto the plane of the shared image object 101, and these multiple users may be able to make voice calls while viewing such images. In other words, the system according to the embodiment of the present invention can be applied to a remote conferencing system. In the example shown in Figure 1, an image of a distribution board is pasted onto the plane of the shared image object 101, but any image can be pasted onto the plane of the shared image object 101.

[0025] The GUI (Graphical User Interface) layer 104 is located in the 3D digital space 100 and is a layer for generating visual information presented to the user. The GUI layer 104 may contain interactive elements (e.g., buttons, text information, and cursors). These interactive elements are incorporated into the images presented to the user and are used to accept user input.

[0026] In an embodiment of the present invention, the 3D object existence region 102 is the region sandwiched between the shared image object 101 and the GUI layer 104, and is the region in which the existence of the 3D object 103 is permitted. The shared image object 101, which is placed in the 3D digital space 100, and the 3D object 103, which is placed in the 3D object existence region 102, are projected onto the GUI layer 104 by perspective projection toward the virtual camera 105 to generate an image presented to the user.

[0027] The user attempts to manipulate the 3D object 103 while viewing the image generated by projection onto the GUI layer 104. However, since the shared image object 101 has a lower dimension than the 3D object 103, it is desirable to improve the expressiveness of the shared image object 101 to support more effective interaction between the shared image object 101 and the 3D object 103. Improving the expressiveness of the shared image object 101 can also be rephrased as extending the functionality of the shared image object 101.

[0028] Therefore, this specification primarily proposes a technology that enables more effective interaction between the shared image object 101 and the 3D object 103.

[0029] The embodiments of the present invention have been described above.

[0030] (1. First Embodiment) Next, we will describe the details of the first embodiment of the present invention.

[0031] (1-1. Configuration of the Shared Image Object Functionality Extension System) First, an example configuration of the shared image object function extension system 20 according to the first embodiment of the present invention will be described. The shared image object function extension system 20 can be implemented by one or more computers. The shared image object function extension system 20 can be implemented by a device that processes various types of information, and therefore can also be referred to as an information processing device.

[0032] Figure 2 shows an example of the functional configuration of a shared image object function extension system 20 according to the first embodiment of the present invention. The shared image object function extension system 20 comprises an input unit 210, a control unit (not shown), a storage unit (not shown), and a display unit 250.

[0033] The control unit (not shown) includes an arithmetic unit such as a CPU (Central Processing Unit), and its functions can be realized by the program stored in ROM (Read Only Memory) being loaded into RAM by the arithmetic unit and executed. In this case, a computer-readable recording medium on which the program is stored may also be provided.

[0034] Alternatively, the control unit (not shown) may be composed of dedicated hardware or a combination of multiple hardware components. Data necessary for calculations by the arithmetic unit is appropriately stored in a storage unit (not shown). The control unit (not shown) comprises an input processing unit 211, a shared image object management unit 222, a shared image object deformation processing unit 226, a texture mapping processing unit 227, a 3D object processing unit 240, and a display control unit 251.

[0035] The storage unit (not shown) is a memory device capable of storing programs and various information for operating the control unit (not shown). For example, the storage unit (not shown) may be composed of non-volatile memory. For example, the storage unit (not shown) can also temporarily store data required during the operation of the control unit (not shown).

[0036] The storage unit (not shown) comprises a shared image storage unit 225 and a shared image object-related database 224.

[0037] (Input section 210) The input unit 210 receives operations entered by the user. In the first embodiment of the present invention, it is mainly assumed that the input unit 210 is an input device that receives operations indicated by two-dimensional coordinates. Such an input device may be implemented by a mouse, keyboard, or touch panel.

[0038] (Display section 250) The display unit 250 is composed of a display and has the function of displaying information according to the control of the display control unit 251. Here, the form of the display is not particularly limited. For example, the display unit 250 may be a liquid crystal display (LCD) device or an OLED (Organic Light Emitting Diode) device.

[0039] (Input processing unit 211) The input processing unit 211 determines the type of operation based on the operation received from the user by the input unit 210. For example, the input processing unit 211 may determine that a selection operation has been performed on an operation UI when a click or tap operation is performed on that operation UI. Similarly, the input processing unit 211 may determine that a selection operation has been performed on an object (a shared image object or a 3D object) when a click or tap operation is performed on that object.

[0040] The input processing unit 211 then outputs to at least one of the following, depending on the type of operation it has identified, that an operation has been input from the user: the shared image object management unit 222, the shared image object deformation processing unit 226, the texture mapping processing unit 227, and the 3D object processing unit 240. The types of operations that can be input by the user will be described in detail later.

[0041] (Shared Image Object Management Unit 222) The shared image object management unit 222 functions as a 2D object placement unit that places shared image objects in a 3D digital space. Shared image objects may be drawn using computer graphics. For example, a shared image object may have a single planar shape of a rectangle composed of vertices, edges, and faces, but the shape of the shared image object is not limited.

[0042] A shared image object may consist of a texture, which is image data pasted onto a single rectangular plane, and the color and texture of the surface of the shared image object. In the first embodiment of the present invention, it is mainly assumed that the shared image object is fixed in a specific position and orientation in 3D digital space. However, the position and orientation of the shared image object may change based on user operations or the like.

[0043] (Shared image object related database 224) The shared image object-related database 224 is a database that stores information about shared image objects. For example, the shared image object-related database 224 may be configured by associating the shared image object's name, position, orientation, size, and shape information with the shared image object's model data, the material data that constitutes the shared image object, and mapping data that shows the correspondence between the shared image object's position and the texture pixels. An example of the configuration of the shared image object-related database 224 will be described in detail later.

[0044] (Shared image storage section 225) The shared image storage unit 225 stores textures, which are image data that can be pasted onto a single rectangular plane. For example, the texture format may be JPEG (Joint Photographic Experts Group) or PNG (Portable Network Graphics). However, the texture format is not limited to a specific format.

[0045] (Shared image object deformation processing unit 226) The shared image object deformation processing unit 226 functions as an example of a 2D object processing unit that executes predetermined processing related to the shared image object (hereinafter also referred to as "shared image object function enhancement processing"). The shared image object function enhancement processing is processing related to the shared image object in a direction different from the direction parallel to the plane of the shared image object. This improves the expressiveness of the shared image object.

[0046] Based on the input processing unit 211's determination that a predetermined operation relating to the shared image object has been input to the input unit 210, the shared image object deformation processing unit 226 executes a function extension process for the shared image object.

[0047] In the first embodiment of the present invention, a predetermined operation includes a region designation operation in which a portion of the shared image object is designated as a specified region. The shared image object deformation processing unit 226 then performs a process to deform the shared image object as a functional extension process for the shared image object by moving the specified region in a direction different from the direction parallel to the plane of the shared image object.

[0048] Furthermore, the predetermined operation may include a movement amount specification operation that specifies the amount of movement of a specified area. In this case, the shared image object deformation processing unit 226 may deform the shared image object by moving the specified area by the specified amount as a function extension process for the shared image object.

[0049] (Texture mapping processing unit 227) The texture mapping processing unit 227 creates mapping data that shows the correspondence between the position of the shared image object and the pixels of the texture, based on the shape data of the shared image object after the function extension processing and the texture stored by the shared image storage unit 225. Then, the texture mapping processing unit 227 updates the mapping data in the shared image object related database 224 with the created mapping data. Along with the update of the mapping data, the state of the texture attached to the plane of the shared image object is also updated.

[0050] (3D object processing unit 240) The 3D object processing unit 240 places 3D objects in a 3D digital space. The 3D objects may be drawn using computer graphics. For example, the 3D objects may have a predetermined shape enclosed by vertices, edges, and faces. For example, the 3D object processing unit 240 can change the orientation of the 3D objects in the 3D digital space.

[0051] Furthermore, the 3D object processing unit 240 identifies the permitted directions of movement for the 3D object in the 3D digital space. The permitted directions of movement may be one direction or multiple directions. When the 3D object processing unit 240 determines that the operation input by the user to the input unit 210 is a movement operation for the 3D object (for example, a drag operation on the 3D object), it changes the position of the 3D object in one of the permitted directions of movement.

[0052] (Display control unit 251) The display control unit 251 generates visual information to be displayed by the display unit 250. The display control unit 251 also controls the display unit 250 so that the generated visual information is displayed by the display unit 250. For example, the display control unit 251 generates visual information based on various objects placed in a 3D digital space. These objects may include 2D objects and 3D objects.

[0053] More specifically, the display control unit 251 generates visual information by projecting various objects using perspective projection. The visual information generated by the display control unit 251 may also include various user interfaces (UIs). The visual information displayed by the display unit 250 can be viewed by the user.

[0054] The above describes an example of the configuration of the shared image object function extension system 20 according to the first embodiment of the present invention.

[0055] (1-2. Functional details of the shared image object function extension system 20) The functional details of the shared image object function extension system 20 according to the first embodiment of the present invention will be described with reference to Figures 3 to 8 (and Figures 1 and 2 as appropriate).

[0056] The shared image object management unit 222 generates shared image objects and places the generated shared image objects in the 3D digital space. The shared image object management unit 222 then registers information about the generated 2D objects in the shared image object related database 224. An example of the configuration of the shared image object related database 224 will be explained with reference to Figure 3.

[0057] Figure 3 shows an example of the configuration of the shared image object association database 224. As shown in Figure 3, the shared image object association database 224 is configured by associating the name, position, orientation, size, model data, and material data that constitute the shared image object with each other.

[0058] The name of a shared image object is a name used to identify the shared image object. For example, the name of a shared image object may be assigned by the user, or it may be automatically assigned by the shared image object management unit 222 if it is a name that is not being used for any other shared image object.

[0059] The position of a shared image object is its three-dimensional position in 3D digital space. That is, the position of a shared image object can be represented by three-dimensional coordinates (x,y,z) in the world coordinate system XYZ that defines the 3D digital space. For example, the position of a shared image object may be its representative position. For example, the representative position of a shared image object may be its centroid. For example, the position of a shared image object may be specified by the user or predetermined.

[0060] The orientation of a shared image object is its orientation in 3D digital space. That is, the orientation of a shared image object can be expressed by three-dimensional coordinates (θx, θy, θz) in the world coordinate system XYZ that represents 3D digital space. For example, the orientation of a shared image object may be specified by the user or predetermined.

[0061] The size of a shared image object is its size in 3D digital space. For example, if a shared image object has a rectangular shape, its size may be expressed as a combination of the rectangle's width and height (w,h). For example, the size of a shared image object may be specified by the user or predetermined.

[0062] The model data of a shared image object is information about the shape of the shared image object in 3D digital space. For example, the model data of a shared image object may be data indicating the three-dimensional positions of each vertex, edge, and face that make up the shared image object. For example, the model data of a shared image object may be specified by the user or predetermined.

[0063] The material data of a shared image object includes reference information for the texture, which is image data applied to the model of the shared image object, as well as data that includes the color and texture of the surface of the shared image object. For example, the material data of a shared image object may be specified by the user or predetermined.

[0064] Referring to Figure 3, an example configuration of the shared image object-related database 224 was described.

[0065] Figure 4 is a diagram illustrating a shared image object in its default state. The shared image object in its default state is the shared image object before deformation. Shared image object 401a shown in Figure 4 is a shared image object in its default state. The illustration of the texture applied to the plane of shared image object 401a is omitted from the diagram of shared image object 401a. In this case, the shape of the shared image object may be composed of a collection of triangular meshes, as shown in shared image object 401c.

[0066] Figure 5 is the first diagram illustrating the deformation of shared image objects. Referring to Figure 5, shared image objects 501a, 502a, and 503a are shown. The plane of shared image object 501a has a texture with a rectangle drawn on it, the plane of shared image object 501b has a texture with a pentagon drawn on it, and the plane of shared image object 501c has a texture with a cube drawn in a planar manner on it. We will assume that these shared image objects 501a, 502a, and 503a are displayed by the display unit 250.

[0067] In the following explanation, we will describe the case where all of these shared image objects 501a, 502a, and 503a are displayed by the display unit 250, and the transformations of the shared image objects 501a, 502a, and 503a are executed in parallel. However, it is not necessary for all of the shared image objects 501a, 502a, and 503a to be displayed by the display unit 250. Also, it is not necessary for the transformations of the shared image objects 501a, 502a, and 503a to be executed in parallel.

[0068] The user can input a region specification operation to the input unit 210. For example, the user can input a region specification operation that designates a region enclosed by a polygon with N points as vertices by inputting an operation (e.g., a click operation or a tap operation) that specifies N points (N≧3) on a shared image object, thereby designating the region as the designated region.

[0069] Note that N may be predetermined as a fixed value, or it may be determined by an operation such as a double-click or double-tap to specify the Nth vertex. In shared image objects 501b, 502b, and 503b, the border of the specified area is indicated by a thick line.

[0070] Based on the input processing unit 211's determination that a region specification operation has been input to the input unit 210, the shared image object deformation processing unit 226 reconstructs a set of triangular meshes according to the vertices and edges that constitute the frame of the specified region.

[0071] Shared image object 501c shows a set of triangular meshes reconstructed based on the specified region of shared image object 501b. Shared image object 502c shows a set of triangular meshes reconstructed based on the specified region of shared image object 502b. Shared image object 503c shows a set of triangular meshes reconstructed based on the specified region of shared image object 503b.

[0072] The shared image object deformation processing unit 226 deforms the shared image object by moving a specified area in the direction normal to the plane of the shared image object in 3D digital space. At this time, the direction of movement of the specified area may be the direction from the plane of the shared image object toward the virtual camera (hereinafter also referred to as the "virtual camera direction"), which is one of the directions normal to the plane of the shared image object. This gives the user a visual effect in which the specified area appears to be pushed forward.

[0073] The amount of movement of the specified area may be predetermined. Alternatively, the user may input a movement amount specification operation to the input unit 210. For example, the user may input a movement amount specification operation to specify the amount of movement of the specified area by inputting a numerical value indicating the amount of movement. If the input processing unit 211 determines that a movement amount specification operation has been input to the input unit 210, the shared image object deformation processing unit 226 may move the specified area by the specified amount.

[0074] Figure 6 is a second diagram illustrating the deformation of a shared image object. Referring to Figure 6, shared image objects 501d, 502d, and 503d are shown. In shared image objects 501d, 502d, and 503d, the triangular mesh to be moved is hatched. The shared image object deformation processing unit 226 moves the triangular mesh to be moved in the direction of the virtual camera.

[0075] Shared image objects 501e, 502e, and 503e are shared image objects after the triangular mesh to be moved has been moved in the direction of the virtual camera. The shared image object deformation processing unit 226 updates the model data in the shared image object related database 224 based on the three-dimensional positions of the vertices, edges, and faces that make up the shared image objects 501e, 502e, and 503e after the triangular mesh to be moved has been moved.

[0076] The texture mapping processing unit 227 creates mapping data based on the shape data of the deformed shared image object and the textures stored by the shared image storage unit 225. Then, the texture mapping processing unit 227 updates the mapping data in the shared image object related database 224 with the created mapping data.

[0077] Note that the pixel information for the sides 501f, 502f, and 503f of the three-dimensional part sandwiched between the specified area before movement and the specified area after movement does not exist in the texture. Therefore, the texture mapping processing unit 227 may assign the pixel information at the position of the frame of the specified area (thick lines in the shared image objects 501b, 502b, and 503b) to the pixel information of the sides 501f, 502f, and 503f.

[0078] The 3D object processing unit 240 determines the permissible movement directions of the 3D object in the 3D digital space where the shared image object is placed. Then, when the 3D object processing unit 211 determines that the operation input by the user to the input unit 210 is a movement operation of the 3D object (for example, a drag operation on the 3D object), the 3D object processing unit 240 moves the 3D object in one of the permissible movement directions.

[0079] Figure 7 shows how various objects are placed in the 3D digital space 100. Referring to Figure 7, the deformed shared image object 101 and the 3D object 31 exist in the 3D digital space 100. In the deformed shared image object 101, the specified region R1 is pushed out from the plane of the shared image object 101 toward the virtual camera 105.

[0080] As shown in Figure 7, the 3D object processing unit 240 can move the 3D object 31 based on the movement operation of the 3D object 31 so that part or all of the area of ​​the 3D object 31 is located behind the designated area R1 with respect to the position of the virtual camera 105. In other words, by moving the designated area R1, it becomes possible to represent the order in which the shared image object 101 and the 3D object 31 are arranged in the depth direction.

[0081] Figure 8 shows an example of visual information generated based on various objects placed in the 3D digital space 100 shown in Figure 7. Referring to Figure 8, visual information 802 is shown. Visual information 802 can be generated by projecting the deformed shared image object 101 and the 3D object 31 using perspective projection. By viewing the visual information 802, the user can understand the order of the shared image object 101 and the 3D object 31 in the depth direction.

[0082] The functional details of the shared image object function extension system 20 according to the first embodiment of the present invention have been described above.

[0083] (1-3. Example of operation of the shared image object function extension system 20) Next, with reference to Figure 9, we will briefly summarize an example of the operation of the shared image object function extension system 20 according to the first embodiment of the present invention. Figure 9 is a flowchart showing an example of operation performed by the shared image object function extension system 20 according to the first embodiment of the present invention. As shared image objects, we will use the shared image objects 501a to 501e shown in Figures 5 and 6 for explanation.

[0084] The display control unit 251 generates visual information to be displayed by the display unit 250. For example, the display control unit 251 generates visual information based on various objects placed in a 3D digital space. These objects may include shared image objects 501a and 3D objects. The display control unit 251 also controls the display unit 250 so that the generated visual information is displayed by the display unit 250.

[0085] The input processing unit 211 determines whether or not a user operation has been input to the input unit 210 (a802). If no user operation has been input to the input unit 210 (NO in a802), the input processing unit 211 executes a802 again. If a user operation has been input to the input unit 210 (YES in a802), the input processing unit 211 proceeds to a803.

[0086] The input processing unit 211 determines whether or not the area specification is complete (a803). As an example, consider a case where the user specifies a rectangle drawn on the shared image object 501a as the specified area. In such a case, the user sequentially inputs operations to the input unit 210 to select the vertices of the rectangle (for example, click operations or tap operations).

[0087] The user selects the fourth vertex of the rectangle by double-clicking or double-tapping it, and simultaneously inputs a completion command. At the time the completion command is input, the number of selected vertices is four, which is three or more vertices required to specify a polygon. Therefore, the shared image object deformation processing unit 226 sets the polygon composed of the four specified vertices as the specified region. For example, the specified region is the area enclosed by the thick line in the shared image object 501b.

[0088] If the input processing unit 211 determines that the region specification is not complete (NO in a803), it returns to a802. On the other hand, if the input processing unit 211 determines that the region specification is complete (YES in a803), the shared image object deformation processing unit 226 reconstructs the set of triangular meshes according to the vertices and edges that constitute the frame of the specified region (a804), and the display control unit 251 controls the display unit 250 so that a display is made to request the specification of the amount of movement of the specified region (a805).

[0089] In shared image object 503c, a set of triangular meshes reconstructed based on the specified region of shared image object 503b is shown. After a804, processing moves on to a808.

[0090] The input processing unit 211 determines whether or not a user operation has been input to the input unit 210 (a806). If no user operation has been input to the input unit 210 (NO in a806), the input processing unit 211 executes a806 again. If a user operation has been input to the input unit 210 (YES in a806), the input processing unit 211 proceeds to a807.

[0091] The input processing unit 211 determines whether the specification of the movement amount has been completed (s807). For example, whether the specification of the movement amount has been completed can be determined by whether or not an operation to specify the movement amount of the specified area has been input to the input unit 210. If the specification of the movement amount has not been completed (NO in a807), the shared image object deformation processing unit 226 returns to a806. On the other hand, if the specification of the movement amount has been completed (YES in a807), the process moves on to a808.

[0092] The shared image object deformation processing unit 226 determines whether or not the preparation for deformation of the shared image object 501c is complete (a808). The preparation for deformation of the shared image object 501c may include setting a specified region, reconstructing the triangular mesh set, and inputting the amount of movement.

[0093] If the shared image object deformation processing unit 226 is not ready to deform the shared image object 501c (i.e., "NO" in a808), it returns to a808.

[0094] On the other hand, when the shared image object deformation processing unit 226 has finished preparing to deform the shared image object 501c (YES in a808), it deforms the shared image object 501c by moving the triangular mesh of the specified region (the hatched triangular mesh of the shared image object 501d) in the direction of the virtual camera by a specified amount (a809). The shared image object 501e is the shared image object after the triangular mesh of the specified region has been moved in the direction of the virtual camera.

[0095] The shared image object deformation processing unit 226 updates the model data in the shared image object related database 224 based on the three-dimensional positions of the vertices, edges, and faces that constitute the deformed shared image object 501e.

[0096] The texture mapping processing unit 227 creates mapping data based on the shape data of the deformed shared image object 501e and the texture stored by the shared image storage unit 225 (a810). The texture mapping processing unit 227 then updates the mapping data in the shared image object related database 224 with the created mapping data. Along with the update of the mapping data, the texture application status of the shared image object 501e to the plane is also updated (a811).

[0097] The above describes an example of the operation of the shared image object function extension system 20 according to the first embodiment of the present invention.

[0098] (1-4. Effects) According to the first embodiment of the present invention, based on input of an operation relating to a shared image object, processing relating to the shared image object is performed in a direction different from the direction parallel to the plane of the shared image object. This improves the expressiveness of the shared image object and supports more effective interaction between the shared image object and 3D objects.

[0099] For example, the shared image object deformation processing unit 226 can deform the shared image object by moving the specified region of the shared image object in a direction different from the direction parallel to the plane of the shared image object, based on the region of the shared image object being specified. This produces an effect such as representing the order in which the shared image object and the 3D object are arranged in the depth direction.

[0100] The effects of the first embodiment of the present invention have been described above.

[0101] (2. Second Embodiment) Next, we will describe the details of a second embodiment of the present invention.

[0102] Figure 10 is a first diagram illustrating an overview of a second embodiment of the present invention. Figure 11 is a second diagram illustrating an overview of a second embodiment of the present invention. Figure 12 is a third diagram illustrating an overview of a second embodiment of the present invention. Here, a 3D object placed in a 3D digital space has a permitted direction of movement. The 3D object can move in the permitted direction in the 3D digital space based on the movement operation of the 3D object.

[0103] In this case, it may be required that the permitted movement direction of the 3D object be set not independently of the shared image object, but according to the shape drawn on the shared image object. For example, it may be required that the permitted movement direction of the 3D object be set in the same direction as the direction obtained by replacing the direction of the edge at the boundary between face B and face C of the cube-shaped 3D object drawn on the shared image object 101 (Figure 10) from a two-dimensional direction to a three-dimensional direction. If such a permitted movement direction is set, the 3D object can move along that three-dimensional direction.

[0104] In a second embodiment of the present invention, to satisfy these requirements, a coordinate system corresponding to the shape drawn on the shared image object is calculated and associated with the shared image object. This allows the 3D object to move along the axis direction of the coordinate system associated with the shared image object. The coordinate system associated with the shared image object may be a three-dimensional Cartesian coordinate system. An overview of the second embodiment of the present invention will be further described with reference to Figures 10 to 12.

[0105] For example, consider a case where an image 107 (lower left in Figure 10), taken from viewpoint G towards the Z' axis, of a cube-shaped 3D object 106 located in the world coordinate system X'Y'Z', as shown in the upper left of Figure 10, is used as a texture to be applied to the plane of a shared image object. In this case, the shared image object 101 located in the world coordinate system XYZ will be as shown in the upper right of Figure 10.

[0106] At this time, the user sees image 110 (the lower right image in Figure 10), and faces A, B, and C of the cube-shaped 3D object 106 are visible to the user. For example, in image 106 in the left image of Figure 11 and image 107 in the right image, face A - coordinate system xa, ya, za, face B - coordinate system xb, yb, zb, and face C - coordinate system xc, yc, zc are shown.

[0107] If a coordinate system equivalent to this one is associated with the shared image object 101, then if there is a movable object in the world coordinate system XYZ, it becomes possible to move that movable object along the axis direction of the coordinate system associated with the shared image object 101. This provides a similar feel to the operation obtained when moving a movable object in the world coordinate system X'Y'Z' along the axis directions of face A-coordinate system xa, ya, za, face B-coordinate system xb, yb, zb, and face C-coordinate system xc, yc, zc.

[0108] For example, consider a left-handed coordinate system xyz where each axis is represented by a unit vector, and the orientation of this coordinate system is as shown in the upper part of Figure 12 relative to the world coordinate system XYZ. In this case, when this coordinate system xyz is viewed in the Z direction, it will be as shown in the lower part of Figure 12. That is, if the origin of this coordinate system xyz lies on the shared image object 101 (Figure 1), and this coordinate system xyz is projected by orthogonal projection onto a plane (XY plane) parallel to the GUI layer 104 (Figure 1), the projected coordinate system xyz will be as shown in the lower part.

[0109] Conversely, consider the case where, when viewing the xyz coordinate system in the Z direction (i.e., when the origin of this xyz coordinate system lies on the shared image object 101 (Figure 1), and this xyz coordinate system is projected onto a plane parallel to the GUI layer 104 (Figure 1) (the XY plane) by orthogonal projection), the xyz coordinate system is a left-handed coordinate system, and each axis is represented by a unit vector. The coordinate system appears in the orientation shown in the lower part of Figure 12. In this case, the orientation of the xyz coordinate system in the world coordinate system XYZ will be as shown in the upper part of Figure 12.

[0110] From these points, the orientation of the xyz coordinate system in the world coordinate system XYZ is determined by setting how the xyz coordinate system looks when viewed in the Z direction (i.e., when the origin of this xyz coordinate system lies on the shared image object 101 (Figure 1), and this xyz coordinate system is projected by orthogonal projection onto a plane parallel to the GUI layer 104 (Figure 1) (the XY plane)), which is a left-handed coordinate system and where each axis is represented by a unit vector.

[0111] In other words, the orientation of the xyz coordinate system in the world coordinate system XYZ is determined by setting the two axis directions of the xyz coordinate system, where each axis is represented by a unit vector. However, depending on the angle formed by the two axis directions of the xyz coordinate system, where each axis is represented by a unit vector, the orientation of the xyz coordinate system in the world coordinate system XYZ may be determined by further setting the length L (0 ≤ L ≤ 1) of one of the two axes. Furthermore, the origin of the xyz coordinate system may be set, and the position of the xyz coordinate system may be determined in accordance with this origin.

[0112] (2-1. Configuration of the Shared Image Object Functionality Extension System) Next, an example configuration of the shared image object function extension system 120 according to the second embodiment of the present invention will be described. The shared image object function extension system 120 can be implemented by one or more computers. The shared image object function extension system 120 can be implemented by a device that processes various types of information, and therefore can also be referred to as an information processing device.

[0113] Figure 13 shows an example of the functional configuration of a shared image object function extension system 120 according to a second embodiment of the present invention. The shared image object function extension system 120 comprises an input unit 1210, a control unit (not shown), a storage unit (not shown), and a display unit 1250.

[0114] In a second embodiment of the present invention, a control unit (not shown) comprises an input processing unit 1211, a shared image object management unit 1222, a shared image object coordinate system generation unit 1228, a 3D object processing unit 240, and a display control unit 1251. In addition, in a second embodiment of the present invention, a storage unit (not shown) comprises a shared image storage unit 225 and a shared image object related database 1224.

[0115] The input unit 1210 accepts operations input by the user, similar to the input unit 210 according to the first embodiment of the present invention.

[0116] Furthermore, the input unit 1210 accepts axis direction specification operations entered by the user. The axis direction specification operation specifies the axis direction after projecting two mutually orthogonal axes in 3D digital space onto a plane (XY plane) parallel to the GUI layer 104 by orthogonal projection. In the following, the two mutually orthogonal axes in 3D digital space are referred to as the first axis (x axis) and the second axis (y axis) in the world coordinate system XYZ. The projected axes are the two axes (x axis and y axis) in the XY plane.

[0117] In the following explanation, we assume that the coordinate system xyz is a left-handed coordinate system, but it may also be a right-handed coordinate system. Furthermore, in the following explanation, we assume that the x, y, and z axes in the coordinate system xyz are unit vectors.

[0118] Furthermore, the input unit 1210 accepts length specification operations entered by the user. The length specification operation is an operation to specify the length L (0≦L≦1) after projection, when the angle formed by the projected axes (x axis and y axis) is of a predetermined magnitude (for example, 0 degrees, ±90 degrees, or 180 degrees), by projecting one of the x and y axes onto a plane parallel to the GUI layer 104 (XY plane) using orthogonal projection.

[0119] The axis on which length L can be specified may be selectable by the user. Alternatively, the axis on which length L can be specified may be predetermined. For example, the axis on which length L can be specified may be predetermined as the x-axis.

[0120] Furthermore, the input unit 1210 accepts coordinate specification operations entered by the user. A coordinate specification operation is an operation that specifies the coordinates inside the shared image object 101 (X and Y values ​​in the world coordinate system XYZ) as the specified coordinates. The position of these specified coordinates (X and Y values) is set as the origin of the xyz coordinate system.

[0121] (Input processing unit 1211) The input processing unit 211 determines the type of operation based on the operation received from the user by the input unit 1210. Then, according to the determined type of operation, the input processing unit 1211 outputs to at least one of the shared image object management unit 1222, the shared image object coordinate system generation unit 1228, and the 3D object processing unit 240 that an operation has been input from the user.

[0122] (Shared Image Object Management Unit 1222) The shared image object management unit 1222 functions as a 2D object placement unit that places shared image objects in a 3D digital space, similar to the shared image object management unit 222 in the first embodiment of the present invention.

[0123] (Shared image object related database 1224) The shared image object-related database 1224 is a database that stores information about shared image objects. For example, the shared image object-related database 1224 may be configured to associate the name of the xyz coordinate system associated with the shared image object with the position of the origin of the xyz coordinate system in the world coordinate system XYZ and the orientation of the xyz coordinate system in the world coordinate system XYZ. An example of the configuration of the shared image object-related database 1224 will be described in detail later.

[0124] (Shared image object coordinate system generation unit 1228) The shared image object coordinate system generation unit 1228 functions as an example of a 2D object processing unit that performs functional enhancement processing for the shared image object. This improves the expressiveness of the shared image object. The shared image object deformation processing unit 226 performs functional enhancement processing for the shared image object based on the input processing unit 1211's determination that a predetermined operation related to the shared image object has been input to the input unit 1210.

[0125] In a second embodiment of the present invention, a predetermined operation includes an axial direction specification operation that specifies the direction of the projected axes (x and y axes in the XY plane) obtained by orthogonal projection of two mutually orthogonal axes (x and y axes) in the world coordinate system XYZ onto a plane parallel to the GUI layer 104 (XY plane).

[0126] Then, as a functional extension process for the shared image object, the shared image object generation unit 1228 calculates two axes (x and y axes) in the world coordinate system XYZ based on the direction of the projected axes (x and y axes in the XY plane), calculates the coordinate system xyz based on the two axes (x and y axes) in the world coordinate system XYZ, and associates the coordinate system xyz with the shared image object.

[0127] Assume that the input processing unit 1211 has determined that a length specification operation has been input to the input unit 1210, which specifies the length L (0≦L≦1) after projecting one of the x-axis and y-axis onto a plane parallel to the GUI layer 104 (XY plane) by orthogonal projection. In such a case, the shared image object coordinate system generation unit 1228 may calculate the coordinate system xyz based on the length L and the directions of the two axes (x-axis and y-axis) in the world coordinate system XYZ.

[0128] Furthermore, consider the case where the input processing unit 1211 determines that a coordinate specification operation has been input to the input unit 1210, specifying the internal coordinates of the shared image object 101 (X and Y values ​​in the world coordinate system XYZ) as the specified coordinates. In such a case, the shared image object coordinate system generation unit 1228 may calculate a coordinate system xyz with the position of these specified coordinates (X and Y values) as the origin of the coordinate system xyz.

[0129] (Display section 1250) The display unit 1250 is composed of a display and has the function of displaying information according to the control of the display control unit 1251. In the second embodiment of the present invention, it is assumed that the axes of the xyz coordinate system generated by the shared image object coordinate system generation unit 1228 are added to the visual information displayed by the display unit 1250.

[0130] (Display control unit 1251) The display control unit 1251 generates visual information to be displayed by the display unit 1250. The display control unit 1251 also controls the display unit 1250 so that the generated visual information is displayed by the display unit 1250. For example, the display control unit 1251 generates visual information based on various objects placed in a 3D digital space. These objects may include 2D objects and 3D objects.

[0131] Furthermore, the display control unit 1251 controls the display unit 1250 so that the axes of the xyz coordinate system are displayed by the display unit 1250. The display control unit 1251 may keep the axes of the xyz coordinate system displayed at all times. Alternatively, the display control unit 1251 may display the axes of the xyz coordinate system when it becomes necessary to determine the permitted movement direction of the 3D object. Then, when the user inputs an operation to select the xyz coordinate system, the permitted movement direction of the 3D object may be determined to be in the axis direction of the xyz coordinate system.

[0132] The above describes an example configuration of the shared image object function extension system 120 according to the second embodiment of the present invention.

[0133] (2-2. Functional details of the shared image object function extension system 120) With reference to Figures 14 and 15 (and Figure 13 as appropriate), the functional details of the shared image object function extension system 120 according to the second embodiment of the present invention will be described.

[0134] Figure 14 shows an example of the configuration of the shared image object association database 1224. As shown in Figure 14, the shared image object association database 1224 may be configured such that an ID (csId) for identifying the xyz coordinate system associated with the shared image object is associated with the xyz coordinate system, the position csPosition(ox,oy,oz) of the origin of the xyz coordinate system in the world coordinate system XYZ is associated with the xyz coordinate system, and the orientation csRotation(θox,θoy,θoz) of the xyz coordinate system in the world coordinate system XYZ is associated with the xyz coordinate system.

[0135] Figure 15 is a diagram illustrating the functional details of the shared image object coordinate system generation unit 1228. As shown in Figure 15, suppose a user wants to set a coordinate system for region B enclosed by thick lines of a shared image object in which faces A, B, and C are drawn, and performs an operation to specify the centroid csPosition(obx,oby,obz) of region B as the origin of the coordinate system. Note that the centroid of region B may be specified by a click or tap operation by the user, or it may be specified by image analysis based on a click or tap operation performed by the user on a point within region B.

[0136] The user inputs operations to specify the projected axis directions (x-axis and y-axis directions) of two of the three axes (x-axis and y-axis) of the xyz coordinate system with the specified position as the origin, projected onto the XY plane. For example, the user inputs operations to specify the direction of the arrow of line lx', which is parallel to line lx along the edge constituting region B and passes through the origin csPosition, as the x-axis direction, and to specify the direction of the arrow of line ly', which is parallel to line ly along the line segment constituting region B and passes through the origin csPosition, as the y-axis direction.

[0137] The x-axis direction may also be specified by clicking or tapping the tip of the arrow of the line lx', so that the direction from the centroid csPosition to the tip of the arrow of line lx' is designated as the x-axis direction. Similarly, the y-axis direction may be specified by clicking or tapping the tip of the arrow of the line ly', so that the direction from the centroid csPosition to the tip of the arrow of line ly' is designated as the y-axis direction.

[0138] In this case, if the angle between the line lx' and the line ly' is not of a predetermined magnitude (for example, 0 degrees, ±90 degrees, or 180 degrees), the orientation of the coordinate system xyz is uniquely determined as follows: x-axis: xb, y-axis: yb, z-axis: zb.

[0139] On the other hand, if the angle between the line lx' and the line ly' is of a predetermined magnitude (for example, 0 degrees, ±90 degrees, or 180 degrees), the orientation of the coordinate system xyz is uniquely determined by specifying the length L (0 ≤ L ≤ 1) after projecting one of the two axes (x axis and y axis) onto a plane parallel to the GUI layer 104 (XY plane) by orthogonal projection.

[0140] The shared image object coordinate system generation unit 1228, once it uniquely determines the xyz coordinate system, registers a set of information about that xyz coordinate system, namely csId, csPosition, and csRotation, in the shared image object related database 1224. The axes of the xyz coordinate system are displayed by the display unit 1250. Furthermore, if the user inputs an operation to select the xyz coordinate system, the permissible movement direction of the 3D object may be specified in the axis direction of the xyz coordinate system.

[0141] The functional details of the shared image object function extension system 120 according to the second embodiment of the present invention have been described above.

[0142] (2-3. Example of operation of the shared image object function extension system 120) Next, with reference to Figures 16 to 18, we will briefly summarize an example of the operation of the shared image object function extension system 120 according to the second embodiment of the present invention. Figures 16 to 18 are flowcharts showing an example of operation performed by the shared image object function extension system 120 according to the second embodiment of the present invention.

[0143] The display control unit 1251 generates visual information to be displayed by the display unit 1250. For example, the display control unit 1251 generates visual information based on various objects placed in a 3D digital space. These objects may include shared image objects with faces A, B, and C (Figure 15) and 3D objects. The display control unit 1251 also controls the display unit 1250 so that the generated visual information is displayed by the display unit 1250.

[0144] The input processing unit 1211 determines whether or not a user operation has been input to the input unit 1210 (a1502). If no user operation has been input to the input unit 210 (NO in a1502), the input processing unit 211 executes a1502 again. If a user operation has been input to the input unit 1210 (YES in a1502), the input processing unit 1211 proceeds to a1503.

[0145] The input processing unit 1211 determines whether the origin of the coordinate system xyz has been specified (a1503). For example, if the user wants to set the coordinate system xyz for region B (Figure 15), they input an operation to specify the centroid csPosition(obx,oby,obz) of region B (for example, a click or tap operation).

[0146] If the input processing unit 1211 determines that the origin of the xyz coordinate system has not been specified (NO in a1503), it returns to a1502. On the other hand, if the input processing unit 1211 determines that the origin of the xyz coordinate system has been specified (YES in a1503), the display control unit 1251 controls the display unit 1250 to display a request for the direction of the first axis (x axis) of the three axes of the xyz coordinate system and the axis (X, Y, or Z) corresponding to the first axis (x axis) (a1504).

[0147] The input processing unit 1211 determines whether or not a user operation has been input to the input unit 1210 (a1506). If no user operation has been input to the input unit 1210 (NO in a1506), the input processing unit 1211 executes a1506 again. If a user operation has been input to the input unit 1210 (YES in a1506), the input processing unit 1211 proceeds to a1507.

[0148] The input processing unit 1211 determines whether the user has completed specifying the request (a1507). If the user has not completed specifying the request (NO in a1507), the input processing unit 1211 executes a1506 again. If the user has completed specifying the request (YES in a1507), the input processing unit 1211 proceeds to a1508.

[0149] The display control unit 1251 controls the display unit 1250 so that it displays a request for the direction of the second axis (y axis) of the three axes of the xyz coordinate system and the axis (X, Y, or Z) corresponding to the second axis (y axis) (a1508).

[0150] The input processing unit 1211 determines whether or not a user operation has been input to the input unit 1210 (a1510). If no user operation has been input to the input unit 1210 (NO in a1510), the input processing unit 1211 executes a1510 again. If a user operation has been input to the input unit 1210 (YES in a1510), the input processing unit 1211 proceeds to a1511.

[0151] The input processing unit 1211 determines whether the user has completed specifying the request (a1511). If the user has not completed specifying the request (NO in a1511), the input processing unit 1211 executes a1510 again. If the user has completed specifying the request (YES in a1511), the input processing unit 1211 proceeds to a1512.

[0152] The shared image object coordinate system generation unit 1228 calculates the angle between the first axis (x-axis) and the second axis (y-axis) in the XY plane. The shared image object coordinate system generation unit 1228 then determines whether the angle between the direction of the first axis (x-axis) and the direction of the second axis (y-axis) in the XY plane is a predetermined angle (0 degrees, ±90 degrees, or 180 degrees) (a1512).

[0153] The shared image object coordinate system generation unit 1228 proceeds to a1513 if the angle between the direction of the first axis (x-axis) and the direction of the second axis (y-axis) in the XY plane is a predetermined angle (0 degrees, ±90 degrees, or 180 degrees) (YES in a1512). On the other hand, the shared image object coordinate system generation unit 1228 proceeds to a1517 if the angle between the first axis (x-axis) and the second axis (y-axis) in the XY plane is not a predetermined angle (0 degrees, ±90 degrees, or 180 degrees) (NO in a1512).

[0154] The display control unit 1251 controls the display unit 1250 to display a request for the specification of the length L (0≦L≦1) after projecting either the first axis (x axis) or the second axis (y axis) onto a plane parallel to the GUI layer 104 (XY plane) by orthogonal projection (a1513).

[0155] The input processing unit 1211 determines whether or not a user operation has been input to the input unit 1210 (a1515). If no user operation has been input to the input unit 1210 (NO in a1515), the input processing unit 1211 executes a1515 again. If a user operation has been input to the input unit 1210 (YES in a1515), the input processing unit 1211 proceeds to a1516.

[0156] The input processing unit 1211 determines whether the user has completed specifying the request (a1516). If the user has not completed specifying the request (NO in a1516), the input processing unit 1211 executes a1515 again. If the user has completed specifying the request (YES in a1516), the input processing unit 1211 proceeds to a1517.

[0157] The shared image object coordinate system generation unit 1228 calculates the coordinate system xyz (a1517). For example, if the shared image object coordinate system generation unit 1228 obtains the origin of the coordinate system xyz and the directions of the projected axes (x-axis and y-axis in the XY plane), it calculates two axes (x-axis and y-axis) in the world coordinate system XYZ based on the origin of the coordinate system xyz and the directions of the projected axes (x-axis and y-axis in the XY plane), and generates the coordinate system xyz based on the two axes (x-axis and y-axis) in the world coordinate system XYZ (a1517).

[0158] On the other hand, the shared image object coordinate system generation unit 1228, upon obtaining the origin of the coordinate system xyz, the directions of the projected axes (x-axis and y-axis in the XY plane), and the projected length L (0≦L≦1) obtained by projecting either the first axis (x-axis) or the second axis (y-axis) onto a plane parallel to the GUI layer 104 (XY plane), calculates the two axes (x-axis and y-axis) in the world coordinate system XYZ based on the origin of the coordinate system xyz, the directions of the projected axes (x-axis and y-axis in the XY plane), and the projected length L, and generates the coordinate system xyz based on the two axes (x-axis and y-axis) in the world coordinate system XYZ (a1517).

[0159] The shared image object coordinate system generation unit 1228 stores a set of information regarding the xyz coordinate system, csId, csPosition, and csRotation, in the shared image object related database 1224 (a1518). This allows the shared image object coordinate system generation unit 1228 to associate the xyz coordinate system with the shared image object.

[0160] When the shared image object coordinate system generation unit 1228 uniquely determines the xyz coordinate system, the axes of the xyz coordinate system are displayed by the display unit 1250. Furthermore, if the user inputs an operation to select the xyz coordinate system, the permissible movement direction of the 3D object may be specified in the axis direction of the xyz coordinate system.

[0161] The above describes an example of the operation of the shared image object function extension system 120 according to the second embodiment of the present invention.

[0162] (2-4. Effects) According to a second embodiment of the present invention, based on input of an operation relating to a shared image object, processing relating to the shared image object is performed in a direction different from the direction parallel to the plane of the shared image object. This improves the expressiveness of the shared image object and supports more effective interaction between the shared image object and 3D objects.

[0163] For example, the shared image object coordinate system generation unit 1228 calculates the two axes (x and y axes) in the world coordinate system XYZ based on the projected axis directions (x and y axes in the XY plane) obtained by projecting two mutually orthogonal axes (x and y axes) in the world coordinate system XYZ onto the GUI layer 104, calculates the coordinate system xyz based on the two axes (x and y axes), and associates the coordinate system xyz with the shared image object. This has the effect of making it possible to align the allowable direction of movement of the 3D object with the axis direction of the coordinate system xyz associated with the shared image object.

[0164] The effects of the second embodiment of the present invention have been described above.

[0165] (3. Hardware Configuration Example) Next, an example of the hardware configuration of the shared image object function extension system 20 according to an embodiment of the present invention will be described.

[0166] In the following, an example of the hardware configuration of the information processing device 900 will be described as an example of the hardware configuration of the shared image object function extension system 20 according to an embodiment of the present invention. Note that the example of the hardware configuration of the information processing device 900 described below is merely one example of the hardware configuration of the shared image object function extension system 20. Therefore, the hardware configuration of the shared image object function extension system 20 may be modified by removing unnecessary components from the hardware configuration of the information processing device 900 described below, or by adding new components.

[0167] Figure 19 shows the hardware configuration of an information processing device 900 as an example of a shared image object function extension system 20 according to an embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0168] The CPU 901 functions as both an arithmetic processing unit and a control unit, controlling the overall operation of the information processing unit 900 according to various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as needed during its execution. These are interconnected by a host bus 904, which consists of a CPU bus and other components.

[0169] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. It is not always necessary to configure the host bus 904, bridge 905, and external bus 906 separately; these functions may be implemented on a single bus.

[0170] The input device 908 consists of input means for the user to input information, such as a mouse, keyboard, touch panel, buttons, microphone, switches, and levers, and an input control circuit that generates input signals based on the user's input and outputs them to the CPU 901. The user operating the information processing device 900 can input various types of data to the information processing device 900 or instruct it to perform processing operations by operating this input device 908.

[0171] The output device 909 includes, for example, display devices such as CRT (Cathode Ray Tube) display devices, liquid crystal display (LCD) devices, OLED (Organic Light Emitting Diode) devices, lamps, and audio output devices such as speakers.

[0172] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. The storage device 910 is composed of, for example, an HDD (Hard Disk Drive). This storage device 910 drives the hard disk and stores programs executed by the CPU 901 and various data.

[0173] The communication device 911 is a communication interface composed of, for example, a communication device for connecting to a network. The communication device 911 may support either wireless or wired communication.

[0174] The above describes an example of the hardware configuration of the shared image object function extension system 20 according to an embodiment of the present invention.

[0175] (4. Various variations) Although preferred embodiments of the present invention have been described in detail above with reference to the attached drawings, the present invention is not limited to these examples. It is clear to any person with ordinary skill in the art to which the present invention belongs that various modifications or alterations can be conceived within the scope of the technical idea described in the claims, and these are also understood to fall within the technical scope of the present invention.

[0176] For example, in the second embodiment of the present invention, the case in which the origin of the xyz coordinate system associated with a shared image object is specified by the user was mainly described. Furthermore, the case in which the centroid of the region drawn on the texture of the shared image object is obtained by image analysis and specified as the origin of the xyz coordinate system was also described. In this case, the centroid of a region of a predetermined size or larger may be obtained by image analysis and specified as the origin of the xyz coordinate system.

[0177] Furthermore, in the second embodiment of the present invention, the case in which two mutually orthogonal axes (x-axis and y-axis) in the world coordinate system XYZ are projected by orthogonal projection onto a plane parallel to the UI layer 104 (XY plane) and the resulting axis direction is specified is mainly described. However, the projected axis directions may be specified by projecting two mutually orthogonal axes (x-axis and y-axis) in the world coordinate system XYZ onto the GUI layer 104 using perspective projection.

[0178] Similarly, in the second embodiment of the present invention, the case where the length L after projection is specified is obtained by orthogonally projecting one of two mutually orthogonal axes (x-axis and y-axis) in the world coordinate system XYZ onto a plane parallel to the GUI layer 104 (XY plane). However, the length L after projection is obtained by projecting one of two mutually orthogonal axes (x-axis and y-axis) in the world coordinate system XYZ onto the GUI layer 104 using perspective projection may also be specified. [Explanation of Symbols]

[0179] 20, 120 Shared Image Object Functionality Extension System 210, 1210 Input section 211, 1211 Input Processing Unit 222, 1222 Shared Image Object Management Department 240 3D Object Processing Unit 224, 1224 Shared Image Object Related Databases 225 Shared image storage department 226 Shared Image Object Transformation Processing Unit 227 Texture Mapping Processing Unit 250, 1250 display section 251, 1251 Display Control Unit 1228 Shared Image Object Coordinate System Generation Unit

Claims

1. A two-dimensional object placement unit that places two-dimensional objects, which are planar objects, into a three-dimensional digital space, A two-dimensional object processing unit that, based on input of a predetermined operation relating to the two-dimensional object, performs a predetermined process relating to the two-dimensional object in a direction different from the direction parallel to the plane of the two-dimensional object, An information processing device equipped with the following features.

2. The aforementioned predetermined operation includes a region designation operation that designates a portion of the two-dimensional object as a designated region, The predetermined process includes a process of deforming the two-dimensional object by moving the designated area in the three-dimensional digital space in the different directions. The information processing apparatus according to claim 1.

3. The aforementioned different directions are the normal directions of the two-dimensional object with respect to the plane in the three-dimensional digital space. The information processing apparatus according to claim 2.

4. The predetermined operation includes a movement amount specification operation that specifies the amount of movement of the specified area, The two-dimensional object processing unit moves the specified area by the specified amount. The information processing apparatus according to claim 2.

5. The aforementioned information processing device is The system includes a three-dimensional object processing unit that, based on movement operations of the three-dimensional object existing in the three-dimensional digital space, can move the three-dimensional object such that part or all of the area of ​​the three-dimensional object is located behind the designated area with respect to the position of the virtual camera. The information processing apparatus according to claim 2.

6. The aforementioned predetermined operation includes an axial direction specification operation that specifies the projected axial direction obtained by projecting two mutually orthogonal axes in the three-dimensional digital space onto a plane parallel to the layer for generating visual information, The predetermined process includes calculating the two axes in the three-dimensional digital space based on the projection axis direction, calculating a three-dimensional Cartesian coordinate system based on the two axes, and associating the three-dimensional Cartesian coordinate system with the two-dimensional object. The information processing apparatus according to claim 1.

7. The two-dimensional object processing unit, when the angle formed by the projected axes is of a predetermined magnitude, obtains a length specification operation that specifies the length after projection of one of the two axes onto the layer, and calculates the three-dimensional Cartesian coordinate system based on the projected axis and the projected length. The information processing apparatus according to claim 6.

8. The aforementioned predetermined operation includes a coordinate specification operation that specifies the coordinates inside the two-dimensional object as specified coordinates, The two-dimensional object processing unit calculates the three-dimensional Cartesian coordinate system with the specified coordinates as the origin. The information processing apparatus according to claim 6.

9. The aforementioned information processing device is The system includes a three-dimensional object processing unit that can move a three-dimensional object in the axis direction of the three-dimensional Cartesian coordinate system based on a movement operation of the three-dimensional object existing in the three-dimensional digital space. The information processing apparatus according to claim 6.

10. Placing a two-dimensional object, which is a planar object, in a three-dimensional digital space, Based on the input of a predetermined operation concerning the two-dimensional object, a predetermined process concerning the two-dimensional object is performed in a direction different from the direction parallel to the plane of the two-dimensional object. A computer-based information processing method, including [a specific example].

11. Computers, A two-dimensional object placement unit that places two-dimensional objects, which are planar objects, into a three-dimensional digital space, A two-dimensional object processing unit that, based on input of a predetermined operation relating to the two-dimensional object, performs a predetermined process relating to the two-dimensional object in a direction different from the direction parallel to the plane of the two-dimensional object, A program that makes it function as such.

Images