Object attitude control program and object attitude control system

Abstract

To allow for easily controlling posture of an object.SOLUTION: An object posture control program makes an information processing device perform steps of displaying an object with multiple interlocking parts placed in a virtual space on a display unit, and controlling posture of the object to move a target part of the multiple interlocking parts according to information on movement of the information processing device. With the object posture control program, the information processing device may include the display unit.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an object posture control program and an object posture control system.

Background Art

[0002] In recent years, with the development of electronic devices, it has become possible to display high-definition still images and moving images on terminals such as smartphones and personal computers. Furthermore, research and development of information processing technologies have also advanced, and display technologies that fuse virtual spaces and real spaces using augmented reality (AR) are becoming practical. For example, Patent Document 1 discloses a technique for displaying an object arranged in a virtual space in a display area of an information terminal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] On the other hand, controlling the posture of an object displayed in a display area may involve complex input processing in specifying a target part.

[0005] In view of such problems, one object of the present invention is to easily control the posture of an object arranged in a virtual space.

Means for Solving the Problems

[0006] According to one embodiment of the present invention, an object attitude control program is provided which causes an information processing device to display an object having a plurality of interconnected parts arranged in a virtual space on a display unit, and to control the attitude of the object so as to move some of the target parts among the plurality of interconnected parts based on information about the movement of the information processing device.

[0007] In the object attitude control program described above, the information processing device has a setting unit, and the setting unit may set a target part of the plurality of interconnected parts based on information input when a part of the object displayed on the display unit is touched.

[0008] In the object attitude control program described above, the setting unit sets a first movable target part of the object, which is a part of the plurality of interconnected parts, and the second target part of the plurality of interconnected parts is set by the setting unit based on the set first target part, and the first target part may move relative to the second target part.

[0009] In the object attitude control program described above, when the first target part reaches the limit of its range of motion, it may be configured to connect to the second target part, set a third target part on the opposite side of the first target part relative to the second target part, and move the second target part of the object relative to the third target part.

[0010] In the object attitude control program described above, when setting the first target part or the second target part of an object displayed on the display unit, a part in which multiple parts overlap is specified, the display direction of the object may be changed.

[0011] In the object attitude control program described above, when setting the first target part or the second target part of an object displayed on the display unit, a part that satisfies predetermined conditions may be set as the first target part or the second target part.

[0012] In the object attitude control program described above, when setting the first target part or the second target part of an object displayed on the display unit, a portion where multiple parts overlap is specified, the first target part or the second target part may be set based on a predetermined operation on the overlapping portion.

[0013] In the object attitude control program described above, a movable fourth target part of the object, different from the first and second target parts, may be set, a fifth target part connected to the fourth target part may be set, and based on information regarding the movement of the information processing device, the first target part of the object may move relative to the second target part, and the fourth target part may move relative to the fifth target part.

[0014] In the object attitude control program described above, when the first target part of the object moves and comes into contact with a movable part different from the first target part and the second target part, it may be set as a fourth target part, a fifth target part connected to the fourth target part may be set, and the fourth target part may move based on the movement of the first target part, with the fifth target part as the reference point.

[0015] In the object posture control program described above, when the information processing device moves while the first target part or the second target part is touched, the first target part may move relative to the second target part of the object in the virtual space while the displayed object remains stationary.

[0016] According to one embodiment of the present invention, an object attitude control system is provided for controlling the attitude of an object placed in a virtual space, wherein the object having a plurality of interlocking parts is displayed on a display unit, and the attitude of the object is controlled to move a portion of the plurality of interlocking parts based on information regarding the movement of an information processing device.

[0017] According to one embodiment of the present invention, an information processing device is provided for controlling the orientation of an object placed in a virtual space, the device displays the object having a plurality of interconnected parts on a display unit, and controls the orientation of the object to move some of the plurality of interconnected parts based on information regarding the movement of the information processing device. [Effects of the Invention]

[0018] By using the present embodiment, the orientation of an object can be easily controlled. [Brief explanation of the drawing]

[0019] [Figure 1] This figure shows the hardware configuration in an object attitude control system according to the first embodiment of the present invention. [Figure 2] This is a functional block diagram of an object attitude control unit according to the first embodiment of the present invention. [Figure 3] This is a flowchart of the object attitude control process according to the first embodiment of the present invention. [Figure 4] This is a flowchart of the first object attitude control process according to the first embodiment of the present invention. [Figure 5] This is an example of alignment between coordinates in a virtual space and coordinates displayed on an information processing device in a first object attitude control process according to the first embodiment of the present invention. [Figure 6] This is an example of the coordinates of each part of an object according to the first embodiment of the present invention. [Figure 7]This is an example of a user interface displayed on the display unit of an information processing apparatus in the first object posture control process according to the first embodiment of the present invention. [Figure 8] This is a flowchart of the second object posture control process according to the first embodiment of the present invention. [Figure 9] This is an example of a user interface displayed on the display unit of an information processing apparatus in the second object posture control process according to the first embodiment of the present invention. [Figure 10] This is an example of a user interface displayed on the display unit of an information processing apparatus in the second object posture control process according to the first embodiment of the present invention. [Figure 11] This is a flowchart of the third object posture control process according to the first embodiment of the present invention. [Figure 12] [[ID=1S]]This is an example when the position of the information processing apparatus in the third object posture control process according to the first embodiment of the present invention is changed. [Figure 13] These are the position coordinates of the terminal before and after movement and the position coordinates of the object in the third object posture control process according to the first embodiment of the present invention. [Figure 14] This is an example of a user interface displayed on the display unit of an information processing apparatus in the third object posture control process according to the first embodiment of the present invention. [Figure 15] This is an example of a user interface displayed on the display unit of an information processing apparatus according to the second embodiment of the present invention. [Figure 16] This is an example of a user interface displayed on the display unit of an information processing apparatus according to the second embodiment of the present invention. [Figure 17] This is a functional block diagram of an object posture control unit according to the third embodiment of the present invention. [Figure 18] This is a flowchart of the third object posture control process according to the third embodiment of the present invention. [Figure 19] This is a data structure showing the movable regions of each part. [Figure 20]This is an example of a user interface displayed on the display unit of an information processing device in the third object attitude control processing according to the third embodiment of the present invention. [Figure 21] This is an example of a user interface displayed on the display unit of an information processing device in the third object attitude control processing according to the third embodiment of the present invention. [Figure 22] This is a functional block diagram of the object attitude control unit according to the fourth embodiment of the present invention. [Figure 23] This is a flowchart of the second object attitude control process according to the fourth embodiment of the present invention. [Figure 24] This is an example of a modified user interface displayed on the display unit of an information processing device in the second object attitude control processing according to the fourth embodiment of the present invention. [Figure 25] This is a flowchart of the third object attitude control process according to the fourth embodiment of the present invention. [Figure 26] This is an example of changing the position of the information processing device in the third object attitude control processing according to the fourth embodiment of the present invention. [Figure 27] This is an example of a user interface displayed on the display unit of an information processing apparatus according to the fourth embodiment of the present invention. [Figure 28] This is an example of a user interface displayed on the display unit of an information processing apparatus according to the fifth embodiment of the present invention. [Figure 29] This is an example of a user interface displayed on the display unit of an information processing apparatus according to the fifth embodiment of the present invention. [Figure 30] This is an example of a user interface displayed on the display unit of an information processing apparatus according to the fifth embodiment of the present invention. [Figure 31] This is an example of a user interface displayed on the display unit of an information processing apparatus according to the sixth embodiment of the present invention. [Figure 32] This is an example of a user interface displayed on the display unit of an information processing apparatus according to the sixth embodiment of the present invention. [Figure 33]This is a data structure of the coordinates of the first target part of an object before and after movement of the terminal according to the seventh embodiment of the present invention. [Figure 34] This is an example of a user interface displayed on the display unit of an information processing apparatus according to the seventh embodiment of the present invention. [Figure 35] This is a modified example of the hardware configuration in the object attitude control system according to the first embodiment of the present invention. [Modes for carrying out the invention]

[0020] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention can be implemented in many different ways, and is not limited to the embodiments described below. The drawings may be schematic in order to clarify the explanation, but they are merely examples and do not limit the interpretation of the present invention. In addition, the letters "1st" and "2nd" attached to each element are convenient indicators used to distinguish each element and have no further meaning unless specifically explained. In the drawings referenced in this embodiment, the same part or parts having similar functions are denoted by the same or similar symbols (simply the number xxx with A or B attached), and repeated explanations may be omitted. Also, some parts of the configuration may be omitted from the drawings. Furthermore, if something can be recognized by a person with ordinary skill in the art to which the present invention belongs, no further explanation will be given.

[0021] In one embodiment of the present invention, the term "target part" refers to a movable part of an object, and a linked part is formed by combining multiple target parts. The first target part corresponds to the movable part among the linked parts. The second target part corresponds to the part among the linked parts that acts as a pivot point for the movement of the first target part based on the set first target part.

[0022] <First Embodiment> An object attitude control system according to the first embodiment of the present invention will be described in detail with reference to the drawings.

[0023] (1-1. Hardware configuration of the object attitude control system) Figure 1 shows the hardware configuration and functional block diagram of the object attitude control system 1. As shown in Figure 1, the object attitude control system 1 includes a terminal 10 and a server 20. The terminal 10 and server 20 together are sometimes referred to as the information processing device.

[0024] Terminal 10 is a computer and includes a display unit 11, a control unit 12, a storage unit 13, an operation unit 14, a communication unit 15, a sensor unit 16, a camera unit 17, and a microphone unit 18. In this example, a smartphone is used as terminal 10. However, it is not limited to smartphones and can also be a mobile phone (feature phone), a tablet device, a notebook PC (personal computer), or an IoT device (a device equipped with a power supply, communication function, and information storage mechanism), as long as it can communicate with the server 20 via a network.

[0025] The display unit 11 is a display device such as a liquid crystal display or an organic EL display, and its display content is controlled by signals input from the control unit 12.

[0026] The control unit 12 includes a CPU (Central Processing Unit) and RAM. This is achieved by using Random Access Memory (Random Access Memory), an Application Specific Integrated Circuit (ASIC), or a Flexible Programmable Gate Array (FPGA). The control unit 12 executes an application, including an object attitude control program, stored in the memory unit 13, based on the operation of the display unit 11 and the operation unit 14.

[0027] The storage unit 13 functions as a database for storing object attitude control programs and spatial information used in the object attitude control programs. The storage unit 13 can use memory, an SSD, or a storage-capable element.

[0028] The operation unit 14 includes a controller, buttons, or switches. When the operation unit 14 is moved up, down, left, or right, pressed, or rotated, information based on that action is transmitted to the control unit 12. In this embodiment, since the terminal 10 is a display device (touch panel) having a touch sensor, the display unit 11 and the operation unit 14 may be placed in the same location. This makes it possible to specify a target part of an object while displaying the object, thus facilitating object posture control.

[0029] The communication unit 15 has the function of sending and receiving data with the server 20. A LAN transceiver (e.g., a Wi-Fi transceiver) is used in the communication unit 15. However, the transceiver is not limited to a LAN transceiver. If the terminal is a portable terminal, a transceiver for mobile terminal communication (e.g., LTE communication) or a transceiver for short-range wireless communication may be provided. The terminal 10 is connected to the server 20 via the network 50.

[0030] The sensor unit 16 has the function of detecting the location information of the terminal. The sensor unit 16 has GPS (Global Positional System), including accelerometer and gyroscope (angular velocity sensor).

[0031] The camera unit 17 has the function of capturing environmental images and captures objects that have markers for displaying objects. For example, a CMOS image sensor is used in the camera unit 17.

[0032] The microphone unit 18 has the function of capturing spoken information from the user and inputting it to the control unit 12 or the storage unit 13.

[0033] Server 20 includes a communication unit 21, a storage unit 22, a control unit 23, and a display unit 24. Server 20 functions as a database and application server. However, if all information related to the object attitude control program is already loaded onto terminal 10, server 20 may not necessarily be used.

[0034] The communication unit 21 has a transceiver and communicates information about object control information with the terminal 10 via the network 50. The communication unit 21 uses an intranet LAN transceiver. However, the transceiver is not limited to an intranet LAN transceiver, and any device capable of communication similar to the terminal 10 can be used.

[0035] The storage unit 22, by using a hard disk and an SSD, functions as a database for information used in the object attitude control program.

[0036] The control unit 23 controls the processing of the object attitude control program using a CPU, RAM, ASIC, or FPGA. Furthermore, a user interface for executing the object attitude control program may be provided to the display unit 24 based on instructions from the control unit 23.

[0037] (1-2. Configuration of the object attitude control unit 100) Figure 2 shows a functional block diagram of the object attitude control unit 100, which is composed of the various components of the terminal 10 and the server 20 in the object attitude control system 1, and controls the program that implements the object attitude control function (object attitude control program). In this example, we will describe a case where the object attitude control program is installed on the terminal.

[0038] The object posture control unit 100 includes a captured image acquisition unit 110, a marker detection unit 120, a spatial definition unit 130, an object display unit 140, a first target area setting unit 150, a second target area setting unit 160, a linked area determination unit 170, a movement information acquisition unit 180, and an object coordinate change unit 190.

[0039] The image acquisition unit 110 has the function of acquiring environmental images of the real space captured by the camera unit 17.

[0040] The marker detection unit 120 has the function of detecting markers from images acquired by the captured image acquisition unit 110. In this example, marker information is stored in the storage unit 13 in advance. Note that marker information does not necessarily have to be stored in the storage unit 13; feature points that can serve as markers may be extracted from the captured image.

[0041] The spatial definition unit 130 has the function of aligning the coordinates of the virtual space with those of the display unit 11. In this embodiment, the spatial definition unit 130 uses a marker-based alignment method to align the coordinates of the virtual space with those of the display unit 11 based on the detected markers.

[0042] The object display unit 140 has the function of displaying objects whose posture is to be controlled, which are placed in the virtual space, on the display unit 11.

[0043] The first target part setting unit 150 has the function of setting a movable first target part from among a plurality of interconnected parts. The second target part setting unit 160 has the function of setting a second target part based on the set first target part. In this embodiment, the first target part setting unit 150 and the second target part setting unit 160 can be collectively referred to as the setting unit, and the setting unit is a general term for the functional unit that sets a plurality of interconnected parts of an object. When it is not necessary to describe the first target part setting unit 150 and the second target part setting unit 160 separately, they will be described as the setting unit.

[0044] The interlocking part determination unit 170 has the function of determining whether multiple parts set in the setting unit are interlocked. The determination of whether or not they are interlocked may be made using information that has been incorporated into the object attitude control program in advance, or it may be made based on signals input from the user.

[0045] The movement information acquisition unit 180 has the function of acquiring movement information based on the coordinates of the terminal 10 before and after movement, which are input from the sensor unit 16.

[0046] The object coordinate modification unit 190 has the function of changing and recording the coordinates of some of the target parts among the linked parts in order to display the object after it has moved on the display unit 11, and transmitting the position information to the object display unit 140. The movement information acquisition unit 180, the object coordinate modification unit 190, and the object display unit 140 acquire movement information of target parts of the object 60 times per second, and by changing the spatial coordinates and coordinates on the display screen of the target parts, the movement (change in posture) of the object can be displayed as a still image or a video.

[0047] (1-3. Object posture control processing) Next, the object attitude control process based on instructions from the object attitude control program in the object attitude control unit 100 will be described. Figure 3 is a flowchart of the object attitude control process. As shown in Figure 3, the object attitude control process includes a first object attitude control process S100, a second object attitude control process S200, and a third object attitude control process S300.

[0048] The first object attitude control process S100 includes image acquisition and display processing, marker information detection processing, spatial definition processing, and superimposed display processing of the captured image and object. The second object attitude control process S200 includes movement mode determination processing, first target part information input processing, first target part setting processing, second target part information input processing, second target part setting processing, and determination processing of whether the parts are linked. The third object attitude control process S300 includes terminal movement amount acquisition processing, object coordinate change processing, display processing of the object after movement, and determination processing of whether the terminal movement has finished. Each object attitude control process will be explained separately.

[0049] (1-3-1. First object attitude control processing S100) Figure 4 shows the first object attitude control process S100. The first object attitude control process S100 is started when an application containing an object attitude control program is launched.

[0050] In the first object orientation processing S100, the captured image acquisition unit 110 captures an image of the real-world environment, acquires the captured image (S110), and displays the captured image on the display unit 11 of the terminal 10.

[0051] Figure 5 shows an example of how terminal 10 captures a real space. In Figure 5, terminal 10, a table 60, and a sheet 70 are placed in the real space. The sheet 70 contains four markers 75. In this example, terminal 10 captures a portion of table 60 and the sheet 70 so that the four markers 75 are included.

[0052] Let's return to Figure 4 for explanation. Next, the marker detection unit 120 performs marker detection processing (S120). In this example, information for four markers is pre-registered in the object attitude control program. The marker information concerns the shape and arrangement of the markers. If no four markers are detected (S120; No), the process returns to acquiring and displaying the captured image.

[0053] Next, if four markers are detected (S120; Yes), the spatial definition unit 130 performs coordinate alignment (spatial definition) between the real space and the virtual space using the marker-based alignment method (S130). At this time, the spatial definition unit 130 compares the marker 75 in the real space displayed on the display unit 11 with the marker in the virtual space stored in the object attitude control program, and adjusts it so that the coordinates in the virtual space can be displayed on the display unit 11 as coordinates in the real space. For example, the position coordinates of marker 75-1 are defined as follows: the position coordinates in the real space are (Xr75-1, Yr75-1, Zr75-1), the position coordinates on the display unit 11 are (Xd75-1, Yd75-1), and the position coordinates in the virtual space are (Xv75-1, Yv75-1, Zv75-1).

[0054] Next, the spatial definition unit 130 performs the process of acquiring object coordinates in the virtual space (S140). Figure 6 shows the coordinates of each part of the object in the virtual space. As shown in Figure 6, the coordinate information of each part of the object is set relative to an origin that is set based on the coordinates of the four markers. Through the above process, the coordinates of each part of the object are obtained by acquiring the information on the display unit 11 of the marker 75.

[0055] Next, the object display unit 140 displays the object image superimposed on the captured image (S150). Figure 7 shows the user interface of the object displayed on the display unit 11 of the terminal 10. In Figure 7, the object 80 is placed and displayed on the sheet 70. In this example, the object 80 has a human-like body shape and has multiple parts 81 and multiple bones 82. Parts 81 are movable parts and correspond to human joints. Part 81 (part 81a) is connected to the adjacent part 81 (part 81b) via bones 82. In other words, parts 81a and 81b can be said to be interconnected parts. Note that the sheet 70 does not need to be displayed when the object 80 is displayed. With this, the first object posture control process S100 is completed.

[0056] (1-3-2. Second object attitude control processing S200) Figure 8 shows a flowchart of the second object attitude control process S200. The second object attitude control process S200 is performed upon completion of the first object attitude control process S100.

[0057] First, it is determined whether the device is in movement mode (S210). Whether it is in movement mode can be determined by whether the sensor unit 16 detects movement of the terminal 10, or it can be determined based on input from the user. If it is in movement mode (S210; Yes), the second object attitude control process is terminated.

[0058] If the device is not in movement mode (S210; No), information about the first target part is input to the first target part setting unit 150 as a target part of the linked parts (S220). Figure 9 shows a user interface that illustrates an example of inputting information about the first target part. As shown in Figure 9, when inputting information about the first target part, the message "Please touch the first target part." may be displayed on the display screen. In this example, when inputting information about the first target part, the user touches the part of the object displayed on the display unit 11 that corresponds to part 81a. Part 81a corresponds to the wrist joint of object 80. As described above, the terminal 10 has a touch panel, and when part 81a is touched, the position information of part 81a is input to the first target part setting unit 150.

[0059] When entering information for body part 81a, you may either trace around the area corresponding to body part 81a, or you may enclose the area corresponding to body part 81a with multiple fingers.

[0060] Let's return to Figure 8 for explanation. Next, based on the input information, the first target area setting unit 150 sets the touched area 81a as the first target area (S230).

[0061] Next, information about the second target area is input to the second target area setting unit 160 (S220). Figure 10 shows a user interface that illustrates an example of inputting information about the second target area. As shown in Figure 10, when inputting information about the second target area, the message "Please touch the second target area." is displayed, similar to the case of the first target area, and the user touches the part of the object displayed on the display unit 11 that corresponds to the part 81b which will be the second target area. Part 81b corresponds to the elbow joint of object 80. At this time, similar to part 81a, when part 81b is touched, the position information of part 81b is input to the second target area setting unit 160.

[0062] Let's return to Figure 8 for explanation. Next, based on the input information, the second target area setting unit 160 sets the touched area 81b as the second target area (S250).

[0063] Next, the linked part determination unit 170 determines whether part 81a and part 81b move in conjunction. If it is determined that part 81a and part 81b are not linked parts (S260; No), the information of the first target part is input again (S220). In this example, since part 81a and part 81b are connected via bone 82a, it is determined that they are linked parts (S260; Yes). With this, the second object posture control process S200 is completed.

[0064] (1-3-3. Third object attitude control processing S300) Figure 11 is a flowchart of the third object attitude control process S300. The third object attitude control process S300 is started when the second object attitude control process S200 is completed.

[0065] In the third object attitude control process S300, first, the movement information acquisition unit 180 acquires the amount of movement of the terminal 10 (also called information about movement) (S310). In this example, tracking of the object's attitude in accordance with changes in the terminal 10's position information is performed using the V-SLAM (Visual Simultaneous Localization and Mapping) method. Figure 12 shows an example of when the terminal 10 moves in accordance with the movement of the user 90. When the terminal 10 moves, the user does not need to be in contact with the display unit 11 of the terminal 10. It is measured using the sensor unit 16 and acquired by the movement information acquisition unit 180. The sensor unit 16 can be a GPS, an acceleration sensor, or an angular velocity sensor. Figure 13 (A) is a data structure that shows the initial value (before movement), the position coordinates of terminal 10 after time t1 (after movement), and the amount of movement of terminal 10 obtained from the coordinates before and after movement.

[0066] Next, based on the acquired amount of movement, the object coordinate modification unit 190 modifies and records the coordinates of the first target part (part 81a), which is the part of the object 80 that is to be moved (S320). Figure 13(B) shows the data structure of the coordinates of the first target part (part 81a) of the object in virtual space and the coordinates in the display unit 11 after time t1. As shown in Figure 13(B), based on the movement information of the terminal 10, the information of the first target part (part 81a) of the object 80 in virtual space and the first target part of the object 80 in the display unit 11 The coordinate information for (part 81a) is changed, and the changed coordinate information is stored (the coordinate information is rewritten).

[0067] Let's return to Figure 11 for explanation. Next, the object display unit 140 displays the object 80 on the display unit 11 using information about the first target part after movement (after time t1) (S330). Figure 14 is a user interface showing the object 80 with the position of the first target part changed on the display unit 11. As shown in Figure 14, the first target part (part 81a) moves from the position shown in Figure 7 with respect to the second target part (part 81b). In other words, based on information about the movement of the information processing device, some of the target parts among multiple interconnected parts can be moved. The above process is repeated as long as the terminal 10 is moving (S340; Yes). In this example, the amount of movement of the terminal is acquired 60 times per second, so the change in the object's posture is displayed smoothly. When the movement of the terminal 10 is finished (S340; No), the third object posture control process S300 is terminated.

[0068] As described above, by using the object attitude control program of this embodiment, the attitude of an object can be easily controlled without complex control by setting the interconnected parts and moving the terminal.

[0069] <Second Embodiment> This embodiment describes an object attitude control unit and object attitude control method that differ from the first embodiment. Specifically, an example in which the pivot point for moving the first target part is different will be described. Note that configurations and methods similar to those in the first embodiment may be omitted as appropriate.

[0070] Figure 15 shows a user interface that illustrates an example of inputting information for the second target area. As shown in Figure 15, the user may input information for the second target area by touching the area corresponding to part 81c and setting it as the second target area. In this case, although parts 81a and 81c are separated, they can be determined to be interconnected parts by pre-programming bone 82a, part 81b, and bone 82b to be considered as a single bone.

[0071] As a result, by using this embodiment, when the terminal 10 is moved, the attitude of the object can be controlled by moving part 81a with respect to part 81c, as shown in Figure 16.

[0072] If bone 82a, part 81b, and bone 82b are not considered to be fixed together as a single bone, then part 81b may be moved in conjunction with the movement of part 81a. In this case, since part 81b is uncontrolled, it may move simultaneously with part 81a or with a delay.

[0073] <Third Embodiment> This embodiment describes an object attitude control unit and object attitude control method that differ from the first embodiment. Specifically, it describes the process of setting the third target part when the movement range of the first target part reaches its limit point during the third object attitude control process S300. Note that configurations and methods similar to those in the first and second embodiments may be omitted as appropriate.

[0074] (3-1. Configuration of the object attitude control unit 100A) Figure 17 shows a functional block diagram of the object attitude control unit 100A in the object attitude control system 1A. As shown in Figure 17, the object attitude control unit 100A includes an image acquisition unit 110, a marker detection unit 120, a spatial definition unit 130, an object display unit 140, a first target area setting unit 150, a second target area setting unit 160, a linked area determination unit 170, a movement information acquisition unit 180, and an object coordinate change unit 190, in addition to a limit point detection unit 195 and a third target area setting unit 200.

[0075] The limit point detection unit 195 has the function of detecting the movement of the first target part and inputting information to the third target part setting unit 200 when the first target part reaches the limit point of the movable range. The third target part setting unit 200 has the function of setting the third target part based on the signal input from the limit point detection unit 195. When the third target part is set, the second target part switches from a pivot point to a moving part, and the second target part moves with the third target part as the reference point.

[0076] (3-2. Object posture control processing) Figure 18 shows a flowchart (S300A) of the second object attitude control process based on commands from the object attitude control program in the object attitude control unit 100A. As shown in Figure 18, terminal 10A is moved, the amount of movement of terminal 10A is obtained (S310), and the coordinates of the first target part (part 81a) in the virtual space and the coordinates on the display unit 11 are changed (S320).

[0077] Figure 19 shows the data structure indicating the range of motion of each part. As shown in Figure 19, the relative position coordinates of the first target part are set as the range of motion of the first target part relative to the second target part. At this time, if the limit point detection unit 195 determines that it does not exceed the limit point of the range of motion of the first target part (part 81a in this example) (S325; No), it displays the object in the same manner as in the first embodiment (S330). On the other hand, as shown in Figure 20, if it determines that it does exceed the limit point of the range of motion of the first target part (S325; Yes), the limit point detection unit 195 inputs that information to the third target part setting unit 200. Based on the input information, the third target part setting unit 200 sets the part connected to the second target part and located on the opposite side of the first target part relative to the second target part as the third target part (S350). In this example, part 81c (corresponding to the shoulder joint), located on the opposite side of part 81a (corresponding to the wrist joint), which corresponds to the first target part, is set as the third target part to part 81b (corresponding to the elbow joint), which corresponds to the second target part. Part 81c is connected to part 81b via bone 82b. In this embodiment, part 81c is set automatically. If it is not set automatically, the third target part may be set by the user after inputting information about the third target part, similar to the first target part. In this case, it may be determined whether it is a linked part.

[0078] Next, after setting the third target part, the amount of movement of terminal 10 is obtained (S360), the coordinates of the second target part (part 81b) are changed (S370), and based on the changed coordinates of the second target part, the object in which the second target part (part 81b) has moved relative to the third target part (part 81c) is displayed as shown in Figure 21 (S380). Thus, by using this embodiment, the posture of the object can be easily controlled even when the moving part reaches the limit of its range of motion.

[0079] Furthermore, the same process as described above may be repeated whenever a moving part, which moves relative to a fixed part, detects the limit of its range of motion. In other words, when the (n-1)th target part moves relative to the nth target part (where n is a natural number greater than or equal to 2), and the (n-1)th target part reaches the limit of its range of motion, it may be connected to the nth target part, and the (n+1)th target part, which is located on the opposite side of the (n-1)th target part, may be set. This allows for smoother control of the object's posture.

[0080] <Fourth Embodiment> This embodiment describes an object attitude control process that differs from that of the first embodiment. Specifically, it describes an example in which multiple parts move based on the movement of a single terminal. Note that configurations and methods similar to those of the first to third embodiments may be omitted as appropriate.

[0081] (4-1. Configuration of the server and object attitude control unit 100B) Figure 22 shows a functional block diagram of the object attitude control unit 100B in the object attitude control system 1B. As shown in Figure 22, the object attitude control unit 100B includes an image acquisition unit 110, a marker detection unit 120, a spatial definition unit 130, an object display unit 140, a first target part setting unit 150, a second target part setting unit 160, a linked part determination unit 170, a movement information acquisition unit 180, and an object coordinate change unit 190, as well as a linked part addition unit 210, a fourth target part setting unit 220, and a fifth target part setting unit 230.

[0082] The linkage part addition unit 210 has the function of setting additional linkage parts (sometimes called the second linkage part) that are different from the linkage parts (sometimes called the first linkage part) that include the first target part and the second target part. The fourth target part setting unit 220 has the function of setting the fourth target part. The fourth target part corresponds to the moving part of the second linkage part. The fifth target part setting unit 230 has the function of setting the fifth target part. The fifth target part corresponds to the part of the second linkage part that serves as the pivot point for the movement of the fourth target part.

[0083] (4-2. Object posture control processing) Figure 23 shows a flowchart of the second object attitude control process S200B based on commands from the object attitude control program in the object attitude control unit 100B. As shown in Figure 23, after setting the first target part and the second target part and determining that they are linked parts (S250; Yes), the linked part addition unit 210 determines whether to add a linked part (S255). The above determination may be made based on input from the user. At this time, the display unit 11 may display "Do you want to add a linked part?" and "Yes" and "No" buttons may be displayed. When the user presses the "Yes" button, a signal to add a linked part is input and the system switches to the mode for adding a linked part (S255; Yes).

[0084] Next, as shown in Figure 24, when part 81d is touched, the position information of part 81d is input to the fourth target part setting unit 220 (S260). Based on the input information, the fourth target part setting unit 220 sets the touched part 81d as the fourth target part (S265).

[0085] Next, when the part 81e shown in Figure 24 is touched, the position information of part 81e is input to the fifth target part setting unit 230 (S270). Based on the input information, the fifth target part setting unit 230 sets the touched part 81e as the fifth target part. (S275).

[0086] Next, the linkage part determination unit 170 determines whether the fourth target part and the fifth target part are linked parts, similar to the first target part and the second target part (S280). If it is determined that the set fourth target part and the fifth target part are linked parts (S280; Yes), the second object posture control process S200B is terminated.

[0087] Figure 25 is a flowchart of the third object control process S300B. The movement of the object accompanying the movement of the terminal 10B is achieved in the same manner as in the first embodiment. However, in this embodiment, as shown in Figure 26, when the position of the terminal 10B is changed from a lying position to an upright position, the coordinate changes of the first target part and the fourth target part are changed based on this amount of movement (S322). As shown in Figure 27, the first target part (part 81a) moves relative to the second target part (part 81b), and at the same time, the fourth target part (part 81d) moves relative to the fifth target part (part 81e), and the resulting object is displayed (S330). In other words, when one terminal moves, multiple linked parts can be moved to control the orientation of the object.

[0088] <Fifth Embodiment> This embodiment describes an object attitude control process that differs from that of the first embodiment. Specifically, it describes an example of setting a target part when multiple parts are displayed overlapping each other. Note that configurations and methods similar to those of the first to fourth embodiments may be omitted as appropriate.

[0089] Figure 28 shows a user interface where multiple body parts are displayed superimposed. As shown in Figure 28, the user's body part 81a (corresponding to the wrist joint) is displayed superimposed on body part 81g (corresponding to the hip joint). In this case, when the superimposed parts are specified by touch operation, the display direction of the object 80 on the display unit 11 may be changed. For example, when the superimposed parts are touched, as shown in Figure 29, the display unit 11 may display "Do you want to change the display direction of the object?" and display "Yes" and "No" buttons. When the "Yes" button is pressed, as shown in Figure 30, the object 80 may be rotated 90 degrees clockwise relative to the sheet 70 and displayed. This allows the overlapping body parts 81a and 81g to be displayed separately, and a specific body part can be set.

[0090] The method for setting a specific part within the overlapping area is not limited to the method described above. For example, a part that meets a predetermined condition may be set as the part to be identified (first target part or second target part). Specifically, if the overlapping area is specified by performing a touch operation, the part displayed at the forefront may be set as the part to be identified.

[0091] Alternatively, the area to be identified may be set based on a predetermined operation on the overlapping area. Specifically, when touching the overlapping area for a short time (e.g., 1 second), the front area may be set as the area to be identified, and when touching the area for a longer time (e.g., 5 seconds), the rear area may be set as the area to be identified.

[0092] <Sixth Embodiment> This embodiment describes an object attitude control process that differs from the first to fifth embodiments. Specifically, it describes an example in which a new connecting part is set when the movement of the first target part causes it to come into contact with a movable part different from the first and second target parts. Note that configurations and methods similar to those in the first to fifth embodiments may be omitted as appropriate.

[0093] Figure 31 shows a user interface illustrating an example where the first target area comes into contact with a different area. As shown in Figure 31, the user's area 81a (corresponding to the left wrist joint) is displayed in contact with area 81e (corresponding to the right elbow joint). In this case, if area 81e has been pre-entered as a movable area, it may be set as the fourth target area. In this case, since area 81e and area 81f are linked via bone 82e, area 81f may be set as the fifth target area.

[0094] In the above, when the first target area, part 81a, moves relative to the second target area, part 81b, as shown in Figure 32, the fourth target area, part 81e, may move relative to the fifth target area, part 81f, based on the movement of part 81a.

[0095] By using this embodiment, the movement of one interconnected part can interfere with and control the posture of other interconnected parts.

[0096] Furthermore, the above control is not limited to the orientation of a single interconnected part. For example, the orientation of other interconnected parts may be controlled by an object different from object 80. In other words, interconnected parts can be set passively, rather than through user operation.

[0097] <Seventh Embodiment> This embodiment describes an object attitude control process that differs from the first to fifth embodiments. Specifically, it describes an example of controlling the object's attitude using an angle θ in addition to the X, Y, and Z coordinates. Note that configurations and methods similar to those in the first to sixth embodiments may be omitted as appropriate.

[0098] Figure 33 shows the data structure of the coordinates in virtual space and the coordinates in the display unit 11 of the first target part (part 81a) of the object before and after the movement of the terminal. As shown in Figure 33, the movement information of the terminal 10 may use not only the X, Y, and Z coordinates but also the angle θ. In this case, it is not necessary to use any of the X, Y, or Z coordinates. In accordance with the movement of the terminal 10, the information of the first target part (part 81a) of the object 80 in virtual space and the coordinate information of the first target part (part 81a) of the object 80 in the display unit 11 are changed, and the changed coordinate information is stored (the coordinate information is rewritten).

[0099] Figure 34 shows an example of the user interface of an object after its orientation has changed, as displayed on the display unit 11. As shown in Figure 34, this embodiment allows for the control of the object's orientation, particularly its twisted orientation.

[0100] Furthermore, within the scope of the concept of the present invention, a person skilled in the art can conceive of various modifications and alterations, and it is understood that such modifications and alterations also fall within the scope of the present invention. For example, any addition, deletion, or design change of components, or addition, omission, or modification of processes, made by a person skilled in the art to the above-described embodiments, is also included within the scope of the present invention, as long as it retains the gist of the present invention.

[0101] (modified version) In the first embodiment of the present invention, an example was shown in which information of a first target area is input by touch operation and the first target area is set, and information of a second target area is input by touch operation and the second target area is set, but the invention is not limited to this. For example, the second target area may be set automatically when the first target area is set. In this case, the second target area may be set automatically as a part that is higher in rank than the first target area.

[0102] In the first embodiment of the present invention, an example was shown in which information on the first and second target areas is input by the user through touch operation, but the invention is not limited to this. For example, the information on the target areas may be input verbally using the microphone unit 18 of the terminal 10.

[0103] Furthermore, in the first embodiment of the present invention, when the terminal 10 is moved, the terminal may be moved while the first target part or the second target part of the object displayed on the display unit 11 is being touched. At this time, the object 80 displayed on the display unit 11 may remain stationary while only the coordinates of the first target part in the virtual space are changed. In the virtual space, the first target part moves relative to the second target part. When the movement of the terminal 10 is completed, the display unit 11 may display an object with the position of the first target part changed.

[0104] In the first embodiment of the present invention, an example was shown in which a captured image and an object in a virtual space are aligned and superimposed using a marker-based alignment method, but the invention is not limited to this. As a method for aligning a virtual space containing an object with a captured image, in addition to the marker-based alignment method, vision-based alignment methods such as natural feature-based alignment methods that do not use specific markers, model-based alignment methods, and sensor-based alignment methods using sensors may be used as appropriate. Furthermore, it is not always necessary to align the captured image with the object in the virtual space; as long as the coordinates of the object in the virtual space can be set, the most appropriate method may be used.

[0105] Furthermore, while the first embodiment of the present invention shows an example of displaying an object using augmented reality, it is not limited to this. One embodiment of the present invention can also be applied to virtual reality (VR), mixed reality (MR), and the like, as long as the coordinates of the object can be identified.

[0106] Furthermore, while the first embodiment of the present invention shows an example of displaying an object and controlling its orientation using a single terminal, the invention is not limited to this. Multiple terminals may be linked via a network 50 to control the orientation of an object. For example, as shown in Figure 35, an object may be displayed on one terminal 10 (terminal 10-1), and the orientation of the object may be controlled by changing the position of another terminal 10 (terminal 10-2). This allows the object to be continuously displayed fixedly on one terminal 10, enabling precise orientation control of the object. Also, terminals 10-1 and 10-2 do not have to be of the same type. For example, a goggle-type terminal may be used as the display terminal, and a watch-type terminal may be used as the orientation control terminal. [Explanation of Symbols]

[0107] 1··Object posture control system, 10··Terminal, 11··Display unit, 12··Control unit, 13··Storage unit, 14··Operation unit, 15··Communication unit, 16··Sensor unit, 17··Camera unit, 18··Microphone unit, 20··Server, 21··Communication unit, 22··Storage unit, 23··Control unit, 24··Display unit, 50··Network, 60··Table, 70··Sheet, 75··Marker, 80··Object, 81··Part, 82··Bone, 90··User 100...Object posture control unit, 110...Captured image acquisition unit, 120...Marker detection unit, 130...Spatial definition unit, 140...Object display unit, 150...First target area setting unit, 160...Second target area setting unit, 170...Interlocking part determination unit, 180...Movement information acquisition unit, 190...Object coordinate change unit, 195...Limit point detection unit, 200...Third target area setting unit, 210...Interlocking part addition unit, 220...Fourth target area setting unit, 230...Fifth target area setting unit

Claims

1. It has a processor and a sensor unit that detects the movement of the machine. The aforementioned processor, An object placed in a virtual space, which includes multiple parts that are defined as joints at different locations, is displayed on the display unit. Of the multiple parts included in the object displayed on the display unit, the part selected by the first touch operation is set as the first target part. Among the multiple parts included in the object displayed on the display unit, a part different from the first target part, selected by the second touch operation, is set as the second target part. Only when the first target part and the second target part are connected via bones and are in a relationship of mutual cooperation, the sensor unit will move and display the first target part relative to the second target part in response to detection. Information processing device.

2. The processor displays an object in a virtual space, which includes multiple parts, each defined as a joint at a different location, on the display unit. The processor sets the part selected by the first touch operation from among the multiple parts included in the object displayed on the display unit as the first target part. The processor sets as the second target part a part different from the first target part selected by the second touch operation, among the multiple parts included in the object displayed on the display unit. The processor moves and displays the first target part relative to the second target part in response to motion detection by the sensor, but only when the first target part and the second target part are connected via bones and are in a relationship of mutual cooperation. Information processing methods.

3. In the processor, An object placed in a virtual space, which includes multiple parts that are defined as joints at different locations, is displayed on the display unit. Of the multiple parts included in the object displayed on the display unit, the part selected by the first touch operation is set as the first target part. Among the multiple parts included in the object displayed on the display unit, a part different from the first target part, selected by the second touch operation, is set as the second target part. Only when the first target part and the second target part are connected via bones and are in a relationship of mutual cooperation, the sensor unit will move and display the first target part relative to the second target part in response to motion detection. program.

Images