Object posture control program and information processing device

Abstract

To allow for easily controlling posture of an object.SOLUTION: An object posture control program makes an information processing device perform steps of displaying, on a display unit, a first image representing an object with multiple parts superimposed on a second image captured by an image capturing unit, and controlling posture of the object by modifying position information of one or more operated parts of the multiple parts of the object according to a positional relationship between the object and an operating body in the second image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an object attitude control program and an information processing device. [Background technology]

[0002] In recent years, advances in electronic devices have made it possible to display high-definition still images and videos on devices such as smartphones and personal computers. Furthermore, research and development in information processing technology has progressed, and display technologies that combine virtual and real spaces using augmented reality (AR) are becoming practical. For example, Patent Document 1 discloses a technology for displaying objects placed in a virtual space on the display area of an information device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-41126 Summary of the Invention [Problem to be solved by the invention]

[0004] On the other hand, controlling the posture of an object displayed in a display area may require complicated input processing in order to specify the target part of the object.

[0005] In view of the above problems, one object of the present invention is to easily control the attitude of an object placed in a virtual space. [Means for solving the problem]

[0006] According to one embodiment of the present invention, an object attitude control program is provided for causing an information processing device to superimpose a first image representing an object having multiple parts on a second image captured by an imaging unit and display it on a display unit, and control the attitude of the object by changing the positional information of at least some of the operated parts of the object based on the positional relationship between the object and an operating body in the second image.

[0007] In the object attitude control program, when the position of the operating body and the position of the operated portion at least partially overlap, processing may be executed to notify the user.

[0008] In the object posture control program, the position of the operated portion may be changed based on a change in position information of the operating body while the positions of the operating body and the operated portion at least partially overlap.

[0009] In the object attitude control program, the position of the operated portion and the position of the operated portion may each be defined as a position in a coordinate system of a space in which the object is placed.

[0010] According to one embodiment of the present invention, there is provided an information processing device that controls the attitude of an object, which displays a first image representing an object having multiple parts on a second image captured by an imaging unit and superimposed on a display unit, and controls the attitude of the object by changing the positional information of at least some of the operated parts of the object based on the positional relationship between the object and an operating body in the second image. [Effects of the Invention]

[0011] By using an embodiment of the present invention, the pose of an object can be easily controlled. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram showing a hardware configuration of an object attitude control system according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a functional block diagram of an object attitude control unit according to the first embodiment of the present invention. [Figure 3] FIG. 3 is a flowchart of an object attitude control process according to the first embodiment of the present invention. [Figure 4] FIG. 3 is a flowchart of a superimposed display process according to the first embodiment of the present invention. [Figure 5] 10 is a diagram illustrating an example of alignment between coordinates in a virtual space and coordinates displayed on an information processing device in the superimposed display processing according to the first embodiment of the present invention. [Figure 6] 3 is a diagram illustrating an example of coordinates of each part of an object according to the first embodiment of the present invention. [Figure 7] 10 is an example of a user interface displayed on a display unit of an information processing device in superimposed display processing according to the first embodiment of the present invention. [Figure 8] FIG. 4 is a flowchart of a first object pose detection process according to the first embodiment of the present invention. [Figure 9] 10 is a diagram showing an example of a user interface displayed on a display unit of the information processing device in the first object orientation detection processing according to the first embodiment of the present invention. [Figure 10] FIG. 4 is a flowchart of a second object pose detection process according to the first embodiment of the present invention. [Figure 11] 10 shows position coordinates of an operating body before and after movement and position coordinates of an object in the second object orientation detection process according to the first embodiment of the present invention. [Figure 12] 10 is an example of a user interface displayed on a display unit of the information processing device in the second object orientation detection process according to the first embodiment of the present invention. [Figure 13] FIG. 10 is a functional block diagram of an object attitude control unit according to a second embodiment of the present invention. [Figure 14] FIG. 10 is a flowchart of a first object pose detection process according to the second embodiment of the present invention. [Figure 15] 10 is an example of a user interface displayed on a display unit of an information processing device in a first object orientation detection process according to a second embodiment of the present invention. [Figure 16] FIG. 10 is a flowchart of a second object pose detection process according to the second embodiment of the present invention. [Figure 17] 10 is an example of a user interface displayed on a display unit of an information processing device according to a second embodiment of the present invention. [Figure 18] 10 is a modified example of a user interface displayed on the display unit of the information processing device according to the first embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings and the like. However, the present invention can be implemented in many different forms, and should not be construed as being limited to the description of the following exemplary embodiments. The drawings may be represented schematically to clarify the description, but these are merely examples and do not limit the interpretation of the present invention. Furthermore, the letters "first" and "second" attached to each element are convenient labels used to distinguish each element and have no further meaning unless otherwise specified. In the drawings referred to in this embodiment, identical parts or parts having similar functions are designated by the same or similar symbols (symbols consisting of the numeral xxx followed by A or B), and repeated explanations may be omitted. Furthermore, some components may be omitted from the drawings. Furthermore, if a person of ordinary skill in the field to which the present invention pertains would recognize such features, no special explanation will be provided.

[0014] The term "part" used in one embodiment of the present invention refers to a movable part of an object, and a linked part is formed by combining a plurality of parts.

[0015] First Embodiment An object attitude control system according to a first embodiment of the present invention will be described in detail with reference to the drawings.

[0016] (1-1. Hardware configuration of the object attitude control system) Fig. 1 shows a hardware configuration and a functional block diagram of an object attitude control system 1. As shown in Fig. 1, the object attitude control system 1 includes a terminal 10 and a server 20. The terminal 10 and the server 20 may be collectively referred to as an information processing device.

[0017] The terminal 10 is a type of computer, and includes a display unit 11, a control unit 12, a memory unit 13, an operation unit 14, a communication unit 15, a sensor unit 16, an imaging unit 17, and a speaker unit 18. In this example, a smartphone is used as the terminal 10. However, the terminal is not limited to a smartphone, and may be a mobile phone (feature phone), a tablet terminal, a notebook PC (personal computer), an IoT device (equipment equipped with a power supply mechanism, a communication function, and an information storage mechanism), or the like, and any device capable of communicating with the server 20 via a network is applicable.

[0018] The display unit 11 is a display device such as a liquid crystal display or an organic EL display, and the display content is controlled by a signal input from the control unit 12.

[0019] The control unit 12 includes a central processing unit (CPU), an application specific integrated circuit (ASIC), a flexible programmable gate array (FPGA), or other arithmetic processing circuit. The control unit 12 executes applications including an object attitude control program stored in the storage unit 13 based on operations on the display unit 11 and the operation unit 14.

[0020] The storage unit 13 functions as a database that stores an object attitude control program and spatial information used in the object attitude control program. The storage unit 13 may be a memory, an SSD, or any other storage device.

[0021] The operation unit 14 includes a controller, a button, or a switch. When the operation unit 14 is moved up, down, left, or right, pressed, or rotated, information based on the operation is transmitted to the control unit 12. In this embodiment, the terminal 10 is a display device (touch panel) having a touch sensor, so the display unit 11 and the operation unit 14 may be located in the same place.

[0022] The communication unit 15 has a function of transmitting and receiving data to and from the server 20. A LAN transceiver (for example, a Wi-Fi transceiver) is used for the communication unit 15. Note that the transceiver is not limited to a LAN transceiver. If the terminal is a portable terminal, a transceiver for portable terminal communication (for example, LTE communication) or a transceiver for short-range wireless communication may be provided. The terminal 10 is connected to the server 20 via a network 50.

[0023] The sensor unit 16 has a function of detecting position information of a marker, an operating body, an object, etc. The sensor unit 16 includes a position sensor, a distance sensor, and a displacement sensor.

[0024] The imaging unit 17 has a function of capturing an environmental image, and captures an image of a target object on which a marker portion is provided for displaying the object. The imaging unit 17 may be, for example, a CMOS image sensor.

[0025] The speaker unit 18 has a function of outputting sound information to the outside based on the detected information.

[0026] The server 20 has a communication unit 21, a storage unit 22, a control unit 23, and a display unit 24. The server 20 functions as a database and an application server. Note that if all information related to the object attitude control program is pre-installed in the terminal 10, the server 20 does not necessarily have to be used.

[0027] The communication unit 21 has a transceiver and communicates control information of objects with the terminal 10 via the network 50. A transceiver for the Internet is used for the communication unit 21. Note that the transceiver is not limited to a LAN transceiver for the Internet, and any device capable of communicating in the same way as the terminal 10 is used.

[0028] The storage unit 22 functions as a database of information used in the object attitude control program by using a hard disk and an SSD.

[0029] The control unit 23 controls the processing of the object attitude control program using a CPU, ASIC, FPGA, or other arithmetic processing circuit. In addition, a user interface for executing the object attitude control program may be provided to the display unit 24 by an instruction from the control unit 23.

[0030] (1-2. Configuration of Object Attitude Control Unit 100) FIG. 2 shows a program that is configured by the components of the terminal 10 and the server 20 in the object attitude control system 1 and that realizes the object attitude control function. 1 shows a functional block diagram of an object posture control unit 100 that controls an object posture control program (object posture control program). In this example, an example in which the object posture control program is installed in a terminal 10 will be described.

[0031] The object posture control unit 100 includes a captured image acquisition unit 110, a marker detection unit 120, a space definition unit 130, an object display unit 140, an operating body detection unit 150, an overlapping part detection unit 160, a notification unit 170, a movement information acquisition unit 180, and an object coordinate change unit 190.

[0032] The captured image acquisition unit 110 has a function of acquiring an environmental image of the real space captured by the imaging unit 17 .

[0033] The marker detection unit 120 has a function of detecting markers from the image acquired by the captured image acquisition unit 110. In this example, information about the markers is stored in advance in the storage unit 13. Note that the information about the markers does not necessarily have to be stored in the storage unit 13, and feature points that can be markers may be extracted from the captured image.

[0034] The space definition unit 130 has a function of aligning the virtual space with the coordinates of the display unit 11. In this embodiment, the space definition unit 130 uses a marker-based alignment method to align the virtual space with the coordinates of the display unit 11 based on the detected marker.

[0035] The object display unit 140 has a function of displaying on the display unit 11 an object that is placed in a virtual space and whose attitude is to be controlled.

[0036] The operating body detection unit 150 has a function of detecting an operating body from an object included in a captured image and setting it as an operating body. The overlapping part detection unit 160 has a function of detecting that the operating body overlaps with a part of multiple parts of an object and setting that part as an operated part.

[0037] The notification unit 170 has a function of notifying the user that the operating object and the operated portion overlap. Note that the operating object and the operated portion overlap, for example, when the operating object and the operated portion overlap in position coordinates at least partially in the space (including virtual space) where the operating object and the operated portion are placed or in the display area of the display unit 11. In this example, the notification is made via sound information emitted from the speaker unit 18 of the terminal 10.

[0038] The movement information acquisition unit 180 has a function of acquiring movement information based on the coordinates before and after the set movement of the operating object.

[0039] In order to display the moved object on the display unit 11, the object coordinate modification unit 190 has a function of modifying and recording the coordinates of the non-operated part that overlaps with the operating body, and transmitting the position information to the object display unit 140. The movement information acquisition unit 180, the object coordinate modification unit 190, and the object display unit 140 acquire movement information of the operated part of the object 60 times per second, and can display the movement (posture change) of the object as a still image or a video by modifying the spatial coordinates of the operating body and the coordinates on the display screen.

[0040] (1-3. Object Attitude Control Processing) Next, we will explain the object posture control processing based on commands from the object posture control program in the object posture control unit 100. The object posture control processing includes a superimposed display processing S100 and an object posture detection processing. Fig. 3(A) is a flow diagram of the superimposed display processing. Fig. 3(B) is a flow diagram of the object posture detection processing. As shown in Fig. 3(B), the object posture detection processing includes a first object posture detection processing S200 and a second object posture detection processing S300. The superimposed display processing and the object posture detection processing can be performed in parallel.

[0041] The superimposed display processing S100 includes captured image acquisition and display processing, marker information detection processing, space definition processing, object coordinate acquisition processing, and captured image / object superimposed display processing. The first object posture detection processing S200 includes operating body detection processing, operating body setting processing, superimposed portion detection processing, operated portion setting processing, and superimposition notification processing. The second object posture detection processing S300 includes operating body movement amount acquisition processing, object coordinate change processing, and processing for determining whether the movement of the terminal has ended. Each of the object posture control processes will be explained separately.

[0042] (1-3-1. Overlay display processing S100) The superimposed display process S100 is shown in Fig. 4. The superimposed display process S100 is started when an application including an object attitude control program is started.

[0043] In the superimposed display process S100, the captured image acquisition unit 110 captures an environmental image of the real space, acquires the captured image (S110), and displays the captured image on the display unit 11 of the terminal 10. (also called the second image).

[0044] Fig. 5 shows an example of capturing an image of a real space with the terminal 10. In Fig. 5, the real space is arranged with the terminal 10, a table 60, a seat 70, and a user's hand 90, which serves as an operating object. The seat 70 includes four markers 75. In this example, the terminal 10 captures an image of a part of the table 60, the seat 70, and the user's hand 90, including the four markers 75.

[0045] Returning to FIG. 4, the explanation will be made. Next, the marker detection unit 120 performs marker detection processing (S120). In this example, information on four markers is registered in advance in the object posture control program. The marker information relates to the shape and arrangement of the markers. If the four markers are not detected (S120; No), the process returns to the capture image acquisition and display processing (S110).

[0046] Next, if four markers are detected (S120; Yes), the space definition unit 130 aligns the coordinates of the real space and the virtual space (space definition) using a marker-based alignment method (S130). At this time, the space definition unit 130 compares the markers 75 in the real space displayed on the display unit 11 with the markers in the virtual space stored in the object attitude control program, and adjusts them so that the coordinates of the virtual space can be displayed on the display unit 11 as coordinates of the real space. For example, the position coordinates of marker 75-1 are defined as (Xr75-1, Yr75-1, Zr75-1) in the real space, (Xd75-1, Yd75-1) on the display unit 11, and (Xv75-1, Yv75-1, Zv75-1) in the virtual space.

[0047] Next, the space definition unit 130 performs a process of acquiring object coordinates in the virtual space (S140). FIG. 6 shows the coordinates of each part of the object in the virtual space. As shown in FIG. 6, the coordinate information of each part of the object is set relative to an origin that is set based on the coordinates of the four markers. By the above process, information on the display unit 11 of the marker 75 is acquired, and the coordinates of each part of the object are acquired.

[0048] Next, the object display unit 140 displays the captured image (second image) and the image of the object (also referred to as the first image) superimposed on each other (S150). FIG. 7 shows a user interface of an object displayed on the display unit 11 of the terminal 10. In FIG. 7, an object 80 in a virtual space arranged on a sheet 70 is displayed together with a user's hand 90. In this example, the object 80 has a humanoid body shape and includes multiple parts 81 and multiple bones 82. The part 81 is a movable part and corresponds to a human joint. The part 81 (part 81a) is connected to an adjacent part 81 (part 81b) via a bone 82 (bone 82a). In other words, the parts 81a and 81b can be said to be linked parts. Note that the sheet 70 does not need to be displayed when the object 80 is displayed.

[0049] Next, a determination process is performed as to whether or not to terminate the superimposed display process (S160). This determination process may be performed based on information input by the user, or may be performed based on whether or not a predetermined condition is met. For example, if there is a moving object in the captured image, it is determined that the superimposed display process should not be terminated (S160; No), and the process returns to the captured image acquisition process (S110) again, and the superimposed display process S100 is repeated. On the other hand, if it is determined that the superimposed display process should be terminated (S160; Yes), the superimposed display process S100 is terminated.

[0050] (1-3-2. First Object Pose Detection Processing S200) 8 shows a flow diagram of the first object pose detection process S200. The first object pose detection process S200 is triggered by input of a detection start signal. The detection start signal may be input by the user, or a predetermined signal may be input from a program.

[0051] First, the operating object detection unit 150 performs an operating object detection process (S210). The detection process is performed based on the parallax occurring in the captured image.

[0052] First, the distance from the terminal 10 (imaging unit 17) to the object is calculated based on the change in the position of the object (parallax) in multiple images captured by the imaging unit 17. Next, an object whose distance from the terminal 10 to the object is shorter than the distance from the imaging unit 17 to the table is detected as the operating object. In this example, since the distance from the terminal 10 to the user's hand 90 is shorter than the distance from the terminal 10 to the table 60, the user's hand 90 is detected as the operating object (S210; Yes) and set (S220). At this time, the part of the user's hand 90 closest to the sheet 70 (more specifically, the tip 90a of the index finger) may be defined as the operating object. When the operating object is not detected (S210; No), the above processing is repeated in a loop.

[0053] Next, the overlapping portion detection unit 160 detects that the object 80 and the operating body overlap (S230). Whether or not there is overlap is determined based on whether or not the coordinates of the target part of the object 80 in the virtual space match the coordinates of the operating body in the real space.

[0054] Fig. 9 is an example of a user interface showing the overlapping of an operating object and an operated part. As shown in Fig. 9, in this example, a tip 90a of an index finger and a tip 90b of a thumb of a user's hand 90 overlap a part of a part 81a (corresponding to the left wrist joint) of an object 80. A method for detecting the overlapping of the tip 90a of the index finger and the part 81a of the object 80 will be described below.

[0055] First, the position (coordinates) of the tip 90a of the index finger in the real space is acquired. The coordinates of the tip 90a of the index finger are relative to the origin, which is the center of the area surrounded by the four markers 75 arranged on the sheet 70. At this time, the coordinates of the tip 90a of the index finger in the real space are (Xr_90a, Yr_90a, Zr_90a). Then, based on the coordinates of the real space, the coordinates (Xv_90a, Yv_90a, Zv_90a) of the tip 90a of the index finger when it is arranged in the virtual space are calculated.

[0056] Next, when the virtual space coordinates of the tip 90a of the index finger match any one of the multiple parts 81 of the object 80 (in this example, the virtual space coordinates (Xv_81a, Yv_81a, Zv_81a) of the part 81a of the object), it is determined that an overlapping part has been detected (S230; Yes). At this time, the part 81a of the object is set as the part to be operated (S240). Note that when an overlapping part is not detected (S230; No), the overlapping part detection process is repeated in a loop.

[0057] When an overlapping portion is detected, the notification unit 170 notifies that the operating object and the operated part are overlapping (S250). In this example, the notification is made using sound information emitted from the speaker unit 18 of the terminal 10. This completes the first object posture detection process S200.

[0058] (1-3-3. Second Object Pose Detection Processing S300) 10 is a flow diagram of the second object pose detection processing S300. The second object pose detection processing S300 is started upon completion of the first object pose detection processing S200.

[0059] In the second object posture detection process S300, first, the movement information acquisition unit 180 acquires the amount of movement of the operating body (S310). In this example, tracking of the posture of the object accompanying changes in position information of the operating body (tip 90a of the index finger of the user's hand 90) is performed by the V-SLAM (Visual Simultaneous Localization and Mapping) method. The position (coordinates) of the tip 90a of the index finger is measured and calculated using the sensor unit 16 and the imaging unit 17, and acquired by the movement information acquisition unit 180. The sensor unit 16 and the imaging unit 17 may be an image sensor, a position sensor, a distance sensor, a displacement sensor, or any other element capable of detecting position.

[0060] 11(A) is a data structure showing the initial value (before movement), the position (coordinates) of the operating object (tip 90a of the index finger) after time t1 (after movement), and the amount of movement of the tip 90a of the index finger calculated from the coordinates before and after the movement. Before movement, it is detected that the operating object and the operated part overlap.

[0061] Next, based on the acquired movement amount of the operating object, the object coordinate change unit 190 changes and records the coordinates of the operated portion (portion 81a) of the object 80, which is the portion to be moved (S320). FIG. 11(B) shows the data structure of the initial values (before movement) and the coordinates of the operated portion (portion 81a) of the object after time t1 in the virtual space and the coordinates on the display unit 11. As shown in FIG. 11(B), the coordinate information of the operated portion (portion 81a) of the object 80 in the virtual space and the coordinate information of the operated portion (portion 81a) of the object 80 on the display unit 11 are changed based on the movement amount of the operating object (tip 90a of the index finger), and the changed coordinate information is stored (the coordinate information is rewritten).

[0062] Returning to FIG. 10, the explanation will be made. At this time, the object display unit 140 displays the object 80 on the display unit 11 using information about the operated part after the movement (after time t1) in the superimposed display processing S100. FIG. 12 shows a user interface in which the object 80, whose position has changed, is displayed on the display unit 11. As shown in FIG. 12, the operated part (part 81a) moves from the position displayed in FIG. 7 using a different part (part 81b, corresponding to the elbow joint) as a reference. In other words, some of the multiple parts of the object can be moved based on information about the movement of the operating body. The above processing is repeated as long as the operating body moves (S330; Yes). In this example, the amount of movement of the operating body is acquired 60 times per second, so that changes in the posture of the object are displayed smoothly. When the movement of the terminal 10 ends (S330; No), the second object posture detection processing S300 ends.

[0063] As described above, by using the object attitude control program of this embodiment, it is possible to easily control the attitude of an object having multiple parts without performing complex control.

[0064] Second Embodiment In this embodiment, an object attitude control process different from that in the first embodiment will be described. Specifically, an example in which multiple parts move based on the movement of one terminal will be described. Note that configurations and methods similar to those in the first embodiment may be omitted as appropriate.

[0065] (2-1. Configuration of Server and Object Attitude Control Unit 100A) Fig. 13 shows a functional block diagram of an object posture control unit 100A in the object posture control system 1A. As shown in Fig. 22, the object posture control unit 100A includes a captured image acquisition unit 110, a marker detection unit 120, a space definition unit 130, an object display unit 140, an operating body detection unit 150, an overlapping portion detection unit 160, a notification unit 170, a movement information acquisition unit 180, and an object coordinate modification unit 190, as well as a second operating body detection unit 195 and a second overlapping portion detection unit 200.

[0066] The second operating body detection unit 195 has a function of detecting an object different from the object detected as the operating body from multiple objects included in the captured image, and setting the object as the second operating body. The second overlapping portion detection unit 200 has a function of detecting that the second operating body overlaps with a part of multiple parts of the object, and setting the part as the second operated part.

[0067] (2-2. Object Attitude Control Processing) FIG. 14 shows a flow diagram of the first object posture detection process S200A based on commands from the object posture control program in the object posture control unit 100A. As shown in FIG. 14, after notification of the overlapping portion (S250), it is determined whether to set an additional operating object (S255). The determination may be made based on whether there are any moving objects other than the object set as the operating object during a predetermined period, or may be made based on information input by the user. Furthermore, the display unit 11 may display a message saying "Do you want to add a new operating object?" along with "Yes" and "No" buttons. If no additional operating object is to be set (S255; No), the first object posture detection process S200A ends.

[0068] If it is determined that an additional operation should be set (S255; Yes), the second operation body detection unit 195 detects an object to be the second operation body (S260). In this example, the user's left hand 91 is detected as the second operation body. The detection method can be the same as the method described in the first embodiment of the present invention. At this time, the part of the user's left hand 91 closest to the seat 70 (More specifically, the tip 91a of the index finger of the left hand) may be set as the operation object (S265).

[0069] Next, the second overlapping portion detection unit 200 detects an overlapping portion (second overlapping portion) between the second operating body and a portion of the object 80 (S270). The second overlapping portion is detected using a method similar to the overlapping portion detection method of the first embodiment of the present invention. If it is determined that the second operating body overlaps with a portion 81c of the object 80 (S270; Yes), the portion 81c of the object is set as a second operated portion (S280). Note that if the second overlapping portion is not detected (S270; No), the second overlapping portion detection process is repeated in a loop.

[0070] After the second superimposed portion is detected, the notification unit 170 may notify that the second superimposition has been detected (S290). The notification method may be the same as that in the first embodiment. This completes the first object pose detection process S200A.

[0071] FIG. 16 is a flow diagram of the third object control process S300A. In this embodiment, the movement of the operated part (part 81a) of the object in accordance with the movement of the operating body (tip 90a of the index finger of the user's hand 90) is realized in the same manner as in the first embodiment. In this embodiment, as shown in FIG. 17, at the same time that the operated part (part 81a) moves, the coordinates of the operated part and the second operated part are changed based on the movement amount of a second operating body (tip 91a of the index finger of the user's left hand 91) different from the operating body so that the second operated part (part 81c, corresponding to the ankle joint) moves with respect to part 81e (S320). In other words, the posture of the object can be controlled by moving multiple operated parts in accordance with the movement of multiple operating bodies.

[0072] In this embodiment, an example in which two operating bodies are used to control the posture of an object is shown, but the present invention is not limited to this and can be similarly applied to a case in which three or more operating bodies are used.

[0073] It should be noted that within the scope of the concept of the present invention, a person skilled in the art may conceive of various modifications and alterations, and it is understood that these modifications and alterations also fall within the scope of the present invention. For example, even if a person skilled in the art appropriately adds or deletes components or modifies the design of each of the above-described embodiments, or adds or omits steps or modifies conditions, these modifications are also included within the scope of the present invention as long as they maintain the gist of the present invention.

[0074] (Variation) In the first embodiment of the present invention, an example was shown in which X, Y, and Z coordinates were used in the movement information of the operating body, but this is not limiting. For example, angle θ may be used in addition to X, Y, and Z coordinates. In this case, any of X, Y, and Z coordinates may not be used. Information on the operated portion (portion 81 a) of the object 80 in the virtual space and the position coordinates of the operated portion (portion 81 a) of the object 80 on the display unit 11 are changed in accordance with the movement of the operating body, and the information on the changed position coordinates is stored (the coordinate information is rewritten). This makes it possible to control the attitude of the object, particularly the twisting attitude.

[0075] In the first embodiment of the present invention, an example is shown in which an operating body is detected and set, and then an overlapping portion is detected and an operated portion is set, but this is not limiting. For example, when an object and a target body overlap, the overlapping object may be detected and set as an operating body, and the overlapping portion of the object may be set as an operated portion.

[0076] Furthermore, in the first embodiment of the present invention, an example in which the tip 90a of the index finger overlaps a portion of the object (portion 81a) has been mainly described, but the present invention is not limited to this. The overlapping portion detection unit 160 may detect overlapping when there are multiple overlapping portions. For example, when overlapping occurs at two or more points, the user's fingers may have a shape that pinches the object, allowing the user to control the posture of the object more naturally.

[0077] Furthermore, when it is determined that multiple portions overlap, additional control different from normal object posture control may be provided. FIG. 18 shows an example of a user interface when it is determined that multiple portions overlap and the operating object is moved. As shown in FIG. 18, in this example, the tip 90a of the index finger and the tip 90b of the thumb overlap with the portion 81(a) of the object. At this time, in accordance with the movement of the tip 90a of the index finger and the tip 90b of the thumb, the portion 81a of the object 80 also moves and the bone 82 (82a) extends based on preset conditions. In this way, by performing additional control different from normal object posture control, the posture of the object can be controlled in a variety of ways.

[0078] Note that detection may be performed by moving the operating object so as to surround the part corresponding to the part 81a, or by surrounding the part corresponding to the part 81a with a plurality of fingers.

[0079] In the first embodiment of the present invention, an example has been shown in which the notification unit 170 notifies using sound information, but this is not limiting. For example, when the notification unit 170 detects an overlapping portion, the notification unit 170 may vibrate the terminal 10 to notify. Alternatively, when the notification unit 170 detects an overlapping portion, the notification unit 170 may notify by causing the terminal 10 to emit light. Alternatively, when the notification unit 170 detects an overlapping portion, the notification unit 170 may notify using text information. Alternatively, when the notification unit 170 detects an overlapping portion, the notification unit 170 may notify by highlighting the overlapping portion by changing its color. Furthermore, the notification unit 170 does not necessarily have to be used.

[0080] The detection of the operating object is not limited to the method using parallax. For example, an object that satisfies a predetermined condition based on the color or shape of the object may be detected as the operating object.

[0081] An object moving at a speed exceeding a predetermined speed may be detected as the operating body.Also, an object may be detected as the operating body when the distance derived from the virtual space coordinates of the object when placed in the virtual space and the virtual space coordinates of the object becomes smaller than a predetermined distance.

[0082] In the first embodiment of the present invention, an example has been shown in which a captured image and an object in a virtual space are aligned by a marker-based alignment method and then superimposed and displayed, but this is not limiting. As a method for aligning a virtual space including an object with a captured image, in addition to the marker-based alignment method, a natural feature-based alignment method that does not use a specific marker, a vision-based alignment method such as a model-based alignment method, or a sensor-based alignment method that uses a sensor may also be used as appropriate.

[0083] In addition, although the first embodiment of the present invention has been described as an example in which an object is displayed using augmented reality, the present invention is not limited to this. The embodiment of the present invention can also be applied to mixed reality (MR) and the like in addition to augmented reality, as long as the coordinates of the object can be identified.

[0084] In the first embodiment of the present invention, an example of displaying on the display unit of a flat terminal having a form such as a smartphone has been shown, but the present invention is not limited to this. For example, a goggle-type terminal may be used as the terminal for display. In this case, a two-lens camera (stereo camera) may be used for the imaging unit 17. This allows the displayed image to be displayed in a more three-dimensional manner. [Explanation of symbols]

[0085] 1 Object attitude control system, 10 Terminal, 11 Display unit, 12 Control unit, 13 Memory unit, 14 Operation unit, 15 Communication unit, 16 Sensor unit, 17 Imaging unit, 18 Speaker unit, 20 Server, 21 Communication unit, 22 Memory unit, 23 Control unit, 24 Display unit, 50 Network, 60 Table, 70 Sheet, 75 Marker, 80 Object, 81 Body part, 82 Bone 90···Hand, 90a···Tip portion, 90b···Tip portion, 91···Hand, 91a···Tip portion, 100···Object posture control unit, 110···Captured image acquisition unit, 120···Marker detection unit, 130···Space definition unit, 140···Object display unit, 150···Operation body detection unit, 160···Superimposed portion detection unit, 170···Notification unit, 180···Movement information acquisition unit, 190···Object coordinate change unit, 195···Second operation body detection unit, 200···Second superimposed portion detection unit

Claims

1. For information processing devices a first image representing an object having a plurality of parts is superimposed on a second image captured by the imaging unit and displayed on the display unit; An object posture control program for controlling the posture of the object by changing the positional information of at least some of the operated parts of the object based on the positional relationship between the object and the operating body in the second image.

2. When the position of the operating object and the position of the operated portion at least partially overlap, a process for notifying the user is executed. The object attitude control program according to claim 1 .

3. changing the position of the operated portion based on a change in position information of the operating body while the position of the operating body and the position of the operated portion are at least partially overlapping; 3. The object attitude control program according to claim 1.

4. the position of the operated portion and the position of the operated portion are each defined as a position in a coordinate system of a space in which the object is placed; 4. The object attitude control program according to claim 2 or 3.

5. An information processing device for controlling the attitude of an object, a first image representing an object having a plurality of parts is superimposed on a second image captured by the imaging unit and displayed on the display unit; An information processing device that controls a posture of the object by changing position information of at least a part of a plurality of parts of the object to be operated based on a positional relationship between the object and an operating body in the second image.

Images