Object Posture Control Program and Information Processing Apparatus

Abstract

To allow for easily controlling posture of an object.SOLUTION: An object posture control program makes an information processing device perform steps of displaying, on a display unit, a first image representing an object with multiple parts superimposed on a second image captured by an image capturing unit, and controlling posture of the object by modifying position information of one or more operated parts of the multiple parts of the object according to a positional relationship between the object and an operating body in the second image.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an object posture control program and an information processing apparatus.

Background Art

[0002] In recent years, with the development of electronic devices, it has become possible to display high-definition still images and moving images on terminals such as smartphones and personal computers. Furthermore, research and development of information processing technologies have also advanced, and display technologies that fuse virtual spaces and real spaces using augmented reality (AR) are becoming practical. For example, Patent Document 1 discloses a technique for displaying an object arranged in a virtual space in a display area of an information terminal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] On the other hand, controlling the posture of an object displayed in a display area may complicate the input process when specifying a part of the object to be targeted.

[0005] In view of such problems, one of the objects of the present invention is to easily control the posture of an object arranged in a virtual space.

Means for Solving the Problems

[0006] According to an embodiment of the present invention, a first image representing an object having a plurality of parts is superimposed on a second image captured by an imaging unit and displayed on a display unit, and based on the positional relationship between the object and an operating body in the second image, the position information of at least some of the plurality of parts of the object, which are parts to be operated, is changed to control the posture of the object. An object posture control program is provided for causing this to be executed.

[0007] In the above object posture control program, when at least a part of the position of the operating body overlaps with the position of the part to be operated, a process for notifying the user may be executed.

[0008] In the above object posture control program, based on a change in the position information of the operating body while at least a part of the position of the operating body overlaps with the position of the part to be operated, the position of the part to be operated may be changed.

[0009] In the above object posture control program, the position of the part to be operated and the position of the part to be operated may each be defined as a position in a coordinate system of a space in which the object is arranged.

[0010] According to an embodiment of the present invention, there is provided an information processing apparatus for controlling the posture of an object, which superimposes a first image representing an object having a plurality of parts on a second image captured by an imaging unit and displays the superimposed image on a display unit, and based on the positional relationship between the object and an operating body in the second image, changes the position information of at least some of the plurality of parts of the object, which are parts to be operated, to control the posture of the object.

Advantages of the Invention

[0011] By using an embodiment of the present invention, the posture of an object can be easily controlled.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings and the like. However, the present invention can be implemented in many different modes and is not to be construed as being limited to the description of the embodiments exemplified below. The drawings may be schematically represented for the sake of clearer explanation, but are merely examples and do not limit the interpretation of the present invention. Also, the characters “first” and “second” appended to each element are for convenience of distinguishing each element and have no further meaning unless otherwise specified. In the drawings referred to in this embodiment, the same parts or parts having the same functions are denoted by the same reference numerals or similar reference numerals (reference numerals obtained by simply adding A and B to the numeral xxx), and the repeated description thereof may be omitted. Also, a part of the configuration may be omitted from the drawings. In addition, what can be recognized by those having ordinary knowledge in the field to which the present invention pertains will not be particularly described.

[0014] The “part” described in one embodiment of the present invention refers to a movable part of an object, and parts that interlock with each other are configured by combining a plurality of parts.

[0015] <First Embodiment> The object attitude control system according to the first embodiment of the present invention will be described in detail with reference to the drawings.

[0016] (1-1. Hardware Configuration of Object Attitude Control System) FIG. 1 shows the hardware configuration and functional block diagram of the object attitude control system 1. As shown in FIG. 1, the object attitude control system 1 includes a terminal 10 and a server 20. The terminal 10 and the server 20 may be collectively referred to as an information processing device.

[0017] The terminal 10 is a computer and has a display unit 11, a control unit 12, a storage unit 13, an operation unit 14, a communication unit 15, a sensor unit 16, an imaging unit 17, and a speaker unit 18. In this example, a smartphone is used as the terminal 10. Note that the terminal 10 is not limited to a smartphone, and may be a mobile phone (feature phone), a tablet terminal, a notebook PC (Personal Computer), an IoT device (a device equipped with a power supply mechanism, a communication function, and an information storage mechanism), etc., as long as it can communicate with the server 20 through a network.

[0018] The display unit 11 is a display device such as a liquid crystal display or an organic EL display, and the display content is controlled by a signal input from the control unit 12.

[0019] The control unit 12 includes a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Flexibe Programable Gate Array), or other arithmetic processing circuits. The control unit 12 executes an application including an object attitude control program stored in the storage unit 13 based on operations of the display unit 11 and the operation unit 14.

[0020] The storage unit 13 has a function as a database that stores an object posture control program and spatial information used in the object posture control program. A memory, SSD, or other storable element is used for the storage unit 13.

[0021] The operation unit 14 includes a controller, buttons, or switches. When operations such as moving up, down, left, or right, pressing, or rotating are performed on the operation unit 14, information based on the operations is transmitted to the control unit 12. In this embodiment, since the terminal 10 is a display device (touch panel) having a touch sensor, the display unit 11 and the operation unit 14 may be arranged in the same place.

[0022] The communication unit 15 has a function of transmitting and receiving data to and from the server 20. A LAN transceiver (for example, a Wi-Fi transceiver) is used for the communication unit 15. Note that the transceiver is not limited to a LAN transceiver. When the terminal is a portable terminal, a transceiver for mobile terminal communication (for example, LTE communication) may be provided, or a transceiver for short-range wireless communication may be provided. The terminal 10 is connected to the server 20 via the network 50.

[0023] The sensor unit 16 has a function of detecting position information of markers, operation bodies, objects, etc. The sensor unit 16 includes a position sensor, a distance sensor, and a displacement sensor.

[0024] The imaging unit 17 has a function of imaging an environmental image, and images an object provided with a marker unit for displaying an object. For example, a CMOS image sensor is used for the imaging unit 17.

[0025] The speaker unit 18 has a function of outputting sound information that outputs external sound information based on the detected information.

[0026] The server 20 includes a communication unit 21, a storage unit 22, a control unit 23, and a display unit 24. The server 20 functions as a database and an application server. Note that if all the information related to the object attitude control program is pre-installed in the terminal 10, the server 20 is not necessarily used.

[0027] The communication unit 21 has a transceiver and performs information communication of control information of the object with the terminal 10 via the network 50. An Internet transceiver is used for the communication unit 21. Note that the transceiver is not limited to a LAN transceiver for the Internet, and a device capable of communication similar to the terminal 10 is used.

[0028] The storage unit 22 functions as a database for information used in the object attitude control program by using a hard disk and an SSD.

[0029] The control unit 23 controls the processing of the object attitude control program by using a CPU, an ASIC, an FPGA, or other arithmetic processing circuits. In addition, a user interface for executing the object attitude control program may be provided on the display unit 24 according to an instruction from the control unit 23.

[0030] (1-2. Configuration of the Object Attitude Control Unit 100) FIG. 2 shows a functional block diagram of an object attitude control unit 100 that controls a program (object attitude control program) configured by each component of the terminal 10 and each component of the server 20 in the object attitude control system 1 to realize an object attitude control function. In this example, an example in which the object attitude control program is provided in the terminal 10 will be described. (Object Attitude Control Program)

[0031] The object posture control unit 100 includes an imaging image acquisition unit 110, a marker detection unit 120, a space definition unit 130, an object display unit 140, an operation detection unit 150, an overlapping part detection unit 160, a notification unit 170, a movement information acquisition unit 180, and an object coordinate change unit 190.

[0032] The imaging image acquisition unit 110 has a function of acquiring an environmental image of the real space captured by the imaging unit 17.

[0033] The marker detection unit 120 has a function of detecting a marker from the image acquired by the imaging image acquisition unit 110. In this example, the information of the marker is stored in the storage unit 13 in advance. Note that the information of the marker does not necessarily have to be stored in the storage unit 13, and feature points that can be markers may be extracted from the captured image.

[0034] The space definition unit 130 has a function of aligning the coordinates of the virtual space and the display unit 11. In the present embodiment, the space definition unit 130 uses the marker-based alignment method to align the coordinates of the virtual space and the display unit 11 based on the detected marker.

[0035] The object display unit 140 has a function of displaying on the display unit 11 an object to be controlled in terms of posture arranged on the virtual space.

[0036] The operation detection unit 150 has a function of detecting an operating body from an object included in the captured image and setting it as the operating body. The overlapping part detection unit 160 has a function of detecting that the operating body overlaps with a part of a plurality of parts of the object, and setting the part as the part to be operated.

[0037] The notification unit 170 has a function of notifying the user that the operating body and the operated part overlap. Note that the overlapping of the operating body and the operated part means that there is an overlap of the position coordinates at least in part between the operating body and the operated part in the space (including the virtual space) where the operating body and the operated part are arranged or in the display area of the display unit 11. In this example, the notification is made via the sound information emitted from the speaker unit 18 of the terminal 10.

[0038] The movement information acquisition unit 180 has a function of acquiring movement information based on the coordinates of the set operating body before and after movement.

[0039] The object coordinate change unit 190 has a function of changing and recording the coordinates of the non-operating part that overlaps with the operating body in order to display the moved object on the display unit 11, and transmitting the position information to the object display unit 140. The movement information acquisition unit 180, the object coordinate change unit 190, and the object display unit 140 acquire the movement information of the operated part in the object 60 times per second, and by changing the space coordinates and the coordinates on the display screen of the operating body, the movement (change in posture) of the object can be displayed as a still image or a moving image.

[0040] (1-3. Object Posture Control Processing) Next, the object posture control processing based on the command of the object posture control program in the object posture control unit 100 will be described. The object posture control processing includes the superimposed display processing S100 and the object posture detection processing. FIG. 3(A) is a flowchart of the superimposed display processing. FIG. 3(B) is a flowchart of the object posture detection processing. As shown in FIG. 3(B), the object posture detection processing includes the first object posture detection processing S200 and the second object posture detection processing S300. The superimposed display processing and the object posture detection processing can be performed in parallel.

[0041] The superimposed display process S100 includes an imaging image acquisition / display process, a marker information detection process, a space definition process, an object coordinate acquisition process, and an imaging image / object superimposed display process. The first object posture detection process S200 includes an operating body detection process, an operating body setting process, a superimposed portion detection process, an operated part setting process, and a superimposed notification process. The second object posture detection process S300 includes an operating body movement amount acquisition process, an object coordinate change process, and a determination process as to whether the movement of the terminal has ended. Each object posture control process will be described separately.

[0042] (1-3-1. Superimposed Display Process S100) FIG. 4 shows the superimposed display process S100. The superimposed display process S100 starts by launching an application including an object posture control program.

[0043] In the superimposed display process S100, the imaging image acquisition unit 110 captures an environmental image of the real space and acquires the captured image (S110), and the captured image (also referred to as the second image) is displayed on the display unit 11 of the terminal 10.

[0044] FIG. 5 is an example when the terminal 10 captures the real space. In FIG. 5, in the actual space, together with the terminal 10, a table 60, a sheet 70, and a user's hand 90 serving as an operating body are arranged. The sheet 70 includes four markers 75. In this example, a part of the table 60, the sheet 70, and the user's hand 90 are captured by the terminal 10 so as to include the four markers 75.

[0045] Returning to FIG. 4 for explanation. Next, the marker detection unit 120 performs a marker detection process (S120). In this example, information on four markers is registered in advance in the object posture control program. The marker information relates to the form and arrangement of the markers. When the four markers are not detected (S120; No), the process returns to the imaging image acquisition / display process (S110).

[0046] Next, when four markers are detected (S120; Yes), the space definition unit 130 performs coordinate alignment (space definition) between the real space and the virtual space using the marker-based alignment method (S130). At this time, the space definition unit 130 collates the marker 75 on the real space displayed on the display unit 11 with the marker on the virtual space stored in the object posture control program, and adjusts the coordinates of the virtual space so that they can be displayed on the display unit 11 as the coordinates of the real space. For example, the position coordinates of the marker 75-1 are defined such that the position coordinates in the real space are (Xr75-1, Yr75-1, Zr75-1), the position coordinates on the display unit 11 are (Xd75-1, Yd75-1), and the position coordinates in the virtual space are (Xv75-1, Yv75-1, Zv75-1).

[0047] Next, the space definition unit 130 performs a process of acquiring object coordinates in the virtual space (S140). FIG. 6 shows the coordinates of each part of the object in the virtual space. As shown in FIG. 6, the coordinate information of each part of the object is set with respect to the origin set based on the coordinates of the four markers. By the above process, when the information of the marker 75 on the display unit 11 is acquired, the coordinates of each part of the object are acquired.

[0048] Next, the object display unit 140 superimposes and displays the captured image (second image) and the image of the object (also referred to as the first image) (S150). FIG. 7 is a user interface of the object displayed on the display unit 11 of the terminal 10. In FIG. 7, an object 80 in the virtual space arranged on the sheet 70 is displayed together with the user's hand 90. In this example, the object 80 has a humanoid body shape and has a plurality of parts 81 and a plurality of bones 82. The part 81 is a movable part and corresponds to a human joint. The part 81 (part 81a) is connected to the adjacent part 81 (part 81b) via the bone 82 (bone 82a). That is, it can be said that the part 81a and the part 81b are interlocking parts. Note that the sheet 70 may not be displayed when the object 80 is displayed.

[0049] Next, a determination process is performed to determine whether to end the superimposed display process (S160). This determination process may be made based on information input from the user, or may be made depending on whether a predetermined condition is satisfied. For example, in the captured image, if there is a moving object, it is determined not to end the superimposed display process (S160; No), and the process returns to the captured image acquisition process (S110) again, and the superimposed display process S100 is repeated. On the other hand, if it is determined to end the superimposed display process (S160; Yes), the superimposed display process S100 ends.

[0050] (1-3-2. First Object Pose Detection Process S200) FIG. 8 shows a flowchart of the first object pose detection process S200. The first object pose detection process S200 is started when a detection start signal is input. The detection start signal may be input by the user, or a predetermined signal may be input from the program.

[0051] First, the operation body detection unit 150 performs a detection process of the operation body (S210). This detection process is determined based on the parallax generated in the captured image.

[0052] First, the distance from the terminal 10 (imaging unit 17) to the object is calculated based on the change (parallax) in the position of the object in a plurality of images captured by the imaging unit 17. Next, an object whose distance from the terminal 10 to the object is smaller than the distance from the imaging unit 17 to the table is detected as the operation body. In this example, since the distance from the terminal 10 to the user's hand 90 is smaller than the distance from the terminal 10 to the table 60, the user's hand 90 is detected as the operation body (S210; Yes) and set (S220). At this time, the portion of the user's hand 90 closest to the seat 70 (more specifically, the tip 90a of the index finger) may be defined as the operation body. When the operation body is not detected (S210; No), the above process is looped.

[0053] Next, the overlapping portion detection unit 160 detects that the object 80 and the operating body overlap (S230). The detection of whether there is an overlap is determined based on whether the coordinates of the target part of the object 80 in the virtual space match the coordinates of the operating body in the real space.

[0054] FIG. 9 is an example of a user interface showing a state where the operating body and the operated part overlap. As shown in FIG. 9, in this example, the tip 90a of the index finger and the tip 90b of the thumb of the user's hand 90 overlap a part of the part 81a (corresponding to the joint of the left wrist) of the object 80. Hereinafter, a method for detecting that the tip 90a of the index finger and the part 81a of the object 80 overlap is shown.

[0055] First, the position (coordinates) of the tip 90a of the index finger in the real space is acquired. For the coordinates of the tip 90a of the index finger, the center of the area surrounded by the four markers 75 arranged on the sheet 70 is used as the origin, and relative coordinates based on the origin are used. At this time, the coordinates of the tip 90a of the index finger in the real space are (Xr_90a, Yr_90a, Zr_90a). Then, based on the above real space coordinates, the coordinates (Xv_90a, Yv_90a, Zv_90a) when the tip 90a of the index finger is arranged in the virtual space are calculated.

[0056] Next, when the coordinates of the tip 90a of the index finger in the virtual space match the coordinates of any one of the plurality of parts 81 of the object 80 (in this example, the coordinates (Xv_81a, Yv_81a, Zv_81a) of the part 81a of the object in the virtual space), it is determined that the overlapping portion has been detected (S230; Yes). At this time, the part 81a of the object is set as the operated part (S240). When the overlapping portion is not detected (S230; No), the above overlapping portion detection process is performed in a loop.

[0057] When the overlapping part is detected, the notification unit 170 notifies that the operating body and the operated part are overlapping (S250). In this example, notification is made using the sound information emitted from the speaker unit 18 of the terminal 10. Thus, the first object posture detection process S200 ends.

[0058] (1-3-3. Second Object Posture Detection Process S300) FIG. 10 is a flowchart of the second object posture detection process S300. The second object posture detection process S300 starts triggered by the end of the first object posture detection process S200.

[0059] In the second object posture detection process S300, first, the movement information acquisition unit 180 acquires the movement amount of the operating body (S310). In this example, the tracking of the object posture accompanying the change in the position information of the operating body (the tip 90a of the index finger of the user's hand 90) is performed by the V-SLAM (Visual Simultaneous Localization and Mapping) method. The position (coordinates) of the tip 90a of the index finger is measured and processed by calculation using the sensor unit 16 and the imaging unit 17, and is acquired by the movement information acquisition unit 180. The sensor unit 16 and the imaging unit 17 use an image sensor, a position sensor, a distance sensor, a displacement sensor, or other elements capable of detecting a position.

[0060] FIG. 11(A) is a data structure showing the initial value (before movement), the position (coordinates) of the operating body (the tip 90a of the index finger) after time t1 (after movement), and the movement amount of the tip 90a of the index finger obtained from the coordinates before and after movement. Before movement indicates the time when it is detected that the operating body and the operated part overlap.

[0061] Next, based on the amount of movement of the acquired operating body, the object coordinate changing unit 190 changes and records the coordinates of the operated part (part 81a), which is the part to be moved, of the object 80 (S320). FIG. 11(B) shows the data structure of the coordinates of the operated part (part 81a) of the object in the virtual space and the coordinates in the display unit 11 after time t1 with the initial value (before movement). As shown in FIG. 11(B), the coordinate information of the operated part (part 81a) of the object 80 in the virtual space and the coordinate information of the operated part (part 81a) of the object 80 in the display unit 11 are changed based on the amount of movement of the operating body (the tip 90a of the index finger), and the changed coordinate information is stored (the coordinate information is rewritten).

[0062] Returning to FIG. 10 for explanation. At this time, the object display unit 140 displays the object 80 on the display unit 11 using the information of the operated part after movement (after time t1) in the superimposed display process S100. FIG. 12 is a user interface that displays the object 80 with the position of the operated part changed on the display unit 11. As shown in FIG. 12, the operated part (part 81a) moves with reference to a different part (part 81b, corresponding to the elbow joint) from the position displayed in FIG. 7. That is, based on the information regarding the movement of the operating body, some parts of the plurality of parts of the object can be moved. The above process is repeated as long as the operating body moves (S330; Yes). In this example, since the amount of movement of the operating body is acquired 60 times per second, the change in the posture of the object is smoothly displayed. When the movement of the terminal 10 ends (S330; No), the second object posture detection process S300 ends.

[0063] As described above, by using the object posture control program of the present embodiment, it is possible to easily control the posture of an object having a plurality of parts without performing complicated control.

[0064] <Second Embodiment> In this embodiment, object posture control processing different from that of the first embodiment will be described. Specifically, an example in which a plurality of parts move based on the movement of one terminal will be described. Note that the same configurations and methods as those in the first embodiment may be omitted as appropriate.

[0065] (2-1. Configuration of Server and Object Posture Control Unit 100A) FIG. 13 shows a functional block diagram of an object posture control unit 100A in an object posture control system 1A. As shown in FIG. 22, the object posture control unit 100A includes an imaging image acquisition unit 110, a marker detection unit 120, a space definition unit 130, an object display unit 140, an operation detection unit 150, an overlapping part detection unit 160, a notification unit 170, a movement information acquisition unit 180, and an object coordinate change unit 190, and further includes a second operation detection unit 195 and a second overlapping part detection unit 200.

[0066] The second operation detection unit 195 has a function of detecting an object different from the object detected as an operation object from a plurality of objects included in the captured image and setting it as a second operation object. The second overlapping part detection unit 200 has a function of detecting that the second operation object overlaps with a part of a plurality of parts of the object and setting the part as a second operated part.

[0067] (2-2. Object Posture Control Processing) FIG. 14 shows a flowchart of a first object posture detection process S200A based on instructions of an object posture control program in the object posture control unit 100A. As shown in FIG. 14, after a notification of an overlapping part is made (S250), it is determined whether to additionally set an operation object (S255). The above determination may be made based on whether there is an object that is moving other than the object set as an operation object within a predetermined period, or may be made based on input information from the user. In addition, the display unit 11 may display "Do you want to add a new operation object?" and buttons "Yes" and "No". If not additionally set (S255; No), the first object posture detection process S200A ends.

[0068] When it is determined that it is set additionally (S255; Yes), the second operation body detection unit 195 detects an object to be the second operation body (S260). In this example, the left hand 91 of the user is detected as the second operation body. As the detection method, the same method as the method described in the first embodiment of the present invention can be used. At this time, the portion of the user's left hand 91 closest to the seat 70 (More specifically, the tip 91a of the index finger of the left hand) may be set as the operation body (S265).

[0069] Next, the second overlapping portion detection unit 200 detects an overlapping portion (second overlapping portion) between the second operation body and a part of the object 80 (S270). As the detection method of the second overlapping portion, the same method as the detection method of the overlapping portion in the first embodiment of the present invention is used. When it is determined that the second operation body overlaps with a part 81c of the object 80 (S270; Yes). At this time, the part 81c of the object is set as the second operated part (S280). When the second overlapping portion is not detected (S270; No), the above second overlapping portion detection process is performed in a loop.

[0070] After the second overlapping portion is detected, the notification unit 170 may notify that the second overlapping has been detected (S290). As the notification method, the same method as in the first embodiment can be used. Thus, the first object posture detection process S200A ends.

[0071] FIG. 16 is a flowchart of the third object control process S300A. In the present embodiment, the movement of the manipulated part (part 81a) of the object accompanying the movement of the operating body (the tip 90a of the index finger of the user's hand 90) is realized in the same manner as in the first embodiment. In the case of the present embodiment, as shown in FIG. 17, simultaneously with the movement of the manipulated part (part 81a), based on the amount of movement of the second operating body (the tip 91a of the index finger of the user's left hand 91) different from the operating body, the coordinates of the manipulated part and the second manipulated part are changed so that the second manipulated part (parts 81c, corresponding to the ankle joint) moves with reference to part 81e (S320). That is, it is possible to move a plurality of manipulated parts in accordance with the movements of a plurality of operating bodies and control the posture of the object.

[0072] Note that, in the present embodiment, an example of controlling the posture of the object using two operating bodies has been shown, but it is not limited thereto. The present embodiment can be similarly applied when there are three or more operating bodies.

[0073] Note that those skilled in the art can conceive of various modification examples and correction examples within the scope of the idea of the present invention, and it is understood that those modification examples and correction examples also belong to the scope of the present invention. For example, with respect to each of the above-described embodiments, those obtained by appropriately adding, deleting, or changing the design of components, or adding, omitting, or changing the conditions of the steps by those skilled in the art are also included in the scope of the present invention as long as they have the gist of the present invention.

[0074] (Modification example) In the first embodiment of the present invention, an example using the X coordinate, Y coordinate, and Z coordinate in the movement information of the operating body is shown, but it is not limited thereto. For example, not only the X coordinate, Y coordinate, and Z coordinate but also the angle θ may be used. In this case, any one of the X coordinate, Y coordinate, and Z coordinate may not be used. In accordance with the movement of the operating body, the information of the operated part (part 81a) of the object 80 in the virtual space and the position coordinates of the operated part (part 81a) of the object 80 in the display unit 11 are changed, and the information of the position coordinates after the change is stored (the coordinate information is rewritten). Thereby, in the posture of the object, particularly the twisting posture can be controlled.

[0075] Also, in the first embodiment of the present invention, an example of detecting the overlapping part and setting the operated part after detecting and setting the operating body is shown, but it is not limited thereto. For example, at the stage where the object and the object overlap, the overlapping object may be detected and set as the operating body, and the overlapping part of the object may be set as the operated part.

[0076] Also, in the first embodiment of the present invention, an example in which mainly the tip 90a of the index finger overlaps with a part (part 81a) of the object is shown, but it is not limited thereto. When the overlapping part detection unit 160 has a plurality of overlapping parts, it may detect that they overlap. For example, when overlapping at two or more points, the user's finger can have a shape that pinches the object, and the posture of the object can be controlled more naturally.

[0077] Also, when it is determined that a plurality of parts overlap, additional control different from normal object posture control may be provided. FIG. 18 is an example of a user interface when it is determined that a plurality of parts overlap and the operating body moves. As shown in FIG. 18, in this example, the tip 90a of the index finger and the tip 90b of the thumb overlap with the part 81(a) of the object. At this time, in accordance with the movement of the tip 90a of the index finger and the tip 90b of the thumb, based on a preset condition, the part 81a of the object 80 also moves, and further the bone 82(82a) extends. In this way, by performing additional control different from normal object posture control, the posture of the object can be controlled in various forms.

[0078] Note that the operating body may be operated and detected so as to surround the portion corresponding to the part 81a, or may be detected by surrounding the portion corresponding to the part 81a with a plurality of fingers.

[0079] In the first embodiment of the present invention, the notification unit 170 has been shown as an example of notifying using sound information, but is not limited thereto. For example, when detecting an overlapping portion, the terminal 10 may be vibrated for notification. Alternatively, when detecting an overlapping portion, the terminal 10 may emit light for notification. Alternatively, when detecting an overlapping portion, character information may be used for notification. Alternatively, when detecting an overlapping portion, it may be emphasized and notified by changing the color of the overlapping portion. Also, the notification unit 170 does not necessarily have to be used.

[0080] Note that the detection of the operating body is not limited to the method using parallax. For example, an object that satisfies a predetermined condition based on the color or shape of the object may be detected as the operating body.

[0081] Also, an object moving at a speed exceeding a predetermined speed may be detected as the operating body. Further, when the distance derived from the coordinates in the virtual space of the object and the coordinates in the virtual space of the object when the object is arranged in the virtual space becomes smaller than a predetermined distance, that object may be detected as the operating body.

[0082] In the first embodiment of the present invention, an example in which a captured image and an object in a virtual space are aligned and superimposed by a marker-based alignment method has been shown, but the present invention is not limited thereto. As a method for aligning a virtual space including an object and a captured image, in addition to the marker-based alignment method, a vision-based alignment method such as a natural feature-based alignment method without using a specific marker, a model-based alignment method, or a sensor-based alignment method using a sensor may be appropriately used.

[0083] Further, in the first embodiment of the present invention, an example in which an object is displayed using augmented reality has been shown, but the present invention is not limited thereto. One embodiment of the present invention can also be applied to mixed reality (MR) or the like other than augmented reality as long as the coordinates of the object can be specified.

[0084] Further, in the first embodiment of the present invention, an example in which the image is displayed on the display unit of a planar terminal having a form such as a smartphone has been shown, but the present invention is not limited thereto. For example, a goggle-type terminal may be used as the terminal for display. At this time, a binocular camera (stereo camera) may be used for the imaging unit 17. Thereby, the displayed image can be displayed more three-dimensionally.

Description of Reference Numerals

[0085] 1... Object posture control system, 10... Terminal, 11... Display unit, 12... Control unit, 13... Memory unit, 14... Operation unit, 15... Communication unit, 16... Sensor unit, 17... Imaging unit, 18... Speaker unit, 20... Server, 21... Communication unit, 22... Memory unit, 23... Control unit, 24... Display unit, 50... Network, 60... Table, 70... Seat, 75... Marker, 80... Object, 81... Part, 82... Bone, 90... Hand, 90a... Tip, 90b... Tip, 91... Hand, 91a... Tip, 100... Object posture control unit, 110... Captured image acquisition unit, 120... Marker detection unit, 130... Space definition unit, 140... Object display unit, 150... Operation detection unit, 160... Overlapping part detection unit, 170... Notification unit, 180... Movement information acquisition unit, 190... Object coordinate change unit, 195... Second operation detection unit, 200... Second overlapping part detection unit

Claims

1. The processor causes a first image representing an object to be superimposed on a first object in a second image obtained by imaging, with an imaging unit, a real space including the first object on which a marker is placed, and to be displayed on a display unit; The processor causes a second object, whose distance from the imaging unit in the real space is shorter than the distance between the imaging unit and the first object, to be set as an operating body; The processor changes the position of a part of the object corresponding to the part where the position coordinates overlap, based on the amount of movement of the operating body after it is detected that at least a part of the position coordinates of the part of the object and the operating body overlap; A program.

2. Comprising one or more processors, the one or more processors cause a first image representing an object to be superimposed on a first object in a second image obtained by imaging, with an imaging unit, a real space including the first object on which a marker is placed, and to be displayed on a display unit; set, as an operating body, a second object whose distance from the imaging unit in the real space is shorter than the distance between the imaging unit and the first object; change the position of a part of the object corresponding to the part where the position coordinates overlap, based on the amount of movement of the operating body after it is detected that at least a part of the position coordinates of the part of the object and the operating body overlap; An information processing system.

3. The processor causes a first image representing an object to be superimposed on a first object in a second image obtained by imaging, with an imaging unit, a real space including the first object on which a marker is placed, and to be displayed on a display unit; The processor sets, as an operating body, a second object whose distance from the imaging unit in the real space is shorter than the distance between the imaging unit and the first object; The processor changes the position of a part of the object corresponding to the part where the position coordinates overlap, based on the amount of movement of the operating body after it is detected that at least a part of the position coordinates of the part of the object and the operating body overlap; An information processing method.

4. Comprising a processor, the processor causes a first image representing an object to be superimposed on a first object in a second image obtained by imaging, with an imaging unit, a real space including the first object on which a marker is placed, and to be displayed on a display unit; sets, as an operating body, a second object whose distance from the imaging unit in the real space is shorter than the distance between the imaging unit and the first object; Based on the amount of movement of the operating body after it is detected that at least some of the position coordinates of the part of the object and the operating body overlap, change the position of the part of the object corresponding to the overlapping part of the position coordinates. An information processing apparatus.

Images