Image processing device, image processing method, and recording medium

Abstract

The present technology relates to an image processing device, an image processing method, and a recording medium capable of improving the operability of a user. An image processing device provided with an image processing unit is provided, which performs the following control: arranging a virtual camera device of a virtual space in a three-dimensional virtual space, and causing a display terminal to display a background image representing the virtual space and a virtual object arranged in the background image; determining the pitch rotation of the display terminal based on a measurement signal measured by the display terminal; rotating the virtual camera device around the virtual object in the pitch direction based on the determination result of the pitch rotation of the display terminal; determining a touch operation on a touch panel of the display terminal; and rotating the virtual camera device around the virtual object in the yaw direction in the virtual space based on the determination result of the touch operation. The present technology can be applied to, for example, a mobile terminal.

Description

Technical Field

[0001] The present technology relates to an image processing apparatus, an image processing method, and a recording medium, and more particularly, to an image processing apparatus, an image processing method, and a recording medium capable of improving operability for a user. Background Art

[0002] There is known a technology for generating a virtual three-dimensional space, setting a viewpoint (virtual camera position) in the three-dimensional space, and displaying the three-dimensional space viewed from the virtual camera position as a 3D image (refer to Patent Documents 1 to 3).

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: WO 2009 / 084213

[0006] Patent Document 2: Japanese Patent Application Laid-Open No. 2003-334379

[0007] Patent Document 3: Japanese Patent Application Laid-Open No. 2002-298160 Summary of the invention

[0008] Problems to be solved by the present invention

[0009] Incidentally, when observing an observation target in a three-dimensional virtual space, it is necessary to enable the virtual camera to be intuitively moved relative to the observation target in order to enhance the operability for the user.

[0010] The present technology has been developed in view of the above-mentioned circumstances, and is intended to improve operability for users.

[0011] Solution to the problem

[0012] According to one aspect of the present technology, an image processing device includes an image processing unit, which: sets a virtual camera device of an area in a designated virtual space in a three-dimensional virtual space, and the area is displayed by a display terminal; performs control so that the display terminal displays a background image representing the virtual space and a virtual object set in the background image; determines a pitch rotation of the display terminal based on a measurement signal measured by the display terminal; rotates the virtual camera device around the virtual object in the pitch direction based on a determination result of the pitch rotation of the display terminal; determines a touch operation on a touch panel of the display terminal; and rotates the virtual camera device around the virtual object in the yaw direction in the virtual space based on a determination result of the touch operation.

[0013] An image processing method according to one aspect of the present technology includes: through an image processing device: setting a virtual camera device of an area in a designated virtual space in a three-dimensional virtual space, the area being displayed by a display terminal; executing control so that the display terminal displays a background image representing the virtual space and a virtual object set in the background image; determining a pitch rotation of the display terminal based on a measurement signal measured by the display terminal; rotating the virtual camera device around the virtual object in the pitch direction based on a determination result of the pitch rotation of the display terminal; determining a touch operation on a touch panel of the display terminal; and rotating the virtual camera device around the virtual object in the yaw direction in the virtual space based on a determination result of the touch operation.

[0014] According to one aspect of the present technology, a recording medium records a program, which is used to cause a computer to function as an image processing unit, which: sets a virtual camera device of an area in a specified virtual space in a three-dimensional virtual space, and the area is displayed by a display terminal; executes control so that the display terminal displays a background image representing the virtual space and a virtual object set in the background image; determines the pitch rotation of the display terminal based on a measurement signal measured by the display terminal; rotates the virtual camera device around the virtual object in the pitch direction based on a determination result of the pitch rotation of the display terminal; determines a touch operation on a touch panel of the display terminal; and rotates the virtual camera device around the virtual object in the yaw direction in the virtual space based on a determination result of the touch operation.

[0015] In an image processing device, an image processing method, and a recording medium according to one aspect of the present technology, a virtual camera device of an area in a designated virtual space is set in a three-dimensional virtual space, and the area is displayed by a display terminal; control is performed so that the display terminal displays a background image representing the virtual space and a virtual object set in the background image; the pitch rotation of the display terminal is determined based on a measurement signal measured by the display terminal; based on the determination result of the pitch rotation of the display terminal, the virtual camera device is rotated around the virtual object in the pitch direction; a touch operation on a touch panel of the display terminal is determined; and based on the determination result of the touch operation, the virtual camera device is rotated around the virtual object in the yaw direction in the virtual space.

[0016] It should be noted that the image processing device according to one aspect of the present technology may be an independent device, or may be an internal block including one device. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a diagram used to describe the pitch and yaw rotations in a screen tilt system.

[0018] Figure 2 is a graph used to describe viewpoints in a hybrid system.

[0019] Figure 3 : is a diagram showing the movement of the fingertip in the X direction and the Y direction in the screen touch operation.

[0020] Figure 4 is a diagram showing the midpoint and distance between two fingertips in a screen touch operation.

[0021] Figure 5 is a diagram showing pitch rotation and yaw rotation in the screen tilt operation.

[0022] Figure 6 : is a diagram showing the relationship between a virtual camera set in a virtual space, a rotation around a virtual object in a pitch direction, and a rotation around the virtual object in a yaw direction.

[0023] Figure 7 is a diagram showing an example of the configuration of a 3D image display system to which the present technology is applied.

[0024] Figure 8 It is shown Figure 7 A diagram of an example of a hardware configuration of a display terminal.

[0025] Fig. 9 It is shown Figure 8 A diagram showing an example of a functional configuration of a display terminal.

[0026] Fig.10 is a flowchart describing the flow of display processing.

[0027] Fig.11 This is a flowchart describing the flow of image processing corresponding to the user's operation.

[0028] Fig.12 is a diagram showing the relationship between the world coordinate system and the virtual camera coordinate system.

[0029] Fig.13 : is a diagram showing the correspondence relationship between the screen touch operation and the screen tilt operation and the movement of the virtual camera in the new system.

[0030] Fig.14 is a diagram showing an example of the configuration of a computer. DETAILED DESCRIPTION

[0031] Hereinafter, embodiments of the present technology will be described with reference to the drawings. Note that the description will be made in the following order.

[0032] 1. Implementation of the present technology

[0033] 2. Modification

[0034] 3. Computer configuration

[0035] <1. Embodiments of the present technology>

[0036] As a user interface (UI) for observing a virtual object displayed together with a background image in a three-dimensional virtual space, there is a system for moving a virtual camera around an observation target. Examples of this system include a screen tilt system and a screen touch system.

[0037] The screen tilt system is a system for rotating the virtual camera around the observation target by tilting the screen of a display terminal such as a smart phone. The screen touch system is a system for moving the position of the virtual camera by a touch operation on the screen of the display terminal. In addition, there is also a hybrid system in which the screen tilt system and the screen touch system are combined.

[0038] In the screen tilt system, the pitch rotation of the display terminal corresponds to the pitch rotation of the user's hand holding the display terminal (the pitch rotation of the wrist). In the case of using the screen tilt system, unlike the case of using the screen touch system, the orientation of the virtual camera device is basically associated with the orientation of the display terminal.

[0039] Therefore, with the screen tilt system, the orientation of the virtual camera device can be intuitively recognized, and it is less likely that the wrong recognition (wrong operation) of the rotation direction will occur as in the case of using the screen touch system. Therefore, from the observation point of the virtual object set in the virtual space, the screen tilt system can provide the user with a more intuitive operation than in the case of using the screen touch system.

[0040] In addition, the screen tilt system is used to perform the rotation of the virtual camera in the pitch direction (rotation around the virtual object) in response to the pitch rotation signal output from the terminal tilt detector (e.g., inertial measurement unit (IMU)). At this time, if the user only moves the wrist, the screen of the display terminal does not face the user due to the pitch rotation of the display terminal held in the hand.

[0041] If the screen is oriented upward or downward by the pitch rotation of the display terminal, the user naturally adjusts his or her posture, similar to daily movement of viewing or looking at a real object, so that the screen faces the user (face).

[0042] More specifically, when the display terminal undergoes pitch rotation through movement of the user's wrist, the user adjusts the position of his hand through the pitch rotation of the arm and simultaneously rotates the head through the pitch rotation, thereby keeping the screen of the display terminal facing the user.

[0043] That is, the difficulty in viewing the screen due to the pitch rotation of the display terminal can be reduced through the user's natural daily movement.

[0044] Regarding the yaw rotation of the display terminal, the user can adjust the position of his or her hand by utilizing the yaw rotation of the arm and simultaneously perform the yaw rotation of the head to keep the screen facing the user.

[0045] Here, the rotation amounts of the pitch rotation and the yaw rotation in the screen tilt system will be considered. The pitch rotation is a rotation for viewing the observation target (virtual object) from above or looking up at the observation target from below. Therefore, the angle of the desired observation target rotation is +90 degrees as the maximum value and -90 degrees as the minimum value. Therefore, it is only necessary for the user to be able to operate the virtual camera device 50 (within a range of 180 degrees of the pitch rotation angle relative to the wrist) Figure 1 A).

[0046] Meanwhile, in yaw rotation, in order to view the observation target from both the front and rear, it is necessary to rotate the observation target by +180 degrees as the maximum value and -180 degrees as the minimum value, which is 360 degrees in total. Therefore, it is only necessary for the virtual camera 50 to be able to rotate 360 ​​degrees with respect to the yaw rotation angle of the user's wrist ( Figure 1 B).

[0047] However, when the user attempts to perform such a rotation using the display terminal, the user faces the back side completely, and the physical burden increases. In addition, in an environment where another person is watching, such a behavior of the user may be considered as a suspicious behavior.

[0048] In fact, the yaw rotation of the virtual camera device with a small amount of yaw rotation of the wrist can also be achieved by multiplying the yaw rotation amount of the user's wrist by a predetermined coefficient. However, because the virtual camera device moves with a small amount of wrist movement, it is difficult to stabilize the virtual camera device. Therefore, it is common to use it in combination with the screen touch system to be described next.

[0049] In the screen touch system, a drag operation in the vertical direction on the touch panel of the display terminal is converted into a rotation of the virtual camera device in the pitch direction, and a drag operation in the horizontal direction on the touch panel of the display terminal is converted into a rotation of the virtual camera device in the yaw direction. Here, it is only necessary to drag to rotate the virtual camera device by a desired amount and fix the virtual camera device at the position, and thus the virtual camera device can be accurately moved to a desired position.

[0050] Note that with a screen tilt system, the virtual camera position always changes by reflecting the tilt of the screen, and the virtual camera position is not fixed. In addition, the accuracy of position specification in a screen tilt system is generally lower than the accuracy of position specification in a screen touch system performed with a fingertip.

[0051] Of course, with a screen touch system, the virtual camera position cannot be changed unless the user touches the screen with a fingertip. Although this may not be a problem depending on the application used, it is difficult to move the virtual camera in a train with a display terminal such as a smartphone held in one hand and a handrail held in the other.

[0052] Therefore, if an operation method can be provided that enables operation while the user moves both hands as little as possible (ie, with less physical burden on the user), the audio-visual experience can be improved when the user cannot use both hands.

[0053] Currently, with the hybrid system, the operations of yaw rotation and pitch rotation can be performed by tilting the screen of the display terminal, and the operations of yaw rotation and pitch rotation can be performed by touching the screen.

[0054] However, in the case of changing the position of the virtual camera device in the pitch direction by dragging in the vertical direction during a touch operation on the screen of the display terminal, even if the user holds the screen of the display terminal straight ahead in the horizontal direction, the sky above, the ground below, etc. are inevitably visible to the user, and thus the user may be confused about which direction the viewpoint faces ( Figure 2 ).

[0055] Figure 2 The following state is schematically shown: when the user's viewpoint exists at the virtual camera position in the virtual space represented by the circle in the figure, the user watching the screen of the display terminal 10 in the real space sees not the image V1 that he wants to watch in the virtual space, but the image V2 above (e.g., the sky).

[0056] Here, the features of the above-mentioned screen tilt system, screen touch system and hybrid system are summarized as follows.

[0057] Examples of the advantages (positive) of the screen tilt system are the following points. First, the user can intuitively perform operations using movements similar to daily movements of watching or viewing real objects. Second, because the user can operate the display terminal with only one hand, the physical burden is low in the case where the rotation angle of pitch rotation or yaw rotation is shallow.

[0058] Meanwhile, the screen tilt system has a disadvantage (negative) in that, as the rotation angles of the pitch rotation and the yaw rotation increase, the user's physical burden when watching / listening also increases.

[0059] The advantage (positive side) of the screen touch system is that the virtual camera position can be accurately moved from the current position to the desired position in the virtual space.

[0060] Meanwhile, the screen touch system has a disadvantage (negative) in that the virtual camera cannot be moved unless the user touches the screen of the display terminal with a fingertip, and therefore, a physical burden is imposed due to the touch operation.

[0061] Examples of the advantages (positive aspects) of the hybrid system are the following points: First, since the user can operate the display terminal with only one hand similarly to the case of the screen tilt system, the burden on the body is small when the rotation angle is shallow.

[0062] Second, similar to the case of the screen touch system, the virtual camera position can be accurately moved from the current position to the desired position in the virtual space. Third, unless the change of the pitch rotation is performed using the screen touch, the user can perform intuitive operations using movements similar to daily movements of viewing or looking up at a real object.

[0063] Meanwhile, the disadvantage (negative) of the hybrid system is that, since not only a screen tilt operation but also a screen touch operation can be performed, an image having a tilt different from the tilt of the screen of the display terminal can be displayed ( Figure 2 ).

[0064] Therefore, in a system to which the present technology is applied (hereinafter referred to as a new system), the virtual camera device is rotated in the yaw direction by touching the screen during a screen touch operation, and is rotated in the pitch and yaw directions by tilting the screen during a screen tilt operation, thereby changing the position of the virtual camera device relative to the virtual object.

[0065] With this arrangement, it is possible to avoid confusion of the user due to the screen display having a tilt different from the tilt of the screen of the display terminal, etc., and to perform a more intuitive operation of moving the virtual camera around the observation target (virtual object). Therefore, the operability of the user can be improved.

[0066] (Definition of terms)

[0067] Here, before describing the details of the new system, terms used in the present disclosure will be defined.

[0068] "Virtual camera" specifies the area displayed by (the screen of) the display terminal in the three-dimensional virtual space. The virtual camera position in the virtual space corresponds to the position of the viewpoint from which the user views the screen of the display terminal.

[0069] The "background image" is an image representing a three-dimensional virtual space. The background image includes a wide-angle image having a viewing angle of at least 180 degrees or wider. The "virtual object" is an object (substance) set in the background image.

[0070] A virtual object is an observation target observed by a user. Although the virtual object varies according to the services provided, athletes, celebrities, etc. that the user wishes to follow are observation targets. The virtual object to be observed is not limited to a person, and may be a creature (e.g., a dog, a cat, or an insect), an object (e.g., a work of art), etc. In addition, the number of virtual objects to be observed is not limited to one, and may be more than one, and furthermore, the virtual object may not necessarily be stationary, and may be moving.

[0071] Note that a space in real life as opposed to a “virtual space” is referred to as a “real space.” Furthermore, an object in real life as opposed to a “virtual object” is referred to as a “real object.”

[0072] The "fingertip movement in the X direction" and "fingertip movement in the Y direction" during screen touch operation are shown as follows: Figure 3 However, Figure 3 A situation is shown in which the screen of the display terminal 10 is in a landscape orientation.

[0073] “Finger tip movement in the X direction” is an operation of moving the user's finger in the horizontal direction (X direction) while the finger is in contact with the screen of the touch panel 110 of the display terminal 10 ( Figure 3 The “fingertip movement in the Y direction” is an operation of moving the user's finger in the vertical direction (Y direction) while the finger is in contact with the screen of the touch panel 110 ( Figure 3 vertical arrow in the middle).

[0074] Note that the “fingertip movement in the X direction” and the “fingertip movement in the Y direction” may not only be the movement of the fingertip of one finger of the user (the fingertip is in the Figure 3 The screen shown in FIG. 1 is a screen that is not ...

[0075] In addition, despite Figure 3 The operation from the left side to the right side in the figure is shown as a horizontal movement, but the horizontal movement can also be an operation from the right side to the left side in the figure. Figure 3 Although the operation from the lower side to the upper side in the figure is shown as movement in the vertical direction, the movement in the vertical direction may be the operation from the upper side to the lower side in the figure.

[0076] In addition, the "distance between two fingertips" and "midpoint between two fingertips" during screen touch operation are expressed as follows: Figure 4 However, Figure 4 A situation is shown in which the screen of the display terminal 10 is in a landscape orientation.

[0077] The “distance between two fingertips” is the distance between the two fingertips when the two fingertips of the user are in contact with the screen of the touch panel 110 ( Figure 4 The “midpoint between the two fingertips” is the position of the midpoint between the two fingertips when the two fingertips of the user are in contact with the screen of the touch panel 110. In other words, it can also be said that the “midpoint between the two fingertips” is the middle position of the “distance between the two fingertips” ( Figure 4 cross mark on the arrow in the middle).

[0078] Note that although Figure 4 The case where the “distance between two fingertips” is the distance between the fingertips of the index fingers of the user's left and right hands is illustrated, but the distance may be the distance between the fingertips of two fingers (eg, the index finger and the middle finger) of one hand.

[0079] The "pitch rotation" and "yaw rotation" in the screen tilt operation are expressed as follows: Figure 5 The rotation shown. However, Figure 5 A situation is shown in which the screen of the display terminal 10 is in a landscape orientation.

[0080] The “pitch rotation” indicates a rotation in the vertical direction with the lateral direction (the dotted line in the horizontal direction) in the figure as an axis ( Figure 5 ). Specifically, Figure 5 As shown, in a case where the screen of the display terminal 10 such as a smartphone is in a landscape orientation, rotation in the vertical direction with the longer direction as an axis is referred to as pitch rotation.

[0081] The "yaw rotation" indicates the rotation in the lateral direction with the vertical direction (dashed line in the vertical direction) as the axis ( Figure 5 ). Specifically, Figure 5 As shown, in a case where the screen of the display terminal 10 is in a landscape orientation, rotation in the landscape direction with the direction (shorter direction) perpendicular to the longer direction as an axis is referred to as yaw rotation.

[0082] Note that although not shown, when the screen of the display terminal 10 is in a vertical orientation, rotation in the vertical direction with the shorter direction as the axis is called "pitch rotation", and rotation in the lateral direction with the longer direction as the axis is called "yaw rotation".

[0083] The "rotation in the pitch direction" and "rotation in the yaw direction" of the virtual camera device in the virtual space are indicated as follows: Figure 6 Rotation shown.

[0084] “Rotation of the virtual camera in the pitch direction” indicates that the virtual camera moves along the surface (spherical surface) of a sphere centered on the virtual object 60 on the YZ plane. Figure 6).

[0085] “Rotation of the virtual camera in the yaw direction” indicates that the virtual camera moves along the spherical surface of a sphere centered on the virtual object 60 on the XZ plane ( Figure 6 ).

[0086] That is, it can also be said that "rotation of the virtual camera in the pitch direction" and "rotation of the virtual camera in the yaw direction" represent the rotation of the virtual camera around the virtual object 60. When the virtual camera rotates around the virtual object 60 as the observation target, the movement of the viewpoint rotating around the virtual object 60 while viewing the virtual object 60 can be used as the position of the user's viewpoint.

[0087] exist Figure 6 In the example, the center (eg, center of gravity) of the virtual object 60 is set as the "rotation center (rotation center position)" of the virtual camera device, and the distance from the rotation center to the virtual camera device is set as the "rotation radius (R)".

[0088] Note that in the present disclosure, a "drag operation" is an operation of moving a finger to a target position while keeping the finger in contact with the screen of the display terminal 10 and then releasing the finger. A "pinch-in operation" is an operation of moving two fingers in a pinching motion on the screen of the display terminal 10 to reduce the screen. A "pinch-out operation" is an operation of moving two fingers in an expanding motion on the screen of the display terminal 10 to enlarge the screen.

[0089] In the following, reference will be made to Figures 7 to 13 Describe the new system in detail.

[0090] (System Configuration)

[0091] Figure 7 An example of the configuration of the 3D image display system 1 compatible with the new system is shown.

[0092] The 3D image display system 1 includes display terminals 10 - 1 to 10 -N (N: an integer of 1 or more), and a distribution server 20 .

[0093] In the 3D image display system 1, the display terminals 10-1 to 10-N and the distribution server 20 are connected to each other via a network 30. The network 30 includes a communication network such as the Internet, an intranet, or a mobile phone network.

[0094] The display terminal 10 - 1 is a mobile terminal having a display, such as a smartphone, a tablet terminal, a wearable device, or a portable game console.

[0095] In response to a user's operation or the like, the display terminal 10 - 1 transmits a distribution request for an application to the distribution server 20 via the network 30 . The display terminal 10 - 1 receives the application distributed from the distribution server 20 via the network 30 .

[0096] This application is software for viewing and listening to 3D images (the images of which are captured in free viewpoint video based on 3D data) of athletes, celebrities, characters (avatars) in computer graphics, etc. Hereinafter, this application is also referred to as a "3D application".

[0097] The 3D data may be distributed in the 3D application, or when the 3D application is executed, the display terminal 10 - 1 may receive (dynamically load) the 3D data from an external server such as the distribution server 20 via the network 30 .

[0098] In the display terminal 10 - 1 , when the 3D application distributed from the distribution server 20 is executed, a 3D image is displayed based on the 3D data.

[0099] Similar to the display terminal 10 - 1 , the display terminals 10 - 2 to 10 -N are configured as mobile terminals such as smartphones. On each of the display terminals 10 - 2 to 10 -N, when a 3D application distributed from the distribution server 20 is executed, a 3D image is displayed.

[0100] Note that in the following description, in the case where there is no particular need to distinguish the display terminals from one another, the display terminals 10 - 1 to 10 -N are also simply referred to as “display terminals 10 ”.

[0101] The distribution server 20 includes one or more servers for distributing 3D applications and installed in a data center, etc. The distribution server 20 is provided by a service operator or the like that provides a content service such as a 3D image distribution service as a service.

[0102] Distribution server 20 transmits (distributes) the 3D application via network 30 in response to a distribution request from display terminal 10. In addition, distribution server 20 transmits (distributes) the 3D data via network 30 in response to a distribution request from display terminal 10 when the 3D application does not include 3D data.

[0103] (Display terminal configuration)

[0104] Figure 8 Shows Figure 7 An example of the hardware configuration of the display terminal 10 in FIG.

[0105] exist Figure 8In the display terminal 10, the display terminal 10 includes a control unit 101, a memory 102, a sensor 103, an image processing unit 104, a storage unit 105, a display unit 106, an input unit 107, and a communication unit 108. In the display terminal 10, the display unit 106 and the input unit 107 constitute a touch panel 110.

[0106] The control unit 101 includes a processor such as a central processing unit (CPU) The control unit 101 is a main control device (processing device) that controls the operation of each unit and performs various arithmetic processing, and controls the operation of each unit of the display terminal 10 .

[0107] The memory 102 includes a semiconductor memory such as a random access memory (RAM). The memory 102 temporarily stores various data processed by the control unit 101.

[0108] The sensor 103 includes various sensor devices. The sensor 103 performs sensing of the user, its vicinity, etc., and supplies sensor data obtained as a result of the sensing to the control unit 101. The control unit 101 performs various processes based on the sensor data supplied from the sensor 103.

[0109] Examples of the sensor 103 include an inertial measurement unit (IMU) that measures three-dimensional angular velocity and acceleration, etc. The inertial measurement unit (IMU) can obtain three-dimensional angular velocity and acceleration through a three-axis gyroscope and a three-way accelerometer.

[0110] Note that the sensor 103 may include a proximity sensor that measures an approaching object, a biosensor that measures information such as heart rate, body temperature, or posture of a living being, a magnetic sensor that measures the magnitude or direction of a magnetic field (magnetic field), and the like.

[0111] The image processing unit 104 includes a processor such as a graphics processing unit (GPU). The image processing unit 104 performs predetermined image processing on 3D data and the like under the control of the control unit 101. Display data obtained by the image processing is supplied to the display unit 106.

[0112] The storage unit 105 is an auxiliary storage device including a semiconductor memory such as a nonvolatile memory. The storage unit 105 may be configured as an internal storage device, or may be an external storage device such as a memory card.

[0113] The storage unit 105 records a 3D application or various data such as 3D data under the control of the control unit 101. The control unit 101 can read and execute (the data in) the 3D application recorded in the storage unit 105.

[0114] The display unit 106 and the input unit 107 constitute a touch panel 110. The display unit 106 is a display device such as a liquid crystal panel or an organic light emitting diode (OLED) panel. The input unit 107 is a position input device installed on the surface of the display unit 106.

[0115] The display unit 106 displays a 3D image or various information corresponding to the display data from the image processing unit 104 under the control of the control unit 101 .

[0116] When the user's finger contacts (touches) the surface of the display unit 106, the input unit 107 supplies operation data corresponding to the contact position (position of a predetermined point) to the control unit 101. The control unit 101 performs various processes based on the operation data supplied from the input unit 107.

[0117] The communication unit 108 is configured as a communication device (communication module) supporting wireless communication or wired communication such as a wireless local area network (LAN) or cellular communication (eg, LTE-Advanced, 5G, etc.) The communication unit 108 communicates with another device via the network 30 under the control of the control unit 101 .

[0118] The communication unit 108 sends a distribution request for a 3D application or 3D data to the distribution server 20 via the network 30. The communication unit 108 receives the 3D application or 3D data distributed from the distribution server 20 via the network 30. The 3D application or 3D data is provided to the image processing unit 104 and processed, or provided to the storage unit 105 and recorded.

[0119] Fig. 9 A display terminal 10 ( Figure 8 )'s functional configuration example.

[0120] exist Fig. 9 In the embodiment, the display terminal 10 includes: a terminal tilt detection unit 121, a screen touch detection unit 122, an image processing unit 123 and a display control unit 124.

[0121] The terminal tilt detection unit 121 and the sensor 103 (including an inertial measurement unit (IMU) and the like) Figure 8 The terminal tilt detection unit 121 detects the tilt of the screen of the display terminal 10 and provides the detection result to the image processing unit 123.

[0122] The screen touch detection unit 122 and the input unit 107 ( Figure 8 The screen touch detection unit 122 detects the contact (touch) of the user's finger on the surface of the display unit 106, and supplies the detection result to the image processing unit 123.

[0123] The image processing unit 123 and the control unit 101 ( Figure 8 ) and image processing unit 104 ( Figure 8 The image processing unit 123 performs predetermined image processing based on the 3D data received by the communication unit 108, and supplies display data obtained by the image processing to the display control unit 124.

[0124] The display control unit 124 corresponds to the control unit 101 ( Figure 8 The display control unit 124 performs control of displaying a 3D image corresponding to the display data supplied from the image processing unit 123 on the display unit 106 .

[0125] Here, a 3D image including a background image and a virtual object is displayed on the screen of the display unit 106 by setting a virtual camera in a virtual space and performing image processing to generate display data corresponding to the background image and the virtual object.

[0126] Furthermore, the detection result of the screen tilt from the terminal tilt detection unit 121 and the detection result of the screen touch from the screen touch detection unit 122 are supplied to the image processing unit 123 .

[0127] The image processing unit 123 sets a parameter for moving the virtual camera set in the virtual space based on at least one of the detection result of the screen tilt or the detection result of the screen touch. By setting the parameter, the virtual camera moves in the virtual space.

[0128] The 3D image display system 1 is configured as described above.

[0129] Note that the above-described configuration is an example of the configuration of the 3D image display system 1 or the display terminal 10 , and another component (eg, device or function) may be added or the above-described component (eg, device or function) may be removed.

[0130] For example, although the description has been made above in which the distribution server 20 is provided as the device on the distribution side in the 3D image display system 1 ( Figure 7 ), but a server for image processing may be provided separately, and 3D data subjected to image processing by the server may be distributed by the distribution server 20. In addition, the display terminal 10 may display a 3D image based not only on a 3D application or 3D data distributed from the distribution server 20 but also on a 3D application or 3D data recorded in the storage unit 105.

[0131] Furthermore, for example, although the above description has been made of a case in which the image processing unit 104 is provided as the display terminal 10 ( Figure 8 ) configuration, but by integrating the control unit 101 and the image processing unit 104 (configured as a system on a chip, etc.), the function of the image processing unit 104 can be included in the function of the control unit 101.

[0132] In addition, for example, a speaker, a camera, a near field wireless communication unit, etc. may be included as the display terminal 10 ( Figure 8 Specifically, in the display terminal 10, the speaker outputs audio (sound) related to the 3D image corresponding to the 3D data under the control of the control unit 101.

[0133] The camera includes an optical system, an image sensor, a signal processing circuit, etc., and provides imaging data obtained by capturing an image of a subject such as a user to the control unit 101. The control unit 101 performs various processes based on the imaging data from the camera. The near-field wireless communication unit is configured as a communication device that supports near-field wireless communication such as Bluetooth (registered trademark). The near-field wireless communication unit performs near-field wireless communication with another device under the control of the control unit 101.

[0134] (Display processing flow)

[0135] Next, we will refer to Fig.10 The flowchart in describes the flow of display processing compatible with the new system performed by the display terminal 10.

[0136] In step S1 , the image processing unit 123 performs predetermined image processing based on 3D data, and sets a virtual camera in a three-dimensional virtual space.

[0137] In step S2 , the display control unit 124 controls display of a background image representing a three-dimensional virtual space and a virtual object provided in the background image based on the result of the image processing.

[0138] With this arrangement, the display terminal 10 displays, on the screen of the display unit 106 , a 3D image including a virtual object as a background with the background image when viewed from a virtual camera position corresponding to the position of the user's viewpoint.

[0139] In step S3, the image processing unit 123 performs image processing corresponding to the user's operation. In this image processing, a parameter for moving the virtual camera position in the virtual space is set at a position corresponding to the user's screen touch operation or screen tilt operation, and the virtual camera position is moved according to the parameter.

[0140] Here, refer to Fig.11 The flowchart in Fig.10The processing in step S3 in corresponds to the flow of image processing in response to user operation.

[0141] Note that the coordinate system used in this image processing includes world coordinates, which are coordinates representing the world itself in a three-dimensional space within a virtual space, and virtual camera coordinates, which are coordinates unique to a virtual camera. Fig.12 The relationship between world coordinates and virtual camera coordinates is shown.

[0142] exist Fig.12 In the figure, it is assumed that the virtual camera 50 rotates in the pitch direction with the origin of the XYZ axis in the virtual camera coordinates as the rotation center on the virtual camera YZ plane including the virtual camera coordinate Y axis and the virtual camera coordinate Z axis. In this case, among the XYZ axes in the world coordinates, the world coordinate Y axis is represented by a dotted line in the direction from the lower side to the upper side in the figure.

[0143] That is, the world coordinates are used as indexes of display positions (coordinates) of virtual objects, etc., as unique absolute coordinates in the virtual space (coordinates that are not affected by the position of the virtual camera (the position of the user's viewpoint)). Therefore, the world coordinates do not have to be consistent with the virtual camera coordinates that are unique to the virtual camera, and Fig.12 In the example of FIG. 5 , the world coordinate Y axis and the virtual camera coordinate Y axis have different axis orientations.

[0144] Back to Fig.11 As described in , first, the image processing unit 123 performs screen touch operation processing corresponding to the number of fingers of the user contacting the screen of the display unit 106 based on the detection result detected by the screen touch detection unit 122.

[0145] In the case where it is determined in the determination process of step S11 that the number of fingers detected by the screen touch detection unit 122 is 1, the processes in steps S12 , S13 are performed.

[0146] That is, the image processing unit 123 sets the rotation amount in the yaw direction (hereinafter described as “Yaw 1”) according to the movement amount of the fingertip in the X direction during Δt ( S12 ).

[0147] Furthermore, the image processing unit 123 sets the movement amount in the Y-axis direction in the world coordinate system (hereinafter described as “ Y1 ”) according to the movement amount of the fingertip moving in the Y direction during Δt ( S13 ).

[0148] In the case where it is determined in the determination process of step S11 that the number of fingers detected by the screen touch detection unit 122 is 2, the processes in steps S14 to S16 are performed.

[0149] That is, the image processing unit 123 sets the movement amount in the X direction in the virtual camera coordinates (hereinafter described as “ X2 ”) according to the movement amount in the X direction of the midpoint between the two fingertips during Δt ( S14 ).

[0150] Furthermore, the image processing unit 123 sets the movement amount in the Y direction in the virtual camera coordinates (hereinafter described as “ Y2 ”) according to the movement amount in the Y direction of the midpoint between the two fingertips during Δt ( S15 ).

[0151] Furthermore, the image processing unit 123 sets a change amount of the rotation radius (hereinafter described as “ R2 ”) according to the increase or decrease in the distance between the two fingertips during Δt ( S16 ).

[0152] When the processing in step S13 or S16 ends, the processing proceeds to step S17. In addition, in the case where the number of detected fingers is determined to be zero in the determination processing in step S11, screen touch operation processing corresponding to the number of fingertips is unnecessary, and thus the processing proceeds to step S17.

[0153] Next, the image processing unit 123 performs a screen tilt operation process corresponding to the current tilt of the screen of the display terminal 10 based on the detection result detected by the terminal tilt detection unit 121 .

[0154] That is, the image processing unit 123 sets the amount of rotation in the yaw direction (hereinafter described as “Yaw 3”) according to the current tilt of the display terminal 10 in the yaw direction ( S17 ).

[0155] Furthermore, the image processing unit 123 sets a rotation amount in the pitch direction (hereinafter referred to as “Pitch 3”) according to the current tilt of the display terminal 10 in the pitch direction ( S18 ).

[0156] The image processing unit 123 sets the rotation radius of the virtual camera 50 to R2 (S19). In addition, the image processing unit 123 sets the rotation amount of the virtual camera 50 around the virtual object in the yaw direction and the rotation amount of the virtual camera 50 around the virtual object in the pitch direction to Yaw 1+Yaw 3 and Pitch 3, respectively (S20).

[0157] The image processing unit 123 sets the rotation center of the virtual camera 50 to move from the current position by X2, Y1+Y2 (S21). However, at this time, the virtual camera coordinates (X2, Y2) are converted to world coordinates, and then addition is performed after Y1 and the coordinate system are matched.

[0158] In this way, the image processing unit 123 sets parameters related to the following: the rotation radius and rotation center of the virtual camera device and the rotation of the virtual camera device in the pitch direction and yaw direction (rotation radius R2, rotation center (X2, Y1+Y2), rotation amount in the pitch direction (Pitch 3), rotation amount in the yaw direction (Yaw 1+Yaw 3)), thereby moving the position of the virtual camera device according to these parameters.

[0159] When the processing in step S21 ends, the processing returns to Fig.10 Step S3 in , and perform subsequent processing.

[0160] The flow of display processing compatible with the new system has been described above. In this display processing, the position of the virtual camera device 50 is changed in response to the user's operation by rotating the virtual camera device 50 in the yaw direction when the user performs a screen touch operation, and by rotating the virtual camera device 50 in the pitch direction and the yaw direction when the user performs a screen tilt operation. Note that at this time, the actual position of the display terminal 10 in the real space is not reflected in the movement of the virtual camera device 50.

[0161] With this arrangement, user confusion due to inversion or the like caused by a screen display having an inclination different from that of the screen of the display terminal 10 can be avoided, and a more intuitive operation of moving the virtual camera around a virtual object can be performed.

[0162] Here, if Fig.13 As shown, the correspondence between the screen touch operation and the screen tilt operation and the movement of the virtual camera 50 in the above-mentioned new system is summarized.

[0163] In the case where the tilt of the screen of the display terminal 10 is a yaw rotation by a screen tilt operation, the virtual camera 50 set in the virtual space rotates around the virtual object in the yaw direction according to the tilt of the screen ( Fig.11 (S17, S20 in Fig.

[0164] Furthermore, in the case where the tilt of the screen of the display terminal 10 is a pitch rotation by a screen tilt operation, the virtual camera 50 rotates around the virtual object in the pitch direction according to the tilt of the screen ( Fig.11 (S18, S20 in Fig.

[0165] In the case where a drag operation in the lateral direction (horizontal direction) is performed on the screen by a screen touch operation with one finger, the virtual camera 50 rotates around the virtual object in the yaw direction according to the movement amount of the fingertip ( Fig.11 (S12, S20 in).

[0166] Furthermore, in the case where a drag operation in the vertical direction (perpendicular direction) is performed on the screen by a screen touch operation with one finger, the rotation center of the virtual camera 50 moves in the Y-axis direction in the world coordinates according to the movement amount of the fingertip ( Fig.11 (S13, S21 in).

[0167] In the case where a drag operation in the lateral direction (horizontal direction) is performed by a screen touch operation with two fingers, the rotation center of the virtual camera 50 moves in the X-axis direction in the virtual camera coordinates according to the movement amount of the fingertips ( Fig.11 (S14, S21 in).

[0168] Furthermore, in the case where a drag operation in the vertical direction (perpendicular direction) is performed by a screen touch operation with two fingers, the rotation center of the virtual camera 50 moves in the Y-axis direction in the virtual camera coordinates according to the movement amount of the fingertips ( Fig.11 (S15, S21 in).

[0169] In the case where a pinch operation is performed by a screen touch operation with two fingers, the rotation radius of the virtual camera 50 increases ( Fig.11 Meanwhile, when the pinch operation is performed by two fingers through the screen touch operation, the rotation radius of the virtual camera device 50 is reduced ( Fig.11 (S16, S19 in the figure).

[0170] <2. Modification>

[0171] (Other examples of configuration)

[0172] Although it has been described above that the display terminal 10 performs image processing corresponding to the user's operation ( Fig.11 ), thereby realizing the movement of the virtual camera device 50 corresponding to the screen touch operation and the screen tilt operation in the new system, but the image processing corresponding to the user's operation ( Fig.11 ) can be performed by another device such as distribution server 20.

[0173] Specifically, in the display terminal 10, the communication unit 108 transmits the detection results of the terminal tilt detection unit 121 and the screen touch detection unit 122 to an external server such as the distribution server 20 via the network 30, thereby generating display data (or 3D data) in consideration of the detection results in the external server. Then, when the external server transmits (distributes) the display data (or 3D data), the display terminal 10 can display a 3D image corresponding to the user's operation based on the display data (or 3D data).

[0174] Furthermore, the above exemplifies the case where the display terminal 10 displays a 3D image corresponding to 3D data by executing the distributed 3D application, the display terminal 10 only needs to acquire 3D data or display data in some way, and does not necessarily need a 3D application.

[0175] Note that in the present disclosure, a system means a group of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via the network 30 and one device housing multiple modules in one housing are both systems.

[0176] (Example of input interface)

[0177] Although the case where an inertial measurement unit (IMU) that measures three-dimensional angular velocity and acceleration is used as an input interface for a screen tilt operation has been described above, another input interface such as a pressure-sensitive sensor may be used.

[0178] Specifically, the pressure-sensitive sensor can be used to turn on / off detection of screen tilt, and when the pressure-sensitive sensor is pressed by pressure corresponding to contact of a user's finger or the like, the virtual camera 50 can rotate in the yaw direction or the pitch direction.

[0179] Furthermore, although the case of detecting a user's drag operation in a lateral or vertical direction or the like by using the touch panel 110 as an input interface for a screen touch operation has been described above, another input interface such as a controller or a remote controller may be used.

[0180] Specifically, a controller or the like can be connected to the display terminal 10, and according to the user's operation on the controller or the like (for example, left key operation / right key operation or up key operation / down key operation), the virtual camera device can be rotated in the yaw direction, or the rotation center of the virtual camera device can be moved in the Y-axis direction in the world coordinate system.

[0181] Note that when the screen touch is in an unstable state due to damage of the touch panel 110 or the like at the time of the screen touch operation, the input interface may be switched from the touch panel 110 to the controller or the like.

[0182] (Example of initial value and limit)

[0183] Although initial values ​​and restrictions regarding the virtual camera 50 are not particularly described in the above description, initial values ​​or restrictions may be set for the rotation radius or movement area of ​​the rotation center, the rotation amount, the rotation rate, and the like of the virtual camera 50 .

[0184] Specifically, regarding the rotation radius or the rotation amount and the movement area of ​​the rotation center, an area that can be used by the user can be pre-set, such as the size or movement area of ​​a virtual object as an observation target, an area from which three-dimensional data can be acquired, or an observation prohibited angle.

[0185] Furthermore, when the rotation radius, the rotation center, the rotation amount, etc. exceed a predetermined limit value, feedback may be performed in a display mode different from the display mode of normal operation by changing the display (e.g., displaying in red) of a predetermined area (e.g., the left end or right end or the upper end or the lower end of the screen) on the screen of the display unit 106. Note that here, feedback may be performed not only by display but also by sound, vibration, etc.

[0186] Furthermore, in the display terminal 10, in the case of detecting a sudden shake, tilt, etc. of the screen that is unintentional during normal operation, the detection result can be ignored on the assumption that the shake, tilt, etc. is not an operation input by the user. Specifically, although it is assumed that the display terminal 10 such as a smart phone detects a certain vibration when the user using the display terminal 10 is on a vehicle such as a car or a train, the display terminal 10 can ignore the vibration.

[0187] (Example of screen touch operation)

[0188] As the above-mentioned screen touch operation, when a drag operation in the vertical direction is performed using two fingers, and the rotation center of the virtual camera device moves in the Y-axis direction of the virtual camera device according to the movement amount of the fingertips, the angle of the virtual camera device in the pitch direction can be set to 0 degrees (the state in which the virtual camera device faces forward) or -90 degrees (the state in which the virtual camera device faces directly downward) to force a transition to a predetermined state.

[0189] With this arrangement, the Y-axis direction in the virtual camera coordinates coincides with the Y-axis direction in the world coordinates, whereby the rotation center of the virtual camera moves in the vertical direction in the world coordinates, or the Y-axis direction in the virtual camera coordinates is parallel to the XZ plane, whereby the rotation center of the virtual camera moves in the horizontal direction in the world coordinates, and therefore, the user can easily understand the direction in which the rotation center of the virtual camera moves. Note that whether the angle of the virtual camera in the pitch direction is set to 0 degrees or -90 degrees can be determined by, for example, whether the angle of the virtual camera in the pitch direction is closer to 0 degrees or -90 degrees at this time.

[0190] In addition, as the above-mentioned screen touch operation, when a drag operation is performed in the vertical direction with one finger, the virtual camera device can be (temporarily) rotated in the pitch direction, and then the rotation in the pitch direction caused by the drag operation can be restored to the original state at a predetermined timing (for example, when the user releases the finger from the screen after the drag operation). With this arrangement, the rotation of the virtual camera device in the pitch direction can be greatly reduced by the drag operation in the vertical direction with one finger.

[0191] In addition, a button UI for changing the virtual object (rotation center) of the observation target observed by the user may be provided on the screen of the display terminal 10, and when the button UI is pressed, a predetermined virtual object at another position may be selected as the observation target to move the rotation center. Specifically, when the button UI is pressed while the user's viewpoint is focused on a certain player, the viewpoint moves to the position of another player.

[0192] As described above, in an image processing unit (image processing unit 123) in an image processing device compatible with the new system (such as the display terminal 10 or the distribution server 20), the pitch rotation and yaw rotation of the display terminal 10 are determined based on a measurement signal measured by the display terminal 10 (such as an IMU signal obtained by measuring three-dimensional angular velocity and acceleration), and based on the determination results of the pitch rotation and yaw rotation of the display terminal 10, the virtual camera device rotates around the virtual object in at least one of the pitch direction or the yaw direction, the touch operation on the touch panel 110 of the display terminal 10 is determined, and based on the determination result of the touch operation (for example, a drag operation in the horizontal direction using one finger), the virtual camera device rotates around the virtual object in the yaw direction in the virtual space.

[0193] That is, the image processing unit of the image processing device compatible with the new system does not reflect the determination result of the touch operation (for example, a drag operation in the vertical direction using one finger) in the rotation of the virtual camera device in the pitch direction in the virtual space. In addition, at this time, the actual position of the display terminal 10 in the real space is not reflected in the movement of the virtual camera device.

[0194] With this arrangement, in the display terminal 10, the virtual camera is rotated in the yaw direction by the operation of touching the screen at the screen touch operation, and the virtual camera is rotated in the pitch direction and the yaw direction by the tilt of the screen at the screen tilt operation, thereby changing the position of the virtual camera relative to the virtual object. Therefore, it is possible to prevent the screen display having a tilt different from the tilt of the screen to avoid confusion of the user due to the appearance of inversion, etc., and perform a more intuitive operation of moving the virtual camera around the observation target (virtual object).

[0195] Therefore, the operability for the user can be improved. In addition, by moving the virtual camera using intuitive operations such as tilting the display terminal 10 or touching the screen, the user can enjoy observing a virtual object as an observation target.

[0196] Note that the above-mentioned Patent Document 1 discloses a technology for moving a virtual camera when the acceleration and each speed exceed a predetermined threshold value. In addition, the above-mentioned Patent Document 2 discloses a game system that displays a camera viewpoint video corresponding to the tilt of the housing. However, Patent Documents 1 and 2 do not disclose technologies related to screen touch operations.

[0197] In addition, the above-mentioned patent document 3 discloses a technology for moving a virtual camera according to displacement of the XYZ axis (IMU signal). However, although the movement of the virtual camera using displacement of the XYZ axis is disclosed, the patent document 3 does not disclose an operation method combined with a screen touch operation.

[0198] Meanwhile, there is a hybrid operation method which is a combination of a screen tilt operation and a screen touch operation, which is implemented by an application for omnidirectional video viewing and listening (existing hybrid application) or the like.

[0199] However, with the existing hybrid application, in the case where the virtual camera device is rotated in the pitch direction by a drag operation in the vertical direction during a screen touch operation, even if the user holds the display terminal (e.g., a smartphone) in the horizontal direction, the sky above or the ground below may inevitably be visible. Therefore, the user may be confused as to which direction the viewpoint faces.

[0200] Note that, with the existing hybrid application, in the case where the tilt of the screen of the display terminal is a yaw rotation or a pitch rotation by a screen tilt operation, the virtual camera device rotates in the yaw direction or the pitch direction according to the tilt of the screen. In addition, with the existing hybrid application, in the case where a drag operation is performed in the horizontal direction or the vertical direction on the screen by a screen touch operation using one finger, the virtual camera device rotates in the yaw direction or the pitch direction according to the movement amount of the fingertip.

[0201] <3. Computer Configuration>

[0202] The above series of processing may be performed by hardware or may be performed by software (eg Fig.10 In the case where a series of processes are executed by software, a program included in the software is installed on a computer in each device.

[0203] Fig.14 : is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0204] In the computer, a central processing unit (CPU) 1001, a read only memory (ROM) 1002, and a random access memory (RAM) 1003 are connected to each other via a bus 1004. In addition, an input / output interface 1005 is connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0205] The input unit 1006 includes a microphone, a keyboard, a mouse, etc. The output unit 1007 includes a speaker, a display, etc. The storage unit 1008 includes a hard disk, a nonvolatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives a removable recording medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0206] In the computer configured as described above, the above-described series of processes is performed by the CPU 1001 loading a program recorded in the ROM 1002 or the storage unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing the program.

[0207] For example, the program executed by the computer (CPU 1001) can be provided by recording it on the removable recording medium 1011 as a package medium, etc. Furthermore, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0208] In the computer, the program can be installed on the storage unit 1008 via the input / output interface 1005 by attaching the removable recording medium 1011 to the drive 1010. In addition, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed on the storage unit 1008. In addition, the program can be pre-installed on the ROM 1002 or the storage unit 1008.

[0209] Here, in this specification, the processing performed by a computer according to a program does not necessarily have to be performed in a time series in the order described in the flowchart. That is, the processing performed by a computer according to a program also includes processing performed in parallel or individually (for example, parallel processing or object processing). In addition, the program may be processed by one computer (processor) or may be subjected to distributed processing by multiple computers.

[0210] Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present technology.

[0211] In addition, display processing ( Fig.10 ) and image processing ( Fig.11 ) Each step in can be performed by one device, or can be performed by being shared by multiple devices. In addition, when multiple processes are included in one step, in addition to being performed by one device, the multiple processes included in one step can be performed by being shared by multiple devices.

[0212] Note that the following configurations can be used with this technique. (1)

[0214] An image processing device, comprising:

[0215] An image processing unit, wherein the image processing unit:

[0216] A virtual camera device is provided in a three-dimensional virtual space to designate an area in the virtual space, the area being displayed by a display terminal,

[0217] performing control so that the display terminal displays a background image representing the virtual space and a virtual object set in the background image,

[0218] determining a pitch rotation of the display terminal based on a measurement signal measured by the display terminal,

[0219] Based on the determination result of the pitch rotation of the display terminal, the virtual camera device is rotated around the virtual object in the pitch direction,

[0220] determining a touch operation on a touch panel of the display terminal, and

[0221] Based on the determination result of the touch operation, the virtual camera is rotated around the virtual object in a yaw direction in the virtual space. (2)

[0223] The image processing device according to (1),

[0224] The image processing unit does not reflect the determination result of the touch operation in the rotation of the virtual camera in the pitch direction in the virtual space. (3)

[0226] The image processing device according to (1) or (2),

[0227] wherein the background image comprises a wide-angle image having a viewing angle of at least 180 degrees or wider, and

[0228] The image processing unit does not reflect the actual position of the display terminal in the real space in the movement of the virtual camera. (4)

[0230] The image processing device according to (1),

[0231] The rotation of the virtual camera device in the pitch direction and the yaw direction includes the rotation of the virtual camera device around the virtual object. (5)

[0233] The image processing device according to any one of (1) to (4),

[0234] Wherein, the image processing unit:

[0235] determining a pitch rotation and a yaw rotation of the display terminal based on a measurement signal measured by the display terminal, and

[0236] Based on the determination result of the pitch rotation and the yaw rotation of the display terminal, the virtual camera is rotated around the virtual object in at least one of the pitch direction or the yaw direction. (6)

[0238] The image processing device according to (2),

[0239] Here, when a predetermined operation in a horizontal direction is performed on the touch panel as the touch operation, the image processing unit causes the virtual camera to rotate around the virtual object in the yaw direction in the virtual space. (7)

[0241] The image processing device according to (6),

[0242] Here, when a predetermined operation in a vertical direction is performed on the touch panel as the touch operation, the image processing unit does not reflect the touch operation in the rotation of the virtual camera in the pitch direction in the virtual space. (8)

[0244] The image processing device according to (7),

[0245] Here, when a predetermined operation in a vertical direction is performed on the touch panel, the image processing unit moves the rotation center of the virtual camera device in a Y-axis direction in a world coordinate system from a current position. (9)

[0247] The image processing device according to (7) or (8),

[0248] The predetermined operation includes a dragging operation of a user's finger. (10)

[0250] The image processing device according to any one of (1) to (3),

[0251] The measurement signal includes a signal related to three-dimensional angular velocity and acceleration, and the three-dimensional angular velocity and acceleration are measured when the user performs an operation of tilting the display terminal. (11)

[0253] The image processing device according to any one of (1) to (10), configured as the display terminal. (12)

[0255] The image processing device according to any one of (1) to (11),

[0256] Wherein, the display terminal includes a mobile terminal. (13)

[0258] An image processing method, comprising:

[0259] By the image processing device:

[0260] Setting a virtual camera device in a three-dimensional virtual space to designate an area in the virtual space, the area being displayed by a display terminal;

[0261] executing control so that the display terminal displays a background image representing the virtual space and a virtual object set in the background image;

[0262] determining a pitch rotation of the display terminal based on a measurement signal measured by the display terminal;

[0263] Based on the determination result of the pitch rotation of the display terminal, rotating the virtual camera device around the virtual object in the pitch direction;

[0264] determining a touch operation on a touch panel of the display terminal; and

[0265] Based on the determination result of the touch operation, the virtual camera is rotated around the virtual object in a yaw direction in the virtual space. (14)

[0267] A program causing a computer to function as an image processing device, the image processing device comprising:

[0268] An image processing unit, wherein the image processing unit:

[0269] A virtual camera device is provided in a three-dimensional virtual space to designate an area in the virtual space, the area being displayed by a display terminal,

[0270] performing control so that the display terminal displays a background image representing the virtual space and a virtual object set in the background image,

[0271] determining a pitch rotation of the display terminal based on a measurement signal measured by the display terminal,

[0272] Based on the determination result of the pitch rotation of the display terminal, the virtual camera device is rotated around the virtual object in the pitch direction,

[0273] determining a touch operation on a touch panel of the display terminal, and

[0274] Based on the determination result of the touch operation, the virtual camera is rotated around the virtual object in a yaw direction in the virtual space. (15)

[0276] A recording medium having recorded thereon a program for causing a computer to function as an image processing unit, the image processing unit:

[0277] A virtual camera device is provided in a three-dimensional virtual space to designate an area in the virtual space, the area being displayed by a display terminal,

[0278] performing control so that the display terminal displays a background image representing the virtual space and a virtual object set in the background image,

[0279] determining a pitch rotation of the display terminal based on a measurement signal measured by the display terminal,

[0280] Based on the determination result of the pitch rotation of the display terminal, the virtual camera device is rotated around the virtual object in the pitch direction,

[0281] determining a touch operation on a touch panel of the display terminal, and

[0282] Based on the determination result of the touch operation, the virtual camera is rotated around the virtual object in a yaw direction in the virtual space.

[0283] Reference numerals list

[0284] 1 3D Image Display System

[0285] 10, 10-1 to 10-N Display terminal

[0286] 20 Distribution Server

[0287] 30 Network

[0288] 50 Virtual Camera Device

[0289] 60 Virtual Objects

[0290] 101 Control Unit

[0291] 102 Memory

[0292] 103 Sensors

[0293] 104 Image Processing Unit

[0294] 105 Storage Unit

[0295] 106 Display Unit

[0296] 107 Input unit

[0297] 108 Communication Unit

[0298] 110 Touch Panel

[0299] 121 Terminal tilt detection unit

[0300] 122 Screen touch detection unit

[0301] 123 Image Processing Unit

[0302] 124 Display control unit

[0303] 1001 CPU

Claims

1. An image processing device, comprising: An image processing unit, wherein the image processing unit: A virtual camera device is provided in a three-dimensional virtual space to designate an area in the virtual space, the area being displayed by a display terminal, performing control so that the display terminal displays a background image representing the virtual space and a virtual object set in the background image, determining a pitch rotation of the display terminal based on a measurement signal measured by the display terminal, Based on the determination result of the pitch rotation of the display terminal, the virtual camera device is rotated around the virtual object in the pitch direction, determining a touch operation on a touch panel of the display terminal, when a predetermined operation in the horizontal direction is performed on the touch panel as the touch operation, rotating the virtual camera around the virtual object in the yaw direction in the virtual space, and When a predetermined operation in the vertical direction is performed on the touch panel as the touch operation, the touch operation is not reflected in the rotation of the virtual camera device in the pitch direction in the virtual space, but the rotation center of the virtual camera device is moved in the Y-axis direction in the world coordinates from the current position.

2. The image processing device according to claim 1, in, The background image comprises a wide-angle image having a viewing angle of at least 180 degrees or wider, and The image processing unit does not reflect the actual position of the display terminal in the real space in the movement of the virtual camera.

3. The image processing device according to claim 1, in, The image processing unit: determining a pitch rotation and a yaw rotation of the display terminal based on a measurement signal measured by the display terminal, and Based on the determination results of the pitch rotation and the yaw rotation of the display terminal, the virtual camera is rotated around the virtual object in at least one of the pitch direction or the yaw direction.

4. The image processing device according to claim 1, in, The predetermined operation includes a drag operation of a user's finger.

5. The image processing device according to claim 1, in, The measurement signal includes a signal related to three-dimensional angular velocity and acceleration measured when a user performs an operation of tilting the display terminal. 6 . The image processing device according to claim 1 , configured as the display terminal.

7. The image processing device according to claim 1, in, The display terminal includes a mobile terminal.

8. An image processing method, comprising: By the image processing device: Setting a virtual camera device in a three-dimensional virtual space to designate an area in the virtual space, the area being displayed by a display terminal; executing control so that the display terminal displays a background image representing the virtual space and a virtual object set in the background image; determining a pitch rotation of the display terminal based on a measurement signal measured by the display terminal; Based on the determination result of the pitch rotation of the display terminal, rotating the virtual camera device around the virtual object in the pitch direction; Determining a touch operation on a touch panel of the display terminal; When a predetermined operation in the horizontal direction is performed on the touch panel as the touch operation, rotating the virtual camera around the virtual object in the yaw direction in the virtual space; as well as When a predetermined operation in the vertical direction is performed on the touch panel as the touch operation, the touch operation is not reflected in the rotation of the virtual camera device in the pitch direction in the virtual space, but the rotation center of the virtual camera device is moved in the Y-axis direction in the world coordinates from the current position.

9. A recording medium having a program recorded thereon, the program for causing a computer to function as an image processing unit, the image processing unit: A virtual camera device is provided in a three-dimensional virtual space to designate an area in the virtual space, the area being displayed by a display terminal, performing control so that the display terminal displays a background image representing the virtual space and a virtual object set in the background image, determining a pitch rotation of the display terminal based on a measurement signal measured by the display terminal, Based on the determination result of the pitch rotation of the display terminal, the virtual camera device is rotated around the virtual object in the pitch direction, determining a touch operation on a touch panel of the display terminal, when a predetermined operation in the horizontal direction is performed on the touch panel as the touch operation, rotating the virtual camera around the virtual object in the yaw direction in the virtual space, and When a predetermined operation in the vertical direction is performed on the touch panel as the touch operation, the touch operation is not reflected in the rotation of the virtual camera device in the pitch direction in the virtual space, but the rotation center of the virtual camera device is moved in the Y-axis direction in the world coordinates from the current position.

Images