Program, information processing method, and portable terminal

Abstract

To improve usability when providing a user with experience of VR etc. by combining a predetermined device with a portable terminal.SOLUTION: A program is executed in a portable terminal 100 which can be incorporated and used in a first device 10 worn on a user's head. In a state where a user wears the first device 10 incorporating the portable terminal 100, the program causes a processor of the portable terminal 100 to perform steps of: displaying a selection screen including at least one content which can be selected; acquiring information regarding motion or attitude of the user or the portable terminal 100; and performing display control of the selection screen or receiving selection of one content included in the selection screen to control processing of one content, on the basis of the information regarding the motion or the attitude.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a program, an information processing method, and a portable terminal. [Background technology]

[0002] Conventionally, there is known a technology for providing a virtual space to a user using a device such as a head-mounted display. For example, Patent Document 1 describes a technology in which a head-mounted display detects whether the user's posture is tilted up, down, left, or right, and the content displayed on the display unit of the head-mounted display is controlled based on the posture and the user's operation.

[0003] In recent years, a virtual space has also been provided to a user by combining a device worn on the user's head with a portable terminal such as a smartphone. For example, Non-Patent Document 1 describes a method in which a smartphone with a video playback app installed is incorporated into VR (Virtual Reality) glasses that act as a housing, and VR videos are played on the smartphone, thereby providing VR videos to a user wearing the VR glasses. Non-Patent Document 1 also describes that video playback can be controlled by line of sight while watching the video. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-208370 [Non-patent literature]

[0005] [Non-Patent Document 1] “What is the DMM VR video player app?”, [online], [Retrieved September 24, 2021], Internet < https: / / www.dmm.com / digital / vr / howto.html> Summary of the Invention [Problem to be solved by the invention]

[0006] If, as in Patent Document 1, tilting of the posture up, down, left, and right is assigned as an input for display control, it is difficult to respond when more than four types of operations are required, and it is difficult to realize a variety of detailed operations, for example, when performing operations related to video playback.

[0007] The technology of Non-Patent Document 1 provides a VR space to the user while watching a video and enables operation by line of sight, but the login screen and video selection screen are 2D screens, and the only operations possible on these screens are touch operations on the smartphone, such as tapping. Therefore, when watching a VR video using the technology of Non-Patent Document 1, the user must first perform a touch operation on the smartphone and then attach the smartphone to the VR glasses when it is time to watch the VR video. Furthermore, if the user wants to watch a different VR video while watching a VR video, the user must end the video viewing, then remove the smartphone from the VR glasses, select the VR video to watch by touching the smartphone, and then attach the smartphone to the VR glasses again, which is a cumbersome operation for the user.

[0008] One aspect of the present disclosure aims to improve usability when a specific device and a mobile terminal are combined to provide users with experiences such as AR (Augmented Reality), MR (Mixed Reality), SR (Substitutional Reality), and VR. [Means for solving the problem]

[0009] According to one embodiment shown in the present disclosure, A program executed in a portable terminal that can be incorporated into a first device worn on a user's head, In a state where the user wears the first device incorporating the portable terminal, the program causes a processor of the portable terminal to displaying a selection screen including at least one selectable content; acquiring information about the movement or attitude of the user, the first device, or the mobile terminal; a step of controlling the display of the selection screen based on the information about the movement or posture, or accepting a selection of one content item included in the selection screen and controlling processing of the one content item; A program is provided to execute the above.

[0010] According to another embodiment of the present disclosure, A program executed on a portable terminal capable of communicating with a second device worn on a predetermined part of a user's head, In a state in which the portable terminal and the second device are communicatively connected, the program causes a processor of the portable terminal to displaying a selection screen including at least one selectable content via the second device; acquiring information about a movement or attitude of the user or the second device; Controlling the display of the selection screen on the second device based on the information about the movement or posture, or accepting a selection of one piece of content included in the selection screen on the second device and controlling processing of the one piece of content; A program is provided to execute the above. [Effects of the Invention]

[0011] According to one embodiment of the present disclosure, usability can be improved when a specific device and a mobile terminal are combined to provide a user with an experience such as AR, MR, SR, or VR. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 2 is a schematic diagram showing an example of a combination of a first device and a portable terminal according to the first embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of a portable terminal according to the first embodiment. [Figure 3] 10 is a flowchart illustrating an example of processing according to the first embodiment. [Figure 4] 10 is a diagram showing an XZ cross section of a viewable area in a virtual space when selecting content, as viewed from the Y direction. [Figure 5] FIG. 10 is a schematic diagram showing an example of a field of view image of a user when selecting content. [Figure 6] 10 is a diagram showing an XZ cross section of a field of view in a virtual space when watching a video, as viewed from the Y direction. [Figure 7] FIG. 10 is a schematic diagram showing an example of a field of view image of a user when watching a video. [Figure 8] FIG. 10 is a schematic diagram showing an example of a combination of a second device and a portable terminal according to a second embodiment. [Figure 9] FIG. 10 is a block diagram illustrating an example of a functional configuration of a second device and a portable terminal according to a second embodiment. [Figure 10] 10 is a flowchart illustrating an example of processing according to the second embodiment. [Figure 11] FIG. 10 is a schematic diagram showing an example of a field of view image of a user during a search. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of this technical idea will be described in detail with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated. In one or more embodiments shown in this disclosure, elements included in each embodiment can be combined with each other, and the combined result also forms part of the embodiment shown in this disclosure.

[0014] The following mainly describes a case where a VR experience is provided to a user by combining a specific device with a portable terminal, but this disclosure can be applied to all so-called XR technologies such as AR, MR, SR, etc. Note that, below, XR experiences such as VR experiences, AR experiences, MR experiences, and SR experiences are also collectively referred to as "VR experiences, etc."

[0015] In this specification, "content" refers to anything that can be selected on a display screen. Specifically, "content" includes videos, still images, music, e-books, games, text, links, application programs, products sold online, etc.

[0016] [First embodiment: configuration example] First, an example of a combination of a first device 10 and a portable terminal 100 according to the first embodiment will be described with reference to Fig. 1. The first embodiment is an example of a case where the portable terminal 100 is incorporated into the first device 10 and used.

[0017] In this example, the first device 10 includes an opening / closing unit 11, a main body 12, a hinge unit 13, and a mounting unit 14. The opening / closing unit 11 and the main body 12 are rotatably connected around the hinge unit 13. The opening / closing unit 11 has a terminal holder (not shown) on the main body 12 side. The portable terminal 100 can be attached and detached to the terminal holder. The terminal holder may be provided on the opening / closing unit 11 side of the main body 12. The main body 12 has two openings (not shown) in which lenses are fitted. The mounting unit 14 is a member for mounting the first device 10 on the user's head. In this example, the mounting unit 14 includes a band that passes through the temples and back of the user's head when worn, and a band that includes the top of the user's head.

[0018] When storing the portable terminal 100, first, the opening / closing unit 11 is rotated to open the opening / closing unit 11, and then the portable terminal 100 is fixed to the terminal holder. Then, the opening / closing unit 11 is rotated in the opposite direction to when it was opened to close the opening / closing unit 11. The lower diagram in FIG. 1 shows the state in which the portable terminal 100 is incorporated into the first device 10 in this manner. In this state, the user wears the first device 10 on their head using the wearing unit 14. Then, the user can experience VR by viewing an image displayed on the portable terminal 100 through the opening in which the lens is fitted.

[0019] There are no particular limitations on the first device 10, as long as it can be worn on the user's head and can incorporate the portable terminal 100. The first device 10 may have a conventionally known structure in addition to or instead of the above-mentioned parts, or may not have one or more of the above-mentioned parts. For example, the first device 10 may have a mounting part 14 structured like the temples of glasses, or may have a structure that is supported by the hand without having a mounting part 14. Furthermore, the first device 10 may be configured without the opening / closing part 11, but with an insertion hole for inserting the portable terminal 100.

[0020] The portable terminal 100 is, for example, a smartphone or a tablet. Fig. 2 is a block diagram showing an example of the functional configuration of the portable terminal 100. As shown in Fig. 2, the portable terminal 100 includes an antenna 110, a wireless communication IF 120, a touch screen 130, an input / output IF 140, a storage unit 150, an audio processing unit 160, a microphone 162, a speaker 161, a sensor unit 170, and a control unit 190.

[0021] The antenna 110 radiates the signal emitted by the mobile terminal 100 into space as a radio wave. The antenna 110 also receives the radio wave from space and provides the received signal to the wireless communication IF 120.

[0022] The wireless communication IF 120 performs modulation / demodulation processing and the like for transmitting and receiving signals via the antenna 110 etc. so that the mobile terminal 100 can communicate with other communication devices. The wireless communication IF 120 is a communication module for wireless communication that includes a tuner, a high-frequency circuit etc., and performs modulation / demodulation and frequency conversion of the wireless signals transmitted and received by the mobile terminal 100, and provides the received signals to the control unit 190.

[0023] Touch screen 130 accepts input from the user and outputs information to the user on display 132. Touch screen 130 includes a touch panel 131 for accepting user input operations, and a display 132 for displaying menu screens, virtual space images, and the like on the screen. Touch panel 131 can detect the approach of a user's finger or the like by using, for example, a capacitance type. Display 132 can be realized by, for example, an LCD (Liquid Crystal Display), an organic EL (Electroluminescence) display, or other display device.

[0024] The input / output IF 140 functions as an interface for receiving information input to the mobile terminal 100 and outputting information to the outside of the mobile terminal 100 .

[0025] The storage unit 150 is configured by a flash memory, a RAM (Random Access Memory), etc. The storage unit 150 can store various data received by the mobile terminal 100. The storage unit 150 stores a program 151 that can provide a VR experience to the user.

[0026] The audio processing unit 160 modulates and demodulates audio signals. The audio processing unit 160 modulates a signal provided from the microphone 162 and provides the modulated signal to the control unit 190. The audio processing unit 160 also provides the audio signal to the speaker 161. The audio processing unit 160 is realized, for example, by a processor for audio processing. The microphone 162 functions as an audio input unit that receives an input of an audio signal and outputs it to the control unit 190. The speaker 161 functions as an audio output unit that outputs the audio signal to the outside of the mobile terminal 100.

[0027] The sensor unit 170 acquires information about the movement or attitude of the user or the portable terminal 100 (hereinafter, also referred to as "attitude information, etc."). The sensor unit 170 includes, for example, a camera and motion sensors such as a gyro sensor and an acceleration sensor. In this embodiment, the user wears the first device 10 incorporating the portable terminal 100 on his / her head. In this state, if information about the movement or attitude of the portable terminal 100 is acquired by the motion sensor, this also means that information about the movement or attitude of the user's head is acquired. Furthermore, the sensor unit 170 may acquire information about the user's movement, such as the user's line of sight, by a camera when the first device 10 incorporating the portable terminal 100 is worn on the user's head. In other words, the information about the user's movement may include information about the movement of the user's eyes and line of sight. The sensor unit 170 may be provided in the first device 10. In this configuration, the attitude information of the first device 10 is transmitted, for example, to the portable terminal 10 connected via a wired or wireless connection (e.g., Bluetooth (registered trademark), WiFi, etc.).

[0028] The control unit 190 controls the operation of the mobile terminal 100 by reading and executing a program stored in the storage unit 150. The control unit 190 is realized by, for example, an application processor.

[0029] The control unit 190 loads and executes the program 151 to thereby perform the functions of a transmission / reception unit 191, an operation reception unit 192, a display control unit 193, and a processing control unit 194.

[0030] The transmitting / receiving unit 191 receives various information from an external device via a wireless network, for example, and transmits various information to an external device. The transmitting / receiving unit 191 receives information resources from a server corresponding to a predetermined URL, for example. Specifically, the transmitting / receiving unit 191 receives information resources for displaying a web screen and information resources for displaying VR content such as VR videos from the server.

[0031] The operation receiving unit 192 determines a user's operation on the touch screen 130. Specifically, the operation receiving unit 192 determines a user's operation such as a so-called "approach operation," "release operation," "tap operation," "double tap operation," "long press operation (long touch operation)," "drag operation (swipe operation)," "move operation," and "flick operation."

[0032] The operation accepting unit 192 also accepts information such as posture acquired by the sensor unit 170 as an operation input. The operation accepting unit 192 accepts the information such as posture as, for example, a pointer movement operation. For example, when the pointer is over a predetermined selection area for a time period equal to or longer than a predetermined value, the operation accepting unit 192 accepts the selection of one piece of content corresponding to the selection area. The predetermined value may be changeable by the user. For example, when a predetermined operation is performed on the portable terminal 100 while the pointer is over the predetermined selection area, the operation accepting unit 192 may accept the selection of one piece of content corresponding to the selection area. The predetermined operation on the portable terminal 100 may be a direct operation on the portable terminal 100 by the user, or an indirect operation of operating the portable terminal 100 via the first device 10 or the like. The operation for selecting the selection area may be any other operation that can be performed while the first device 10 is worn on the head, such as pressing a physical button on the portable terminal 100. The operation receiving unit 192 may be configured to receive information such as posture as an operation input, triggered by, for example, the start of a VR mode.

[0033] The display control unit 193 controls the content displayed on the display 132. The display control unit 193 can perform display in two modes, for example, 2D mode and VR mode. The 2D mode is the same as the conventional one, so a description thereof will be omitted.

[0034] VR mode is a mode in which content such as web pages and videos corresponding to a specified URL is displayed in a VR space, providing the user with a VR experience. VR space is a space in which the user can freely change their viewpoint and move within the virtual space by accepting, for example, posture and other information as input operations. In the VR space, the web pages, videos, etc. may be displayed so as to unfold over a specified angle (e.g., 360 degrees or 180 degrees) in the VR space, or may be displayed unfolded on a flat display area provided in the VR space.

[0035] In VR mode, the display control unit 193 displays a selection screen including at least one selectable content and controls the display content of the selection screen based on posture and other information. The selection screen may include an image area or text area corresponding to at least one content and a selection area. The image area is an area that displays, for example, a thumbnail image or icon image of the content, or other images related to the content. The text area is an area that displays, for example, text information related to the content. The selection screen may include both an image area and a text area. The selection area is an area for selecting content and may be located in a position different from the image area or text area, or may be located in a position overlapping the image area or text area. From the viewpoint of reducing the likelihood of erroneous selection operations, it is preferable that the selection area be smaller than the image area or text area. Furthermore, the display control unit 193 performs, for example, scrolling the selection screen, screen transitions such as web page transitions, zooming in or out of images included in the selection screen, search and display of search results, etc., based on posture and other information. The display control unit 193 also displays images generated by the processing control unit 194 performing content-related processing.

[0036] Furthermore, the display control unit 193 can display virtual controls such as a virtual keyboard and a playback UI (User Interface) related to video playback. Each key constituting the virtual keyboard and each button constituting the playback UI are selection areas that can be selected by a pointer that moves based on information such as posture.

[0037] An input receiving UI such as a virtual keyboard or a playback UI may be displayed, for example, when an operation input for displaying the input receiving UI (e.g., shaking one's head) is made. Selection areas other than the input receiving UI may also be displayed when an operation input for displaying them is made. A pointer may be displayed, for example, when the input receiving UI or other selection areas are displayed, or when an operation input for displaying the pointer is made. Furthermore, one or more of the input receiving UI, other selection areas, and pointer may be hidden when an operation input for hiding them is made, or when a predetermined time has passed since the start of display, or when no input is received or no operation input is made for a predetermined time.

[0038] When the operation receiving unit 192 receives a selection of one piece of video content on the selection screen, the processing control unit 194 starts controlling processing related to the selected piece of video content. The processing related to the content is not particularly limited, but may include, for example, adding the content to favorites, displaying detailed information (for example, comments, reviews, star ratings, etc.), and moving to a purchase page. The processing control unit 194 also includes a playback control unit 195 that controls processing related to content playback among the processing related to the content. When video content is selected on the selection screen, for example, the selection screen transitions to a playback screen for playing the video content, and the video content is played on the playback screen. Specifically, the video content is played on an area spanning a predetermined angle in the VR space. When the operation receiving unit 192 receives an operation input to end the playback of the video content, for example, the playback screen transitions to the selection screen.

[0039] Note that the playback control unit 195 may play back video content in 2D mode. For example, the playback control unit 195 may play back video content in 2D on a flat display area provided in a VR space.

[0040] Furthermore, the playback control unit 195 controls various operations when viewing content. Specifically, the playback control unit 195 controls one or more of the operations of playing, rewinding, fast-forwarding, and stopping the content.

[0041] [First embodiment: Operation processing] Next, a case where video content is played back will be described as a specific example of the operational processing by the mobile terminal 100 according to the first embodiment with reference to Fig. 3 to Fig. 7. Fig. 3 is a flowchart showing an example of the processing according to the first embodiment. Note that the order of the processing constituting the flowchart described in this specification may be random, and may be executed in parallel, as long as no contradiction or inconsistency occurs in the processing content.

[0042] 3 is executed when the VR mode is selected and the user wears the first device 10 incorporating the portable terminal 100 on his / her head. That is, the processes from step S1 to step S9 can be executed while the user is wearing the first device 10, without removing the first device 10.

[0043] First, in step S1, control unit 190 displays a selection screen including multiple video contents on display 132. The selection screen includes thumbnails and selection areas corresponding to the video contents. A pointer that moves based on information such as posture is also displayed on the selection screen.

[0044] Next, in step S2, the control unit 190 acquires information such as posture from the sensor unit 170. Next, in step S3, the control unit 190 moves the pointer based on the information such as posture. When the pointer moves onto the selection area, time counting begins. If the time the pointer has been over the selection area is equal to or greater than a predetermined value (e.g., approximately 0.3 to 3 seconds) (Yes in step S4), the control unit 190 accepts the selection of video content corresponding to the selection area in step S5. On the other hand, if the time the pointer has been over the selection area is less than the predetermined value and the pointer has moved out of the selection area (No in step S4), the counted time is reset, and the process returns to step S2. Note that the "predetermined value" may be a value set as an initial value or a value arbitrarily set by the user. Furthermore, the "predetermined value" may be set to a different value depending on the location or type of the selection area. Specifically, the "predetermined value" may be set longer for selection areas where erroneous selection can have a significant negative effect, such as a selection area for logging out or a selection area for making a final decision on a product purchase, than for other selection areas.

[0045] Steps S1 to S5 will now be described in detail with reference to Figures 4 and 5. Figure 4 is a diagram showing an XZ cross section of a field of view area in virtual space 401 when content is selected, as viewed from the Y direction. Figure 5 is a schematic diagram showing an example of a field of view image 501 of the user when content is selected. In steps S1 to S5, for example, virtual space 401 as shown in Figure 4 is generated, and field of view image 501 as shown in Figure 5 is provided to the user.

[0046] In the example of FIG. 4, a virtual camera 402 and a display area 403 are arranged in a virtual space 401. The virtual space 401 is a VR space unfolded over 360 degrees. A dotted line extending from the virtual camera 402 indicates the user's field of view. The virtual camera 402 can move and change direction based on information such as its posture. The display area 403 is an area where a selection screen including multiple contents is displayed. In this example, the shape of the display area 403 is a television type as shown in FIG. 5, but is not limited to this and may be another shape. The shape of the display area 403 may be, for example, a three-dimensional shape such as a sphere.

[0047] A field of view image 501 shown in Fig. 5 is, for example, an image captured by virtual camera 402 of virtual space 401 shown in Fig. 4, and is an image within the field of view area indicated by the dotted line. In the example of Fig. 5, a display area 403 is displayed in field of view image 501. Also displayed within display area 403 are scroll buttons 502 and 503, thumbnails 504a to 504f and selection areas 505a to 505f corresponding to contents A to F, respectively, and an end button 507.

[0048] Note that "Watch List" is displayed at the top of display area 403. That is, in the example of Fig. 5, a watch list is displayed as a selection screen, and contents A to F are video contents that the user has registered in the watch list.

[0049] A star-shaped pointer 506 is displayed in approximately the center of the field of view image 501. The pointer 506 moves based on the user's head posture or line of sight, the posture of the mobile terminal 100, or the like. For example, when the user tilts their head to look upward or moves their line of sight upward, the pointer 506 also moves upward, and when the user tilts their head to look right or moves their line of sight right, the pointer 506 also moves rightward. Note that the virtual camera 402 also moves or changes direction based on the posture of the mobile terminal 100, and the field of view image 501 can also change accordingly.

[0050] For example, when pointer 506 is moved onto selection area 505a and the time that pointer 506 remains on selection area 505a reaches or exceeds a predetermined value, the selection of content A is accepted and playback control begins in step S6, which will be described later. Similarly, when another content is to be selected, pointer 506 is moved onto the selection area corresponding to that content and the time that pointer 506 remains on that selection area reaches or exceeds a predetermined value.

[0051] Scroll buttons 502 and 503 are selection areas for scrolling the selection screen displayed in display area 403. When pointer 506 is on scroll button 502 for a predetermined time or more, the selection screen displayed in display area 403 scrolls upward, and other content registered in the watch list is displayed. In the case of scroll button 503, the selection screen scrolls downward. Exit button 507 is a selection area for exiting VR mode.

[0052] 4 and 5, the selection screen is displayed in 2D in the display area 403 arranged in the virtual space 401, which is a VR space, but the selection screen may be displayed so as to unfold over a predetermined angle in the virtual space 401. In other words, the selection screen itself may be configured to be provided as VR content.

[0053] Returning to the description of the flowchart in FIG. 3, in step S6, the control unit 190 transitions to the playback screen and starts playback control of the video content selected in step S5. If there is no request to transition to a selection screen based on the information on posture, etc. (No in step S7), for example, playback control of the video content continues until a transition request is made. On the other hand, if there is a request to transition to a selection screen based on the information on posture, etc. (Yes in step S7), in step S8, the control unit 190 executes processing for transitioning from the playback screen to the selection screen and ends. Note that the transition request is made, for example, when a selection area for transitioning to the selection screen (for example, the transition button 707 in FIG. 7) is selected by a user's operation input. Furthermore, the transition request may not be a transition to a selection screen including multiple video contents as displayed in step S1, but may be a request for transition to, for example, a screen displaying detailed information about the video content being played.

[0054] Steps S6 to S8 will now be described in detail with reference to Figs. 6 and 7. Fig. 6 is a diagram showing an XZ cross section of a field of view area in virtual space 601 when watching a video, as viewed from the Y direction. Fig. 7 is a schematic diagram showing an example of a field of view image 701 of a user when watching a video. In step S6, for example, a process is executed to transition from virtual space 401 shown in Fig. 4 to virtual space 601 shown in Fig. 6. Also, in step S6, field of view image 701 as shown in Fig. 7 is provided to the user.

[0055] 6, virtual camera 402 is placed in virtual space 601. Virtual camera 402 is as described above. Furthermore, an image of the selected video content is displayed as panoramic image 602 along the outer edge of virtual space 601. In other words, the selected video content is played back as a 360-degree video.

[0056] 7 is an image captured by virtual camera 402 of virtual space 601 shown in FIG. 5, and is an image within the field of view area indicated by the dotted line. In the example of FIG. 7, field of view image 701 displays an image of the selected video content played back. The image displayed in field of view image 701 is a part of panoramic image 602.

[0057] Also displayed in the field of view image 701 are a pointer 506 and a playback UI 702 related to video playback. The playback UI 702 is displayed when a predetermined operation input is received, such as when the user moves their head at a predetermined speed or more or tilts their head at a predetermined angle or more. Also, a selection area for displaying the playback UI 702 may be displayed in the field of view image 701, and the playback UI 702 may be displayed when the pointer 506 remains in the selection area for a predetermined period of time or more.

[0058] The playback UI 702 includes, as selection areas corresponding to various operations, a play / stop button 703, a rewind button 704, a fast-forward button 705, a skip button 706 for starting playback of the next video content, a transition button 707 to a selection screen, and a seek bar 708. When the time that the pointer 506 remains over each selection area exceeds a predetermined value, various operations corresponding to that selection area are executed.

[0059] For example, when skip button 706 is selected, the currently playing video content is terminated without transitioning to a new virtual space, and the next video content registered in the watch list is started. On the other hand, when transition button 707 is selected, for example, a process of transitioning from virtual space 601 to virtual space 401 is executed, and a selection screen is displayed within field of view image 501 as shown in FIG.

[0060] [Second embodiment: configuration example] Next, with reference to FIG. 8 , an example of a combination of the second device 20 and the portable terminal 100 according to the second embodiment will be described. The second embodiment is an example in which the second device 20 and the portable terminal 100 are used through a communication connection. The communication connection may be a wired connection or a wireless connection (e.g., Bluetooth (registered trademark), WiFi, etc.). In the second embodiment, output from the portable terminal 10 is output to the second device 20 or another medium via the second device 20, and a selection screen is displayed. Note that the content described in the first embodiment can be incorporated into the second embodiment to the extent that it does not cause a contradiction. Similarly, the content described below regarding the second embodiment can also be incorporated into the first embodiment to the extent that it does not cause a contradiction. Specifically, the descriptions regarding the first device 10 and the portable terminal 100 in the first embodiment can be incorporated into the second device 20 and the portable terminal 100 in the second embodiment to the extent that it does not cause a contradiction. Similarly, the descriptions regarding the second device 20 and the portable terminal 100 in the second embodiment can be applied to the first device 10 and the portable terminal 100 in the first embodiment, respectively, to the extent that no contradiction occurs.

[0061] There are no particular limitations on the second device 20, as long as it can be worn on a predetermined part of the user's head, can be communicatively connected to the mobile terminal 100, and has a display unit. Here, the "predetermined part of the head" refers to, for example, one or more parts of the face or the back of the head, such as the eyes, nose, or ears. For example, VR glasses, smart glasses, AR glasses, MR glasses, contact lens displays, or the like, which have the above functions, can be used as the second device 20.

[0062] 9 is a block diagram showing an example of the functional configuration of the second device 20 and the portable terminal 100 according to the second embodiment. A description of the portable terminal 100 will be omitted. The second device 20 includes a wireless communication IF 220, a display unit 230, an input / output IF 240, a storage unit 250, an audio processing unit 260, a microphone 262, a speaker 261, a sensor unit 270, and a control unit 290.

[0063] The wireless communication IF 220 performs modulation / demodulation processing and the like for transmitting and receiving signals via an antenna (not shown) and the like so that the second device 20 can communicate with other communication devices such as the mobile terminal 100. The wireless communication IF 220 is a communication module for wireless communication that includes a tuner, a high-frequency circuit, and the like, and performs modulation / demodulation and frequency conversion of wireless signals transmitted and received by the second device 20, and provides the received signals to the control unit 290.

[0064] Display unit 230 accepts input from a user and outputs information to the user on display 232. For example, a menu screen, a virtual space image, etc. can be displayed on display 232. Display 232 is realized by, for example, an LCD, an organic EL, or other display device.

[0065] The input / output IF 240 functions as an interface for receiving information input to the second device 20 and outputting information to the outside of the second device 20.

[0066] The storage unit 250 is configured by a flash memory, a RAM, etc. The storage unit 250 can store various data received by the second device 20. The storage unit 250 stores a program 251 that can provide a VR experience to the user.

[0067] The audio processing unit 260 modulates and demodulates audio signals. The audio processing unit 260 modulates a signal provided from the microphone 262 and provides the modulated signal to the control unit 290. The audio processing unit 260 also provides the audio signal to the speaker 261. The audio processing unit 260 is realized, for example, by a processor for audio processing. The microphone 262 functions as an audio input unit that receives an input of an audio signal and outputs it to the control unit 290. The speaker 261 functions as an audio output unit that outputs the audio signal to the outside of the second device 20.

[0068] The sensor unit 270 acquires information about the movement or posture of the user or the second device 20 (hereinafter also referred to as "posture information, etc."). The sensor unit 270 includes, for example, a camera and motion sensors such as a gyro sensor and an acceleration sensor. In this embodiment, the user wears the second device 20, which is communicatively connected to the portable terminal 100, on his / her head. In this state, if information about the movement or posture of the second device 20 is acquired by the motion sensor, this also means that information about the movement or posture of the user's head is acquired. Furthermore, the sensor unit 270 may acquire information about the movement of the user, such as the user's line of sight, by a camera when the second device 20 incorporating the portable terminal 100 is worn on the user's head.

[0069] The control unit 290 controls the operation of the second device 20 by reading and executing a program stored in the storage unit 250. The control unit 290 is realized by, for example, an application processor.

[0070] The control unit 290 loads and executes the program 251 to thereby perform the functions of a transmission / reception unit 291, an operation reception unit 292, and a display control unit 293.

[0071] The transmitting / receiving unit 291 receives various types of information from the portable terminal 100 via, for example, a wireless network, and transmits various types of information to the portable terminal 100. The transmitting / receiving unit 291 receives, for example, information resources for display on the display 232 from the portable terminal 100. The transmitting / receiving unit 291 also transmits information such as posture acquired by the sensor unit 270 to the portable terminal 100.

[0072] The operation receiving unit 292 receives, as an operation input, the information on the attitude etc. acquired by the sensor unit 270. The information on the attitude etc. is also transmitted to the mobile terminal 100, and can be received as an operation input for the mobile terminal 100.

[0073] The operation receiving unit 292 receives, for example, information on posture, etc., as a pointer movement operation. Furthermore, for example, when the pointer is over a predetermined selection area for a time period equal to or longer than a predetermined value, the operation receiving unit 292 receives the selection of one piece of content corresponding to the selection area. The operation receiving unit 292 may be configured to receive information on posture, etc., as an operation input, triggered, for example, by the start of VR mode.

[0074] Furthermore, the operation reception unit 292 may include a touch panel and may be configured to receive various touch operations on the touch panel as operation inputs. The touch panel is disposed, for example, on the side of the second device 20. Furthermore, the operation reception unit 292 may be configured to detect the user's hand movements or the like using the sensor unit 270 and receive the hand movements or the like as operation inputs.

[0075] The display control unit 293 controls the content to be displayed on the display 232. The display control unit 293 controls the content to be displayed on the display 232 based on, for example, information transmitted from the mobile terminal 100, and is capable of displaying in VR mode. The VR mode is as described in the first embodiment.

[0076] When the display control unit 293 displays an image of the video content played back on the display 232, the playback control of the video content is executed, for example, on the mobile terminal 100 side. The display control unit 293 preferably has a so-called mirroring function that duplicates the image to be displayed on the display 232 based on image information generated in the mobile terminal 100. Furthermore, the display control unit 293 preferably has the same function as the display control unit 193.

[0077] [Second embodiment: Operation processing] Next, a case of shopping on an EC (Electronic Commerce) site will be described as a specific example of the operation processing according to the second embodiment with reference to Fig. 10 and Fig. 11. Fig. 10 is a flowchart showing an example of the processing according to the second embodiment.

[0078] 10 is executed in a state where the VR mode is selected and the user wears the second device 20, which is communicatively connected to the portable terminal 100, on his / her head. That is, the processes from step S11 to step S19 can be executed while the user is wearing the second device 20, without removing the second device 20.

[0079] First, in step S11, the control unit 290 displays a selection screen including a plurality of products (contents) and a search box on the display 232. The selection screen includes thumbnails and selection areas corresponding to each product. A pointer that moves based on information such as posture is also displayed on the selection screen.

[0080] The display of the selection screen in step S11 is executed based on the information resource transmitted from the mobile terminal 100. Furthermore, in each of the subsequent processes, the display on the display 232 is executed based on the information resource transmitted from the mobile terminal 100, even if not specifically mentioned.

[0081] In step S12, the control unit 290 displays a keyboard image in response to receiving a predetermined operation input. The predetermined operation input in step S12 is not particularly limited, and may be, for example, a touch operation on a touch panel provided on the second device 20 or the touch panel 131 of the portable terminal 100, or the time that the pointer remains on the search window exceeds a predetermined value. The predetermined operation input in step S12 may also be based on the movement of the user's hand detected by the sensor unit 270 (for example, a camera).

[0082] When control unit 290 receives an operation input, the information is transmitted to portable terminal 100, and various processes are executed in portable terminal 100. Then, based on the results of the processing in portable terminal 100, information resources for display on display 232 are generated, and the information resources are transmitted to control unit 290, whereby various display controls are performed by control unit 290. The same applies to each of the subsequent processes.

[0083] In step S13, control unit 290 acquires information such as posture from sensor unit 270 and accepts it as operation input. Next, in step S14, control unit 290 moves the pointer based on the information such as posture. When the pointer moves onto a selection area (each key) on the keyboard, time counting begins, and if the time the pointer has been over the selection area reaches or exceeds a predetermined value (Yes in step S15), in step S16 control unit 190 accepts the characters corresponding to the selection area as a search word. On the other hand, if the time the pointer has been over the selection area is less than the predetermined value and the pointer has moved out of the selection area (No in step S15), the counted time is reset, and the process returns to step S13.

[0084] Steps S11 to S16 will now be described in detail with reference to Fig. 11. Fig. 11 is a schematic diagram showing an example of a user's field of view image 1101 during a search. Note that the field of view image 1101 in Fig. 11 is an image captured of the field of view area of ​​a virtual camera in a space such as the virtual space 601 shown in Fig. 6, and is an image in which the selection screen is displayed as a panoramic image.

[0085] In the example of FIG. 11, a field of view image 1101 displays a pointer 506, thumbnails 1102a to 1102f and selection areas 1103a to 1103f corresponding to the contents A1 to F1, respectively, a search box 1104, and a keyboard image 1105.

[0086] Contents A1 to F1 are products sold on an EC site. Thumbnails 1102a to 1102f are thumbnails of the respective products, and selection areas 1103a to 1103f are selection areas for purchasing the respective products. Note that selection areas 1103a to 1103f may also be selection areas for transitioning to product detail pages.

[0087] The search box 1104 is an area for entering a search word to search within the site or other websites. The keyboard image is displayed, for example, when the time that the pointer 506 remains over the search box 1104 exceeds a predetermined value. The keyboard image 1105 is a selection area for entering characters into the search box 1104. When the time that the pointer 506 remains over each key included in the keyboard image 1105 exceeds a predetermined value, the character corresponding to that key can be entered as a search word. Note that the system may be configured to automatically display one or more search word candidates predicted from the user's past search history and one or more of the characters currently entered. In this case, a word selected from the search word candidates is entered as the search word.

[0088] Returning to the description of the flowchart in Fig. 10, in step S17, the control unit 290 displays the search results in response to an operation input for executing a search. Step S17 can be executed, for example, in response to a search execution button or the like provided as a selection area being selected after the input of the character string to be searched for has been completed by repeatedly inputting a search word. Note that the display of the search results in step S17 can involve a transition of the display screen.

[0089] If a transition request to transition the display screen based on the posture information, etc. is not made (No in step S18), the display screen currently being displayed (for example, a screen displaying search results) continues to be displayed. On the other hand, if a transition request to transition the display screen based on the posture information, etc. is made (Yes in step S18), the display screen is transitioned in accordance with the transition request and the display content is changed in step S19. For example, if an operation input is made to select content displayed on the search result screen, a transition to a purchase screen for that content or a transition to a screen displaying detailed information about that content is made.

[0090] The above-described embodiments are merely examples for facilitating understanding of the present invention, and are not intended to limit the present invention. The present invention can be modified and improved without departing from the spirit thereof, and it goes without saying that the present invention includes equivalents thereof.

[0091] For example, in the above embodiments, the pointer is moved based on posture information and the content selection is accepted by the pointer, but the user's hand movement or the like may be detected and the content selection may be accepted based on the hand movement or the like. Also, the content selection may be accepted based on an operation on a touch panel or physical button provided on each of the first device 10, the second device 20, and the mobile terminal 100. Also, the content selection may be accepted by a method that appropriately combines the above-mentioned selection methods.

[0092] Although the above embodiments have been described with reference to examples of playing video content and shopping on an e-commerce website, the present disclosure can also be applied to other cases, such as web browsing, online credit card registration or membership registration, various online applications, game playing, and reading e-books, all of which are provided to users as VR experiences.

[0093] [Additional Notes] The contents of the present disclosure are listed as follows:

[0094] (Item 1) A program executed in a portable terminal that can be incorporated into a first device worn on a user's head, In a state where the user wears the first device incorporating the portable terminal, the program causes a processor of the portable terminal to displaying a selection screen including at least one selectable content; acquiring information about the movement or attitude of the user, the first device, or the mobile terminal; a step of controlling the display of the selection screen based on the information about the movement or posture, or accepting a selection of one content item included in the selection screen and controlling processing of the one content item; A program that executes. This makes it possible to improve usability when providing a user with a VR experience or the like by combining a specific device with a portable terminal. Specifically, the user can select content, process the content (e.g., play it), or change the display content on the selection screen simply by changing their posture while wearing the device. This eliminates the need to put on and take off the device on their head, or to remove and reattach the portable terminal from the device during these operations, as was previously the case, and provides a comfortable VR experience for the user.

[0095] (Item 2) A program executed on a portable terminal capable of communicating with a second device worn on a predetermined part of a user's head, In a state in which the portable terminal and the second device are communicatively connected, the program causes a processor of the portable terminal to displaying a selection screen including at least one selectable content via the second device; acquiring information about a movement or attitude of the user or the second device; Controlling the display of the selection screen on the second device based on the information about the movement or posture, or accepting a selection of one piece of content included in the selection screen on the second device and controlling processing of the one piece of content; A program that executes. This makes it possible to improve usability when providing a user with a VR experience or the like by combining a specific device with a portable terminal. Specifically, users can select content, process (for example, play) content, or change the display content on the selection screen simply by changing their posture while wearing the device. This eliminates the need to put on and take off the device on their head, or to detach and reattach the portable terminal from the device, as was previously required, and provides users with a comfortable R experience.

[0096] (Item 3) the selection screen includes an image area or a text area corresponding to the at least one content, and a selection area; When a time during which a pointer that moves based on the information about the movement or posture is on the selection area is equal to or longer than a predetermined value, selection of one content item corresponding to the selection area is accepted. Item 1 or Item 2. The program according to item 1 or 2. This makes it easy to perform selection operations even while wearing the device by displaying a pointer. Also, even if the user moves their posture or other factors to change the field of view and the pointer unintentionally moves onto the selection area, the selection is not accepted unless the pointer has been on the selection area for a predetermined period of time or more, which prevents the situation in which a selection unintended by the user is accepted.

[0097] (Item 4) the display control includes one or more of scrolling the selection screen, page transition from the selection screen, enlarging or reducing an image included in the selection screen, and searching and displaying search results; The control of the process includes one or more of playing, rewinding, fast-forwarding, and stopping the one content. Item 3. The program according to any one of items 1 to 3. The present disclosure makes it possible to provide the user with a comfortable VR experience, etc., even when performing the above-mentioned display control and playback control, without the need to remove and reattach the device from the head.

[0098] (Item 5) the step of controlling the processing includes controlling playback of the content; the playback control includes, when accepting selection of one content included on the selection screen based on the information about the movement or posture, playing the one content on a playback screen different from the selection screen. Item 4. The program according to any one of items 1 to 4. This allows a series of operations, such as selecting content on the selection screen and playing content on the playback screen, to be performed while wearing the device, further improving usability. In particular, in the past, a user had to select content on the selection screen, attach a portable terminal to the device, and then wear the device on their head to watch a video. Therefore, the above configuration of the present disclosure can provide a user with a VR experience, etc. that is incomparably more comfortable than in the past.

[0099] (Item 6) the step of controlling the processing includes controlling playback of the content; In a state where the user wears the first device incorporating the portable terminal or in a state where the portable terminal and the second device are communicatively connected, the program further causes the processor of the portable terminal to When the one content is being played back on a playback screen different from the selection screen, executing a step of transitioning from the playback screen to the selection screen based on the information about the movement or posture. 6. The program according to any one of items 1 to 5. This allows the user to return from playing content on the playback screen to the selection screen while still wearing the device, further improving usability. In particular, in the past, in order to transition from a state in which a device was worn on the head and watching a video to a selection screen, the user had to remove the device and perform a tap operation on the portable terminal, so the above configuration of the present disclosure can provide the user with a VR experience, etc. that is incomparably more comfortable than before.

[0100] (Item 7) An information processing method executed in a portable terminal that can be incorporated into a first device worn on a user's head, In a state where the user wears the first device incorporating the portable terminal, the information processing method includes causing a processor of the portable terminal to: displaying a selection screen including at least one selectable content; obtaining information about the movement or attitude of the user, the first device, or the mobile terminal; a step of controlling the display of the selection screen based on the information about the movement or posture, or accepting a selection of one content item included in the selection screen and controlling processing of the one content item; An information processing method, including causing a computer to execute the above. This makes it possible to improve usability when providing a user with a VR experience or the like by combining a specific device with a portable terminal.

[0101] (Item 8) An information processing method executed in a portable terminal capable of communicating with a second device worn on a predetermined part of a user's head, In a state in which the portable terminal and the second device are communicatively connected, the information processing method includes causing a processor of the portable terminal to: displaying a selection screen including at least one selectable content via the second device; acquiring information about a movement or attitude of the user or the second device; Controlling the display of the selection screen on the second device based on the information about the movement or posture, or accepting a selection of one piece of content included in the selection screen on the second device and controlling processing of the one piece of content; An information processing method, including causing a computer to execute the above. This makes it possible to improve usability when providing a user with a VR experience or the like by combining a specific device with a portable terminal.

[0102] (Item 9) A portable terminal that can be incorporated into a first device worn on a user's head, In a state where the user is wearing the first device incorporating the portable terminal, a processor of the portable terminal Displaying a selection screen including at least one selectable content; Obtaining information about a movement or attitude of the user, the first device, or the mobile terminal; Controlling the display of the selection screen based on the information about the movement or posture, or accepting the selection of one content item included in the selection screen and controlling processing of the one content item. Portable device. This makes it possible to improve usability when providing a user with a VR experience or the like by combining a specific device with a portable terminal.

[0103] (Item 10) A portable terminal capable of communicating with a second device worn on a predetermined part of a user's head, In a state in which the portable terminal and the second device are communicatively connected, the processor of the portable terminal displaying a selection screen including a plurality of selectable contents via the second device; obtaining information regarding a movement or attitude of the user or the second device; Controlling the display of the selection screen on the second device based on the information about the movement or posture, or accepting a selection of one piece of content included on the selection screen on the second device and controlling processing of the one piece of content. Portable device. This makes it possible to improve usability when providing a user with a VR experience or the like by combining a specific device with a portable terminal. [Explanation of symbols]

[0104] 10: First device 20: Secondary device 100: Portable terminal 150: Memory section (of a portable terminal) 190: (Mobile terminal) control unit 250: Storage section (of the second device) 290: Control unit (of the second device)

Claims

1. A program executed in a portable terminal that can be incorporated into a first device worn on a user's head, In a state where the user wears the first device incorporating the portable terminal, a computer of the portable terminal is a display means for displaying a selection screen including at least one selectable content; an acquisition means for acquiring information regarding a movement or an attitude of the user, the first device, or the mobile terminal; and a control means for controlling the display of the selection screen based on the information about the movement or posture, or for accepting a selection of one piece of content included in the selection screen and controlling processing of the one piece of content; It functions as the control means controls playback of the content; the playback control includes, when accepting selection of one piece of content included on the selection screen based on the information about the movement or posture, playing the one piece of content on a playback screen different from the selection screen; the control means displays a selection area for playing the one piece of content on the playback screen when the user's movement is at or above a predetermined speed or when the user's posture is tilted at or above a predetermined angle; the selection screen includes an image area or a text area corresponding to the at least one content, and a selection area; When a time during which a pointer that moves based on the information about the movement or posture is on the selection area is equal to or longer than a predetermined value, selection of one piece of content corresponding to the selection area is accepted; The control means sets different values ​​as the predetermined value depending on the display location or type of the selection area.

2. A program executed on a portable terminal capable of communicating with a second device worn on a predetermined part of a user's head, In a state where the portable terminal and the second device are communicatively connected, a computer of the portable terminal is a display control means for displaying a selection screen including at least one selectable content via the second device; acquiring means for acquiring information regarding a movement or an attitude of the user or the second device; and a control means for controlling the display of the selection screen on the second device based on information about the movement or posture, or for accepting a selection of one piece of content included in the selection screen on the second device and controlling processing of the one piece of content; It functions as the control means controls playback of the content; the playback control includes, when accepting selection of one piece of content included on the selection screen based on the information about the movement or posture, playing the one piece of content on a playback screen different from the selection screen; the control means displays a selection area for playing the one piece of content on the playback screen when the user's movement is at or above a predetermined speed or when the user's posture is tilted at or above a predetermined angle; the selection screen includes an image area or a text area corresponding to the at least one content, and a selection area; When a time during which a pointer that moves based on the information about the movement or posture is on the selection area is equal to or longer than a predetermined value, selection of one piece of content corresponding to the selection area is accepted; The control means sets different values ​​as the predetermined value depending on the display location or type of the selection area.

3. the display control includes one or more of scrolling the selection screen, page transition from the selection screen, enlarging or reducing an image included in the selection screen, and searching and displaying search results; The processing of the one content includes one or more of playing, rewinding, fast-forwarding, and stopping the one content. The program according to claim 1 or 2.

4. In a state where the user is wearing the first device incorporating the portable terminal or in a state where the portable terminal and the second device are communicatively connected, the computer further When the one content is being played back on a playback screen different from the selection screen, the transition means is configured to transition from the playback screen to the selection screen based on the information about the movement or posture. The program according to any one of claims 1 to 3.

5. An information processing method executed in a portable terminal that can be incorporated into a first device worn on a user's head, In a state where the user wears the first device incorporating the portable terminal, a computer of the portable terminal is a display means for displaying a selection screen including at least one selectable content; an acquisition means for acquiring information regarding a movement or an attitude of the user, the first device, or the mobile terminal; and a control means for controlling the display of the selection screen based on the information about the movement or posture, or for accepting a selection of one piece of content included in the selection screen and controlling processing of the one piece of content; including functioning as the control means controls playback of the content; the playback control includes, when accepting selection of one piece of content included on the selection screen based on the information about the movement or posture, playing the one piece of content on a playback screen different from the selection screen; the control means displays a selection area for playing the one piece of content on the playback screen when the user's movement is at or above a predetermined speed or when the user's posture is tilted at or above a predetermined angle; the selection screen includes an image area or a text area corresponding to the at least one content, and a selection area; When a time during which a pointer that moves based on the information about the movement or posture is on the selection area is equal to or longer than a predetermined value, selection of one piece of content corresponding to the selection area is accepted; The information processing method, wherein the control means sets different values ​​as the predetermined value depending on the display location or type of the selection area.

6. An information processing method executed in a portable terminal capable of communicating with a second device worn on a predetermined part of a user's head, In a state where the portable terminal and the second device are communicatively connected, a computer of the portable terminal is a display control means for displaying a selection screen including at least one selectable content via the second device; an acquisition means for acquiring information regarding a movement or an attitude of the user or the second device; a control means for controlling the display of the selection screen on the second device based on information about the movement or posture, or for accepting a selection of one piece of content included in the selection screen on the second device and controlling processing of the one piece of content; including functioning as the control means controls playback of the content; the playback control includes, when accepting selection of one piece of content included on the selection screen based on the information about the movement or posture, playing the one piece of content on a playback screen different from the selection screen; the control means displays a selection area for playing the one piece of content on the playback screen when the user's movement is at or above a predetermined speed or when the user's posture is tilted at or above a predetermined angle; the selection screen includes an image area or a text area corresponding to the at least one content, and a selection area; When a time during which a pointer that moves based on the information about the movement or posture is on the selection area is equal to or longer than a predetermined value, selection of one piece of content corresponding to the selection area is accepted; The information processing method, wherein the control means sets different values ​​as the predetermined value depending on the display location or type of the selection area.

Images