VR video control method

Abstract

The invention relates to the technical field of virtual reality, and provides a VR video control method. The method is applied to the mobile terminal and comprises the steps that in response to operation of a user, a target control mode is determined, and the target control mode comprises a gyroscope control mode, a rocker control mode or a touch control mode; acquiring a control parameter corresponding to the target control mode; and performing interaction control on a VR video on a display screen of the display terminal based on the control parameter. According to the method provided by the invention, the control parameter corresponding to the target control mode is acquired through the mobile terminal, so that interaction control of the user on the VR video is realized based on the control parameter.

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, and in particular to a VR video control method. Background Art

[0002] At present, virtual reality (VR) technology is becoming more and more mature, and the popularization of 5G has led to the vigorous development of VR applications. VR technology uses computer simulation systems to simulate the external environment. The main simulation objects include environment, skills, sensor equipment and perception, etc., to provide users with multi-information, three-dimensional dynamic and interactive simulation experience.

[0003] Smart TVs developed based on Android are widely used in VR panoramic content due to their low cost, large screen, and ability to reduce the discomfort of users wearing VR headsets for a long time. However, smart TVs currently do not have interactive devices such as handles and touch screens. The TV remote controls equipped with smart TVs only support up, down, left, and right operations. It is impossible to use existing TV remote controls to interactively control VR videos. Summary of the invention

[0004] The embodiment of the present application provides a VR video control method to solve the technical problem in the prior art that the existing remote controller cannot realize interactive control of VR video.

[0005] In a first aspect, an embodiment of the present application provides a VR video control method, which is applied to a mobile terminal, and the method includes:

[0006] In response to the user's operation, determining a target control mode, wherein the target control mode includes a gyroscope control mode, a joystick control mode, or a touch control mode;

[0007] Obtaining control parameters corresponding to the target control mode;

[0008] The VR video on the display screen of the display terminal is interactively controlled based on the control parameters.

[0009] In one embodiment, the obtaining the control parameter corresponding to the target control mode includes:

[0010] Based on the correspondence between the control mode and the control strategy, determining the target control strategy corresponding to the target control mode; the target control strategy is used to characterize the type of parameters to be obtained under the target control mode;

[0011] Based on the parameter type, a control parameter corresponding to the target control mode is acquired.

[0012] In one embodiment, the target control mode is a gyroscope control mode, and the parameter type includes a swing parameter type of a gyroscope in the mobile terminal;

[0013] The acquiring, based on the parameter type, a control parameter corresponding to the target control mode includes:

[0014] When receiving a field of view adjustment instruction input by a user, acquiring a first swing parameter of the gyroscope based on the swing parameter type of the gyroscope;

[0015] The first swing parameter is determined as the control parameter.

[0016] In one embodiment, the target control mode is a joystick control mode, and the parameter type includes a joystick parameter type;

[0017] The acquiring, based on the parameter type, a control parameter corresponding to the target control mode includes:

[0018] Based on the joystick parameter type, obtaining the joystick movement distance and joystick movement direction input by the user through the joystick control; the joystick control is displayed in the interface corresponding to the joystick control mode;

[0019] The joystick moving distance and the joystick moving direction are determined as the control parameters.

[0020] In one embodiment, the target control mode is a touch control mode, and the parameter type includes a touch parameter type;

[0021] The acquiring, based on the parameter type, a control parameter corresponding to the target control mode includes:

[0022] Based on the touch parameter type, acquiring the touch distance and touch direction input by the user through the touch area; the touch area is displayed in the interface corresponding to the touch control mode;

[0023] The touch distance and the touch direction are determined as the control parameters.

[0024] In one embodiment, the obtaining the control parameter corresponding to the target control mode includes:

[0025] Acquire a second swing parameter of the gyroscope in the mobile terminal, and determine the second swing parameter as the control parameter;

[0026] The interactively controlling the VR video on the display screen of the display terminal based on the control parameter includes:

[0027] sending the second swing parameter to the display terminal; the second swing parameter is used to instruct the display terminal to display a virtual ray determined based on the second swing parameter, the virtual ray pointing to the first object to be selected in the VR video;

[0028] Acquire a first selection instruction input by a user for a first target object to be selected;

[0029] The first selection instruction is sent to the display terminal; the first selection instruction is used to instruct the display terminal to select the first target object to be selected.

[0030] In one embodiment, the obtaining the control parameter corresponding to the target control mode includes:

[0031] In the case where the target control mode includes a joystick control mode, obtaining a click instruction of a user for a target direction control, and determining the click instruction as the control parameter; the target direction control is displayed in an interface corresponding to the joystick control mode;

[0032] The interactively controlling the VR video on the display screen of the display terminal based on the control parameter includes:

[0033] Sending the click instruction to the display terminal; the click instruction is used to instruct the display terminal to determine the second object to be selected in the VR video based on the click instruction;

[0034] Acquire a second selection instruction input by a user for a second target object to be selected;

[0035] The second selection instruction is sent to the display terminal; the second selection instruction is used to instruct the display terminal to select the second target object to be selected.

[0036] In one embodiment, the gyroscope control mode, the joystick control mode and the touch control mode are switchable.

[0037] In a second aspect, an embodiment of the present application provides a VR video control method, which is applied to a display terminal, and the method includes:

[0038] Receiving control parameters corresponding to a target control mode sent by a mobile terminal; the target control mode includes a gyroscope control mode, a joystick control mode or a touch control mode;

[0039] Based on the control parameters, the VR video on the display screen of the display terminal is interactively controlled.

[0040] In one embodiment, the interactively controlling the VR video on the display screen of the display terminal based on the control parameter includes:

[0041] Determining a first three-dimensional coordinate of the mobile terminal in a three-dimensional space based on the control parameter;

[0042] Mapping the first three-dimensional coordinates onto a display screen of the display terminal to obtain first two-dimensional coordinates corresponding to the first three-dimensional coordinates;

[0043] The field of view of the VR video on the display screen is controlled based on the first two-dimensional coordinates.

[0044] In one embodiment, the control parameter includes a second swing parameter of a gyroscope acquired by the mobile terminal in the target control mode;

[0045] The interactively controlling the VR video on the display screen of the display terminal based on the control parameter includes:

[0046] displaying a virtual ray based on the second swing parameter;

[0047] Determine the object in the VR video pointed by the virtual ray as the first object to be selected;

[0048] receiving a first selection instruction sent by the mobile terminal;

[0049] In response to the first selection instruction, a first target to-be-selected object corresponding to the first selection instruction is selected.

[0050] In one embodiment, displaying a virtual ray based on the second swing parameter comprises:

[0051] determining a second three-dimensional coordinate of the mobile terminal in the three-dimensional space based on the second swing parameter;

[0052] Mapping the second three-dimensional coordinates onto the display screen of the display terminal to obtain second two-dimensional coordinates corresponding to the second three-dimensional coordinates;

[0053] The virtual ray is displayed at the position of the second two-dimensional coordinate.

[0054] In one embodiment, the control parameter includes a user click instruction for a target direction control obtained by the mobile terminal in the target control mode; the target control mode includes a joystick control mode; the target direction control is displayed in an interface corresponding to the joystick control mode;

[0055] The interactively controlling the VR video on the display screen of the display terminal based on the control parameter includes:

[0056] In response to the click instruction, determining a second to-be-selected object corresponding to the click instruction;

[0057] receiving a second selection instruction sent by the mobile terminal;

[0058] In response to the second selection instruction, a second target to-be-selected object corresponding to the second selection instruction is selected.

[0059] In a third aspect, an embodiment of the present application provides a VR video control device, which is applied to a mobile terminal, and the device includes:

[0060] A first determination module, configured to determine a target control mode in response to a user's operation, wherein the target control mode includes a gyroscope control mode, a joystick control mode, or a touch control mode;

[0061] An acquisition module, used for acquiring control parameters corresponding to the target control mode;

[0062] A control module is used to interactively control the VR video on the display screen of the display terminal based on the control parameters.

[0063] In a fourth aspect, an embodiment of the present application provides a VR video control device, applied to a display terminal, the device comprising:

[0064] A receiving module, used for receiving control parameters corresponding to a target control mode sent by a mobile terminal; the target control mode includes a gyroscope control mode, a joystick control mode or a touch control mode;

[0065] A control module is used to interactively control the VR video on the display screen of the display terminal based on the control parameters.

[0066] In a fifth aspect, an embodiment of the present application provides a mobile terminal, including a gyroscope, a touch screen, a processor, and a memory storing a computer program;

[0067] The gyroscope is used to obtain swing parameters; the touch screen is used to receive touch operations; when the processor executes the computer program, the steps of any one of the VR video control methods described in the first aspect above are implemented.

[0068] In a sixth aspect, an embodiment of the present application provides a display terminal, including a display, a processor, and a memory storing a computer program;

[0069] The display is used to display VR video; when the processor executes the computer program, the steps of any one of the VR video control methods described in the second aspect above are implemented.

[0070] In a seventh aspect, an embodiment of the present application provides a VR video control system, comprising the mobile terminal described in the fifth aspect and the display terminal described in the sixth aspect.

[0071] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the VR video control method described in the first aspect or the second aspect are implemented.

[0072] In a ninth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the VR video control method described in the first aspect or the second aspect.

[0073] The VR video control method provided by the embodiment of the present application, in which the mobile terminal responds to the user's operation and determines the target control mode, and obtains the control parameters corresponding to the target control mode to realize interactive control of the VR video based on the control parameters. Among them, the target control mode includes a gyroscope control mode, a joystick control mode or a touch control mode. In any target control mode, the user can realize interactive control of the VR video based on the obtained control parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0075] Figure 1 This is one of the flow charts of the VR video control method provided in the embodiment of the present application;

[0076] Figure 2 is a functional schematic diagram of a gyroscope control mode provided in an embodiment of the present application;

[0077] Figure 3 is a functional schematic diagram of a joystick control mode provided in an embodiment of the present application;

[0078] Figure 4 is a functional schematic diagram of a touch control mode provided in an embodiment of the present application;

[0079] Figure 5 This is the second flow chart of the VR video control method provided in the embodiment of the present application;

[0080] Figure 6 is a schematic diagram of constructing a virtual camera provided in an embodiment of the present application;

[0081] Figure 7 It is a flowchart of the VR video control method provided in the embodiment of the present application;

[0082] Figure 8is a signaling flow chart of the VR video control method provided in an embodiment of the present application;

[0083] Fig. 9 This is one of the structural schematic diagrams of the VR video control device provided in the embodiment of the present application;

[0084] Fig.10 This is the second structural diagram of the VR video control device provided in the embodiment of the present application;

[0085] Fig.11 is a schematic diagram of the physical structure of a mobile terminal provided in an embodiment of the present application;

[0086] Fig.12 It is a schematic diagram of the physical structure of the display terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0088] It should be noted that the serial numbers assigned to the objects described in this application, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning.

[0089] Virtual reality VR video can be any VR application displayed in the form of video, such as VR roaming, VR games, or VR movies and other VR panoramic applications displayed in the form of video. In addition to using VR dedicated devices such as VR headsets to display VR videos, VR videos can also be displayed using display terminals such as smart TVs.

[0090] Existing smart TVs are usually controlled by remote controls. Traditional remote controls are only equipped with control buttons, which can support planar remote control operations such as up, down, left, and right. The remote control does not have a built-in gyroscope or a touch screen, which makes it impossible to obtain three-dimensional remote control actions input by the user. Therefore, when using a smart TV to display VR videos, the remote control cannot obtain the three-dimensional control parameters input by the user, resulting in poor controllability of VR video control and failure to meet user needs for VR panoramic applications.

[0091] Based on this problem, an embodiment of the present application provides a VR video control method, wherein the VR video control method applied to a mobile terminal can obtain control parameters corresponding to a target control mode through the mobile terminal, and interactively control the VR video on a display screen displayed by a display terminal based on the control parameters to meet user needs.

[0092] Figure 1 This is one of the flow charts of the VR video control method provided in the embodiment of the present application. Figure 1 , an embodiment of the present application provides a VR video control method, applied to a mobile terminal, and the method may include steps 110 to 130.

[0093] Step 110: In response to the user's operation, determine a target control mode, where the target control mode includes a gyroscope control mode, a joystick control mode, or a touch control mode.

[0094] Specifically, the control mode may be various modes suitable for controlling VR videos through a mobile terminal, and the control mode may be, for example, a gyroscope control mode, a joystick control mode, or a touch control mode. A control mode is determined in each control mode, and the control mode is the target control mode. For example, if a user chooses to control a VR video through a gyroscope control mode among the gyroscope control mode, the joystick control mode, or the touch control mode, the gyroscope control mode is the target control mode. The determination of the target control mode may be determined in response to a user selection through a selection control, or may be determined in response to other user operations, and the embodiments of the present application are not limited to this.

[0095] The gyroscope control mode can be a control mode for controlling the field of view of the VR video based on the gyroscope in the mobile terminal. The joystick control mode can be a control mode for controlling the field of view of the VR video based on the control joystick. The touch control mode can be a control mode for controlling the field of view of the VR video based on the touch area.

[0096] Step 120: Obtain control parameters corresponding to the target control mode.

[0097] Specifically, the user inputs the control parameters in the target control mode through the mobile terminal, and the mobile terminal can obtain the control parameters corresponding to the target control mode through its own control parameter acquisition device. The mobile terminal can be, for example, a mobile phone, a tablet computer, a smart watch or other intelligent mobile terminal devices. The control parameter acquisition device can be, for example, a gyroscope or a touch screen.

[0098] For example, when a user uses a mobile phone to input control parameters in gyroscope control mode, the mobile phone can obtain the control parameters input by the user by collecting three-degree-of-freedom coordinate parameters through the gyroscope; or, when a user uses a mobile phone to input control parameters through the touch screen in joystick control mode, the mobile phone can obtain the control parameters input by the user by collecting electrical signal parameters through the touch screen.

[0099] Step 130: Interactively control the VR video on the display screen of the display terminal based on the control parameters.

[0100] Specifically, after obtaining the control parameters, the mobile terminal can send the control parameters to the display terminal. For example, the mobile terminal can establish a communication connection with the display terminal and transmit the control parameters using the communication connection. The display terminal can be, for example, a smart TV, a smart screen, or other electronic device that can display VR videos. The VR video on the display screen of the display terminal is interactively controlled, for example, the field of view of the VR video can be controlled, the game characters in the VR video can be controlled or the options can be selected, the VR video can be switched to a 360-degree viewing angle, and so on. Among them, the field of view can be understood as the visible range of the VR video displayed in the display interface of the display terminal.

[0101] Exemplarily, for example, taking the field of view control as an example, the above control parameters can be used to instruct the display terminal to determine the first three-dimensional coordinates of the mobile terminal in the three-dimensional space, and map the first three-dimensional coordinates to the display screen of the display terminal, and after obtaining the first two-dimensional coordinates corresponding to the first three-dimensional coordinates, control the field of view of the VR video on the display screen based on the first two-dimensional coordinates. The first three-dimensional coordinates can be the coordinates of the mobile terminal in the three-dimensional space of the real world. For example, the mobile terminal obtains data such as the azimuth and pitch angle of the gyroscope, and the display terminal can determine the first three-dimensional coordinates of the mobile terminal in the three-dimensional space based on the acquired data, and can map the first three-dimensional coordinates to the display screen of the display terminal to obtain the first two-dimensional coordinates corresponding to the first three-dimensional coordinates. It can be understood that the display terminal establishes a corresponding relationship between the position of the mobile terminal in the three-dimensional space and the center position of the light source of the virtual reality in the VR video based on the acquired control parameters. Among them, the center position of the light source of the virtual reality can also be understood as the position of the virtual camera in the VR video. The display terminal can control the field of view of the VR video on the display screen based on the first two-dimensional coordinates. For example, the field of view of the VR video can be adjusted or the perspective can be switched.

[0102] Taking the mobile terminal as a mobile phone and the display terminal as a smart TV as an example, the mobile phone responds to the user's control mode selection operation and determines that the target control mode is the gyroscope control mode; the mobile phone obtains the control parameters input by the user in the gyroscope control mode in real time, and sends the obtained control parameters to the smart TV. After the smart TV receives the control parameters sent by the mobile phone, it determines the first three-dimensional coordinates of the mobile phone in the three-dimensional space based on the control parameters, and maps the first three-dimensional coordinates to the display screen of the smart TV to obtain the first two-dimensional coordinates corresponding to the first three-dimensional coordinates, and based on the first two-dimensional coordinates, the field of view of the VR video on the display screen can be controlled, for example, the field of view can be adjusted to meet the user's viewing needs of 360-degree panoramic viewing of VR videos.

[0103] For another example, by operating the control joystick, you can complete functions such as 360-degree viewing angle switching and game character walking control in the VR video of the smart TV terminal, and realize the separate control of selection control commands and viewing angle switching commands.

[0104] For another example, after establishing the connection through command data, the TV end uses a three-dimensional ellipsoid fitting algorithm to start verifying the original instruction set of the mobile phone end, thereby determining the three-dimensional sphere center of the TV end VR video, and according to the ray principle of the camera, the azimuth and pitch angle data of the mobile terminal's gyroscope and the joystick parameters are mapped to the TV end screen, rendered as virtual rays, and respond to the user's corresponding operation events at the ray projection point, thereby realizing the interaction between the mobile terminal and the smart TV end VR video.

[0105] The VR video control method provided by the embodiment of the present application, in which the mobile terminal responds to the user's operation and determines the target control mode, and obtains the control parameters corresponding to the target control mode to realize interactive control of the VR video based on the control parameters. Among them, the target control mode includes a gyroscope control mode, a joystick control mode or a touch control mode. In any target control mode, the user can realize interactive control of the VR video based on the obtained control parameters.

[0106] In order to accurately obtain the control parameters corresponding to the target control mode, the parameter type to be obtained under the target control mode can be obtained based on the target control strategy corresponding to the target control mode, and the control parameters can be accurately obtained based on the parameter type.

[0107] In one embodiment, obtaining the control parameters corresponding to the target control mode can be achieved in the following manner: based on the correspondence between the control mode and the control strategy, determining the target control strategy corresponding to the target control mode; the target control strategy is used to characterize the parameter type to be obtained under the target control mode; based on the parameter type, obtaining the control parameters corresponding to the target control mode.

[0108] Specifically, the control strategy may be a strategy set for the parameters corresponding to the control mode, and the control strategy may be used to characterize the parameter type to be obtained in the control mode. The control strategies corresponding to each control mode may be preset, and based on the correspondence between the control mode and the control strategy, the corresponding target control strategy may be determined after the target control mode is determined. The parameter type may be the type to which the specific parameter value belongs, and the parameter type may include, for example, the swing parameter of the gyroscope, the joystick parameter type, and the touch parameter type.

[0109] For example, the gyroscope control mode corresponds to a gyroscope control strategy, and the gyroscope control strategy indicates that the parameter types to be obtained by the mobile terminal in the gyroscope control mode include the swing parameters of the gyroscope. For another example, the joystick control mode corresponds to a joystick control strategy, and the joystick control strategy indicates that the parameter types to be obtained by the mobile terminal in the joystick control mode include the joystick parameter type. For another example, the touch control mode corresponds to a touch control strategy, and the touch control strategy indicates that the parameter types to be obtained by the mobile terminal in the touch control mode include the touch parameter type.

[0110] In this embodiment, based on the correspondence between the control mode and the control strategy, after determining the target control mode, the target control strategy corresponding to the target control mode can be determined. Based on the parameter type to be obtained represented by the target control strategy, the mobile terminal can accurately obtain the control parameters corresponding to the target control mode, and then based on the control parameters, the controllability of the VR video can be improved.

[0111] Existing display terminals, such as smart TVs, are equipped with remote controls without built-in gyroscopes. When users use smart TVs to watch 360-degree VR videos, they control the field of view of the VR video by operating the up, down, left, and right keys separately. The adjustment accuracy and flexibility are low, that is, the controllability is poor, resulting in a strong sense of fragmentation in user experience. In the embodiment of the present application, the first swing parameter of the gyroscope in the mobile terminal can be obtained by using the mobile terminal, and the first swing parameter can be determined as the control parameter to improve the controllability of controlling the VR video.

[0112] In one embodiment, the target control mode is a gyroscope control mode, and the parameter type includes a swing parameter type of a gyroscope in the mobile terminal; based on the parameter type, the control parameter corresponding to the target control mode is obtained, which can be specifically implemented in the following manner:

[0113] When a field of view adjustment instruction input by a user is received, a first swing parameter of the gyroscope is acquired based on the swing parameter type of the gyroscope; and the first swing parameter is determined as a control parameter.

[0114] Specifically, when the target control mode is the gyroscope control mode, the parameter type may include the swing parameter type of the gyroscope in the mobile terminal, and the corresponding control parameter may be obtained based on the swing parameter type of the gyroscope. The field of view adjustment instruction input by the user may be an instruction for instructing to adjust the field of view of the VR video, for example, the field of view adjustment instruction may be input by long pressing the instruction button.

[0115] The first swing parameter of the gyroscope may be a parameter used to adjust the field of view of the VR video, and the first swing parameter may be acquired by the gyroscope in the mobile terminal. The first swing parameter may be, for example, a three-degree-of-freedom coordinate parameter acquired by the gyroscope, or a parameter such as an azimuth angle, a pitch angle, a roll angle, or an angular velocity acquired when the gyroscope swings. When the target control mode is the gyroscope control mode and a field of view adjustment instruction input by the user is received, the first swing parameter of the gyroscope in the mobile terminal obtained is determined as a control parameter based on the swing parameter type of the gyroscope.

[0116] For example, Figure 2 is a functional schematic diagram of the gyroscope control mode provided in the embodiment of the present application, referring to Figure 2 As shown, the gyroscope control mode can be displayed in the interface corresponding to the mobile terminal. Taking the mobile terminal as a mobile phone and the display terminal as a smart TV as an example, when the user needs to control the field of view of the VR video, when the target control mode is the gyroscope control mode, the user can hold the mobile phone, long press the ok key 31 in the mobile phone interface and move the mobile phone, that is, input the control parameter in the target control mode, wherein the ok key 31 is the command key, and long pressing the ok key 31 is the field of view adjustment command. After the mobile phone obtains the first swing parameter generated by the gyroscope when the user moves the mobile phone, the first swing parameter can be determined as the control parameter, and the control parameter is sent to the smart TV of the display terminal. Based on the control parameter, the smart TV can adjust the viewing angle of the VR video to enable the user to browse the VR video in 360 degrees.

[0117] In this embodiment, the target control mode is the gyroscope control mode, and the parameter type includes the swing parameter type of the gyroscope in the mobile terminal. When the field of view adjustment instruction input by the user is received, the first swing parameter of the gyroscope is obtained, and the first swing parameter of the gyroscope is determined as the control parameter. Based on the control parameter, the field of view of the VR video can be controlled, and the mobile terminal can be used as a virtual reality handle by relying on the built-in gyroscope component of the mobile terminal to achieve more flexible interaction and more precise control of the VR video displayed in the display terminal, thereby improving the controllability of the VR video control.

[0118] Compared to controlling the field of view of the VR video by operating the up, down, left, and right keys separately, controlling the field of view of the VR video by controlling the joystick is more flexible and accurate, which can improve the controllability of the VR video.

[0119] In one embodiment, the target control mode is a joystick control mode, and the parameter type includes a joystick parameter type; based on the parameter type, the control parameter corresponding to the target control mode is obtained, which can be specifically implemented in the following manner:

[0120] Based on the joystick parameter type, the joystick moving distance and joystick moving direction input by the user through the joystick control are obtained; the joystick is displayed in the interface corresponding to the joystick control mode; and the joystick moving distance and joystick moving direction are determined as control parameters.

[0121] Specifically, when the target control mode is the joystick control mode, the parameter type may include the joystick parameter type. Based on the joystick parameter type, the mobile terminal obtains the joystick movement distance and joystick movement direction input by the user through the control joystick, and determines the joystick movement distance and joystick movement direction as the control parameters. For example, in the interface of the joystick control mode displayed on the touch screen of the mobile terminal, a control joystick is displayed, and the user operates the control joystick to move. The mobile terminal can obtain the specific data of the joystick movement distance and joystick movement direction generated when the control joystick moves, and determines the specific data of the joystick movement distance and joystick movement direction as the control parameters.

[0122] The joystick movement distance can be understood as the distance from the initial position to the final position of the control joystick; the joystick movement direction can be understood as the direction from the initial position to the final position of the control joystick. For example, a control joystick is displayed on the touch screen of a mobile terminal. By collecting the changes in the electrical signals of the touch screen when the user operates the control joystick, the joystick movement distance and joystick movement direction can be calculated. The obtained joystick movement distance and joystick movement direction can be determined as control parameters, based on which the field of view of the VR video can be controlled.

[0123] For example, Figure 3 is a functional schematic diagram of the joystick control mode provided in the embodiment of the present application, referring to Figure 3As shown, the joystick control mode can be displayed in the interface corresponding to the mobile terminal. Taking the mobile terminal as a mobile phone and the display terminal as a smart TV as an example, the touch screen of the mobile phone can display the control joystick 32 in the joystick control mode. When the user needs to control the field of view of the VR video, when the target control mode is the joystick control mode, the user can hold the mobile phone, press and hold the finger and drag the control joystick 32 in the mobile phone interface, that is, input the control parameters in the target control mode. After the mobile phone obtains the joystick movement distance and joystick movement direction generated by the user dragging the control joystick, the joystick movement distance and joystick movement direction can be determined as control parameters, and the control parameters are sent to the smart TV of the display terminal. Based on the control parameters, the smart TV can adjust the field of view of the VR video in real time, or switch the perspective to achieve 360-degree browsing of the VR video. Alternatively, based on the control parameters, the smart TV can control the walking of the game characters within the field of view of the VR video in real time to achieve real-time interaction during VR games.

[0124] In this embodiment, the target control mode is the joystick control mode, and the parameter type includes the joystick parameter type; based on the parameter type, the joystick movement distance and the joystick movement direction of the user's operation control joystick input are obtained, and the joystick movement distance and the joystick movement direction are determined as control parameters. Based on the control parameters, the field of view of the VR video can be controlled, and the multi-touch screen of the mobile terminal is relied upon. The mobile terminal is used as a virtual reality handle, and more flexible interaction and more precise control of the VR video displayed in the display terminal are achieved, thereby improving the controllability of the VR video control.

[0125] Compared to controlling the field of view of the VR video by operating the up, down, left, and right keys separately, controlling the field of view of the VR video through the touch area is more flexible and accurate, which can improve the controllability of the VR video.

[0126] In one embodiment, the target control mode is a touch control mode, and the parameter type includes a touch parameter type; based on the parameter type, the control parameter corresponding to the target control mode is obtained, which can be specifically implemented in the following manner:

[0127] Based on the touch parameter type, the touch distance and touch direction input by the user through the touch area are obtained; the touch area is displayed in an interface corresponding to the touch control mode; and the touch distance and touch direction are determined as control parameters.

[0128] Specifically, when the target control mode is a touch control mode, the parameter type may include a touch parameter type. Based on the touch parameter type, the mobile terminal obtains the touch distance and touch direction input by the user through the touch area, and determines the touch distance and touch direction as control parameters. For example, in the interface of the touch control mode displayed on the touch screen of the mobile terminal, a touch area is displayed. The user slides in the touch area with a finger or a touch tool to generate a touch distance and a touch direction. By determining the touch distance and the touch direction as control parameters, the field of view of the VR video can be controlled based on the control parameters. Among them, the touch tool can be an operating tool that causes the touch screen to generate an electrical signal, such as a capacitive pen or a touch screen pen.

[0129] The touch distance can be understood as the distance from the starting position to the ending position of the touch operation, and the touch direction can be understood as the direction from the starting position to the ending position of the touch operation. For example, a touch area is displayed on the touch screen of a mobile terminal. By collecting the changes in the electrical signals of the touch screen during the user's touch operation, the touch distance and touch direction can be calculated. The obtained touch distance and touch direction are determined as control parameters, and the field of view of the VR video can be controlled based on the control parameters.

[0130] For example, Figure 4 is a functional schematic diagram of the touch control mode provided in the embodiment of the present application, referring to Figure 4 As shown, the touch control mode can be displayed in the interface corresponding to the mobile terminal. Taking the mobile terminal as a mobile phone and the display terminal as a smart TV as an example, the touch screen of the mobile phone can display the touch area 33 in the touch control mode. When the user needs to control the field of view of the VR video, when the target control mode is the touch control mode, the user can hold the mobile phone, press the touch area with a long finger and slide, that is, input the control parameters in the target control mode. After the mobile phone obtains the touch distance and touch direction generated when the user's finger slides, the touch distance and touch direction can be determined as control parameters, and the control parameters are sent to the smart TV of the display terminal. Based on the control parameters, the smart TV can adjust the field of view of the VR video in real time, or switch the viewing angle to achieve 360-degree viewing of the VR video.

[0131] In this embodiment, the target control mode is the touch control mode, and the parameter type includes the joystick parameter type. Based on the parameter type, the touch distance and touch direction input by the user are obtained, and the touch distance and the touch direction are determined as control parameters. Based on the control parameters, the field of view of the VR video can be controlled, and a multi-touch screen based on the mobile terminal is realized. The mobile terminal is used as a virtual reality handle, and more flexible interaction and more precise control are performed on the VR video displayed in the display terminal, thereby improving the controllability of the VR video control.

[0132] Alternatively, if Figure 2-Figure 4 As shown, in the interface of the gyroscope control mode, the joystick control mode and the touch control mode, a volume up key 1, a volume down key 2, a return key 4, a home key 5, a lock key 6, a connection control key 7 and a mode switch key 8 may be included.

[0133] Among them, the volume up key 1 can be used to increase the volume of the display terminal; the volume down key 2 can be used to reduce the volume of the display terminal; the return key 4 can be used to control the display terminal VR application system to return to the previous level in the current interface; the home key 5 can be used to control the VR application system to return to the home page, for example, no matter which interface is currently in, the display terminal VR application system will directly terminate the current operation and return to the application home page; the lock key 6 can be used to lock the current perspective or task, and no longer transmit control parameters to the display terminal; the connection control key 7 can be used to control the mobile terminal to open the connection menu, for example, after clicking the connection control key 7, the mobile terminal will display the connection control menu, and operations such as disconnection, reconnection, and connection to other display terminals can be performed. After disconnecting from the current display terminal VR application system, the display terminal VR application system will no longer be able to be controlled. The mode switch key 8 can be used to switch the target control mode. For example, after clicking the mode switch key 8, different control modes can be selected to achieve fast switching of the target control mode in each control mode. Optionally, in the interface of the touch control mode, only the return key 4, the connection control key 7 and the mode switch key 8 can be retained.

[0134] In order to further improve the user's control visibility when experiencing VR videos, the camera's ray principle can be used to determine a virtual ray based on the second swing parameter of the gyroscope in the mobile terminal. The virtual ray is pointed at the first object to be selected, which can improve the control visibility when controlling VR videos and enhance the experience of human-computer interaction.

[0135] In one embodiment, obtaining control parameters corresponding to the target control mode includes: obtaining a second swing parameter of a gyroscope in a mobile terminal, and determining the second swing parameter as a control parameter; interactively controlling a VR video on a display screen of a display terminal based on the control parameter, including: sending the second swing parameter to the display terminal; the second swing parameter is used to instruct the display terminal to display a virtual ray determined based on the second swing parameter, the virtual ray pointing to a first object to be selected in the VR video; obtaining a first selection instruction for the first target object to be selected input by a user; sending the first selection instruction to the display terminal; the first selection instruction is used to instruct the display terminal to select the first target object to be selected.

[0136] Specifically, the second swing parameter can be a swing parameter obtained based on a gyroscope in the mobile terminal and used to instruct the display terminal to display virtual rays. The second swing parameter can be, for example, a three-degree-of-freedom coordinate parameter collected by the gyroscope, or parameters such as azimuth, pitch angle, roll angle or angular velocity obtained when the gyroscope swings. The second swing parameter can be determined as a control parameter.

[0137] The virtual ray may be a ray pointed to the first object to be selected in the VR video and displayed in the display terminal. The virtual ray may be understood as a fictitious auxiliary ray used to assist the user in locating the first object to be selected. The first object to be selected may be any selectable object in the VR video. After the first object to be selected is designated by the virtual ray, the user may more clearly identify the first object to be selected. The first object to be selected may be, for example, an icon or an option in a menu; the first target object to be selected may be the first object to be selected selected by the user, for example, an icon selected by the user.

[0138] Exemplarily, the first selection instruction may be any instruction for instructing the display terminal to determine the first object to be selected currently pointed to by the virtual ray as the first target object to be selected. For example, when the target control mode is the gyroscope control mode, the first selection instruction may be to click the OK key 31; when the target control mode is the joystick control mode, the first selection instruction may be to click the center of the joystick 32; when the target control mode is the touch control mode, the first selection instruction may be to click any position of the touch area 33. In the target control mode, the first selection instruction may also be instructions of other forms, which are not limited in the embodiment of the present invention.

[0139] Taking the mobile terminal as a mobile phone and the display terminal as a smart TV as an example, when the target control mode is the gyroscope control mode, an ok key 31 is displayed on the touch screen of the mobile phone. The user holds the mobile phone and swings the mobile phone, and the gyroscope of the mobile phone can obtain the second swing parameter. The mobile phone sends the second swing parameter to the smart TV in real time, and the smart TV can determine and display the virtual ray after receiving the second swing parameter. In the VR video displayed on the smart TV, the virtual ray points to different icons as the mobile phone swings. When the user clicks the ok key 31, the icon pointed to by the virtual ray is determined as the first target object to be selected, and the icon is the icon selected by the user.

[0140] Taking the mobile terminal as a mobile phone and the display terminal as a smart TV as an example, when the target control mode is the joystick control mode, a control joystick 32 is displayed on the touch screen of the mobile phone. The user holds the mobile phone and swings the mobile phone, and the gyroscope of the mobile phone can obtain the second swing parameter. The mobile phone sends the second swing parameter to the smart TV in real time, and the smart TV can determine and display the virtual ray after receiving the second swing parameter. In the VR video displayed on the smart TV, the virtual ray points to different icons as the mobile phone swings. When the user clicks the center position of the control joystick 32, the icon pointed to by the virtual ray is determined as the first target to be selected, and the icon is the icon selected by the user.

[0141] Taking the mobile terminal as a mobile phone and the display terminal as a smart TV as an example, when the target control mode is the touch control mode, a touch area 33 is displayed in the touch screen of the mobile phone. The user holds the mobile phone and swings the mobile phone, and the gyroscope of the mobile phone can obtain the second swing parameter. The mobile phone sends the second swing parameter to the smart TV in real time, and the smart TV can determine and display the virtual ray after receiving the second swing parameter. In the VR video displayed on the smart TV, the virtual ray points to different icons as the mobile phone swings. When the user clicks anywhere in the touch area 33, the icon pointed to by the virtual ray is determined as the first target object to be selected, and the icon is the icon selected by the user.

[0142] Based on this, the virtual ray determined by the second swing parameter can effectively point to the direction of the field of view within the field of view of the VR video, thereby improving the flexibility and control visibility of the VR video control; at the same time, based on the first selection instruction, menu selection, page switching and other functions can be efficiently executed, further improving the controllability of the VR video.

[0143] In the process of controlling VR video, the selection control command and the view switching command are prone to conflict, and the problem of how to operate the selection control command and the view switching command separately is particularly prominent. In order to realize the operation of the selection control command and the view switching command separately, the target direction control can be set.

[0144] In one embodiment, obtaining control parameters corresponding to a target control mode includes: when the target control mode includes a joystick control mode, obtaining a user's click instruction for a target direction control, and determining the click instruction as a control parameter; the target direction control is displayed in an interface corresponding to the joystick control mode; interactively controlling a VR video on a display screen of a display terminal based on the control parameters includes: sending the click instruction to the display terminal; the click instruction is used to instruct the display terminal to determine a second object to be selected in the VR video based on the click instruction; obtaining a second selection instruction input by the user for the second target object to be selected; sending the second selection instruction to the display terminal; the second selection instruction is used to instruct the display terminal to select the second target object to be selected.

[0145] Specifically, the target direction control may be a control for obtaining a click instruction. For example, in an interface corresponding to the joystick control mode, a target direction control is set around the joystick. By clicking the target direction in the target direction control, a click instruction input by the user may be obtained. The obtained click instruction may be determined as a control parameter. Figure 3 As shown, the target direction controls set around the control stick include triangle controls in four directions: up, down, left, and right. Clicking the triangle control in any direction can input a click command in that direction.

[0146] The second object to be selected may be any selectable object in the VR video. After receiving the click instruction, the second object to be selected currently indicated in the VR video may be switched based on the click instruction. The second object to be selected may be, for example, an icon or an option in a menu; the second target object to be selected may be the second object to be selected selected by the user. The second selection instruction is used to instruct the display terminal to select the second target object to be selected.

[0147] Exemplarily, the second selection instruction may be any instruction for instructing the display terminal to determine the current second object to be selected as the second target object to be selected. For example, when the target control mode includes the joystick control mode, the second selection instruction may be an instruction generated by clicking any position of the joystick 32.

[0148] In this embodiment, the target direction control is used to obtain the click instruction input by the user, and the click instruction is determined as a control parameter, and the click instruction is sent to the display terminal. The display terminal can determine the second object to be selected in the VR video based on the click instruction, that is, to implement the switching operation of each second object to be selected; based on the user's operation of the control joystick, the joystick movement distance and joystick movement direction are determined as control parameters, and the field of view of the VR video can be controlled. Therefore, this embodiment can realize the separate control of the selection control instruction and the perspective switching instruction, solves the problem that the selection control instruction and the perspective switching instruction are prone to conflict during the control process of the VR video, so that the method can be applied to VR videos that need to control the perspective and switch the object to be selected, and improves the applicability of the method.

[0149] Exemplarily, in order to facilitate controlling the field of view of the VR video with different target control modes, the control modes can be switched.

[0150] In one embodiment, the gyroscope control mode, the joystick control mode and the touch control mode are switchable.

[0151] Specifically, when the current target control mode is any one of the gyroscope control mode, the joystick control mode and the touch control mode, the current target control mode can be switched to another one of the gyroscope control mode, the joystick control mode and the touch control mode by switching the control mode. Figure 2-Figure 4 As shown, switching the control mode can be achieved by using the mode switching key 8; for another example, switching the control mode can also be achieved by voice commands or gesture commands.

[0152] For example, when the current target control mode is the gyroscope control mode, the user can switch the target control mode to the joystick control mode or the touch control mode by clicking the mode switch button in the mobile phone interface. When switching, the control mode can be switched according to the preset switching sequence of the gyroscope control mode, the joystick control mode and the touch control mode, or it can be directly switched to the control mode required by the user according to the user's needs. In this embodiment, the gyroscope control mode, the joystick control mode and the touch control mode can be switched, the control mode can be changed quickly, the target control mode can be determined, the applicability of the method of the present invention is improved, and the user experience is improved.

[0153] For example, existing display terminals such as smart TVs rely on remote control commands and cannot be controlled through touch screens. The logic of remote control control cannot implement effective control selection commands such as text input and 3D cursor display. Based on the mobile terminal controlling the field of view of VR video, the touch screen of the mobile terminal can be used to call a text input box or a 3D cursor, which can realize text input or display of a 3D cursor in VR video.

[0154] Figure 5 This is the second flow chart of the VR video control method provided in the embodiment of the present application. Figure 5 , an embodiment of the present application provides a VR video control method, applied to a display terminal, and the method may include step 510 and step 520.

[0155] Step 510: Receive control parameters corresponding to a target control mode sent by a mobile terminal; the target control mode includes a gyroscope control mode, a joystick control mode, or a touch control mode.

[0156] Specifically, referring to the description of steps 110 to 130, the control parameters input by the user in the target control mode can be obtained through the mobile terminal, and the control parameters can be sent to the display terminal through the mobile terminal. After the display terminal receives the control parameters, the field of view of the VR video can be controlled based on the control parameters.

[0157] Step 520: Based on the control parameters, interactively control the VR video on the display screen of the display terminal.

[0158] In one embodiment, based on control parameters, interactive control is performed on a VR video on a display screen of a display terminal, including: determining a first three-dimensional coordinate of a mobile terminal in a three-dimensional space based on the control parameters; mapping the first three-dimensional coordinate to the display screen of the display terminal to obtain a first two-dimensional coordinate corresponding to the first three-dimensional coordinate; and controlling the field of view of the VR video on the display screen based on the first two-dimensional coordinate.

[0159] Specifically, the first three-dimensional coordinates may be the coordinates of the mobile terminal in the three-dimensional space of the real world. For example, the mobile terminal obtains data such as the azimuth and pitch angle of the gyroscope, or obtains the joystick movement distance and joystick movement direction input by controlling the joystick, or obtains control parameters such as the touch distance and touch direction input by the touch area, and the first three-dimensional coordinates of the virtual camera in the three-dimensional space can be determined, where the virtual camera can be understood as the light source center of the VR video. The position of the mobile terminal in the three-dimensional space is mapped to the position of the virtual camera in the VR video.

[0160] Exemplarily, the first two-dimensional coordinate may be a coordinate corresponding to the first three-dimensional coordinate after the first three-dimensional coordinate is mapped to the display screen of the display terminal. For example, based on the first three-dimensional coordinate, a GPU (Graphics Processing Unit) rendering pipeline is used to map the first three-dimensional coordinate of the virtual camera in the three-dimensional space to the first two-dimensional coordinate on the display screen of the display terminal.

[0161] Exemplarily, the first three-dimensional coordinate is mapped to the display screen of the display terminal, and the first two-dimensional coordinate corresponding to the first three-dimensional coordinate can be obtained by using a three-dimensional ellipsoid fitting algorithm to verify the data, and the first two-dimensional coordinate corresponding to the first three-dimensional coordinate can be determined, and the determined first two-dimensional coordinate is the center position of the sphere. When using the three-dimensional ellipsoid fitting algorithm, the standard ellipsoid equation shown below needs to be used:

[0162]

[0163] Wherein, x, y, and z represent the coordinate values ​​of the first three-dimensional coordinate respectively; x , O y , O z Represent the coordinates of the center of the sphere, R x , R y , R z Represent the lengths of the three semi-axes of the sphere respectively. After expanding the standard ellipsoid equation, we can get the following polynomial:

[0164]

[0165] After finishing, we can get:

[0166] y 2 A+z2 B+xC+yD+zE+F=-x 2

[0167] Right now:

[0168]

[0169] Write the above formula as KX=Y. According to the least squares solution, for the ellipsoid fitting data of the gyroscope data set, the form to be estimated is:

[0170] X=(K T K) -1 K T Y

[0171] Where:

[0172]

[0173] The final equation for ellipsoid calibration is:

[0174]

[0175]

[0176]

[0177]

[0178] After verification according to this method, the fitting rate of the result obtained according to this method with the origin in actual virtual reality applications exceeds 99.8%, and the ellipsoid fitting result is accurate, and the position of the center point of the sphere can be determined.

[0179] Specifically, the display terminal receives the control parameters sent by the mobile terminal in real time, and updates the first three-dimensional coordinates of the mobile terminal in the three-dimensional space in real time based on the control parameters, maps the first three-dimensional coordinates to the display screen of the display terminal, and obtains the updated first two-dimensional coordinates. The field of view of the VR video on the display screen can be controlled by the first two-dimensional coordinates.

[0180] Figure 6 is a schematic diagram of virtual camera construction provided in an embodiment of the present application, such as Figure 6 As shown, the first three-dimensional coordinates of the mobile terminal in the three-dimensional space are determined, that is, the center position of the sphere where the virtual camera is located is determined, and the first three-dimensional coordinates are mapped to the display screen of the display terminal to obtain the first two-dimensional coordinates corresponding to the first three-dimensional coordinates. The field of view of the VR video on the display screen can be controlled based on the first two-dimensional coordinates.

[0181] In this embodiment, the display terminal can determine the first three-dimensional coordinates of the mobile terminal based on the received control parameters, and map the first three-dimensional coordinates to the display screen of the display terminal, so as to obtain the first two-dimensional coordinates corresponding to the first three-dimensional coordinates, and control the field of view of the VR video on the display screen based on the first two-dimensional coordinates. In this way, the field of view of the VR video displayed on the display terminal can be controlled based on the mobile terminal, thereby improving the controllability of the VR video.

[0182] In one embodiment, the control parameters include a second swing parameter of a gyroscope obtained by the mobile terminal in a target control mode; based on the control parameters, the VR video on the display screen of the display terminal is interactively controlled, including: displaying a virtual ray based on the second swing parameter; determining an object in the VR video pointed by the virtual ray as a first object to be selected; receiving a first selection instruction sent by the mobile terminal; and in response to the first selection instruction, selecting a first target object to be selected corresponding to the first selection instruction.

[0183] Specifically, the display terminal can receive a second swing parameter sent by the mobile terminal. The second swing parameter can be, for example, a three-degree-of-freedom coordinate parameter collected by a gyroscope, or parameters such as azimuth, pitch angle, roll angle or angular velocity obtained when the gyroscope swings.

[0184] When the target control mode includes a gyroscope control mode, a touch control mode or a joystick control mode, the display terminal may display a virtual ray based on the second swing parameter. The virtual ray may be a ray directed to the first object to be selected in the VR video displayed in the display terminal, and the virtual ray may be understood as a fictitious auxiliary ray used to assist the user in locating the first object to be selected.

[0185] The first object to be selected may be any selectable object in the VR video. After the first object to be selected is designated by the virtual ray, the user can more clearly identify the first object to be selected. The first object to be selected may be, for example, an icon or an option in a menu; the first target object to be selected may be the first object to be selected selected by the user, for example, an icon selected by the user.

[0186] Exemplarily, the first selection instruction may be any instruction for instructing the display terminal to determine the first object to be selected currently pointed to by the virtual ray as the first target object to be selected. For example, when the target control mode includes a gyroscope control mode, the first selection instruction may be a single click on the OK key 31; when the target control mode includes a joystick control mode, the first selection instruction may be a single click on the center of the joystick 32; when the target control mode includes a touch control mode, the first selection instruction may be an instruction generated by single clicking on any position of the touch area 33.

[0187] Based on this, the virtual ray determined by the second swing parameter improves the visibility of the control during VR video control, and based on the second swing parameter, functions such as menu selection and page switching can be realized, thereby improving the controllability of VR video control.

[0188] In one embodiment, displaying a virtual ray based on a second swing parameter includes: determining a second three-dimensional coordinate of a mobile terminal in three-dimensional space based on the second swing parameter; mapping the second three-dimensional coordinate to a display screen of a display terminal to obtain a second two-dimensional coordinate corresponding to the second three-dimensional coordinate; and displaying the virtual ray at the position of the second two-dimensional coordinate.

[0189] Specifically, the second three-dimensional coordinates may be coordinates of the mobile terminal in the three-dimensional space of the real world determined based on the second swing parameter. For example, the mobile terminal obtains data such as the azimuth and pitch angle of the gyroscope, or obtains the joystick movement distance and joystick movement direction input by controlling the joystick, or obtains control parameters such as the touch distance and touch direction input by the touch area, and the second three-dimensional coordinates of the mobile terminal in the three-dimensional space can be determined. Based on the second three-dimensional coordinates mapped to the display screen of the display terminal, the second two-dimensional coordinates corresponding to the second three-dimensional coordinates can be obtained, and the virtual ray is displayed at the position of the second two-dimensional coordinates.

[0190] The virtual ray can be understood as a ray emitted from a virtual camera, which points to the first object to be selected in the VR video. Displaying the virtual ray on the display screen of the display terminal can improve the visibility of the control when controlling the VR video, can assist the user to clearly identify the first object to be selected, avoid the phenomenon of incorrect determination of the first target object to be selected due to the unclear first object to be selected, can more conveniently realize menu selection, page switching and other functions, and can improve the experience of human-computer interaction.

[0191] In one embodiment, the control parameters include a user's click instruction for a target direction control obtained by the mobile terminal in a target control mode; the target control mode includes a joystick control mode; the target direction control is displayed in an interface corresponding to the joystick control mode; based on the control parameters, the VR video on the display screen of the display terminal is interactively controlled, including: responding to the click instruction, determining a second object to be selected corresponding to the click instruction; receiving a second selection instruction sent by the mobile terminal; and responding to the second selection instruction, selecting a second target object to be selected corresponding to the second selection instruction.

[0192] Specifically, the click instruction input by the user can be obtained through the target direction control. After the display terminal receives the click instruction sent by the mobile terminal, it can determine the second to-be-selected object corresponding to the click instruction in response to the click instruction.

[0193] For example, when a user clicks on a right-pointing triangle control in a target direction control, the mobile terminal sends the click instruction to the display terminal in real time. The display terminal receives and responds to the click instruction and determines a second object to be selected to the right of the second object to be selected determined before receiving the click instruction as the current second object to be selected.

[0194] The second selection instruction may be any instruction for instructing the display terminal to determine the current second object to be selected as the second target object to be selected. For example, after receiving the second selection instruction, the display terminal selects the current second object to be selected as the second target object to be selected in response to the second selection instruction.

[0195] In this embodiment, when the target control mode includes a joystick control mode, the display terminal can flexibly select the second target to-be-selected object corresponding to the second selection instruction in response to a click instruction, thereby further improving the controllability of controlling the VR video.

[0196] In the field of smart TV, limited by the limitations of traditional remote controls, VR applications in the industry still use direction keys as the main interaction method, and the interactive experience of VR box panoramic content interaction and text input is poor. With the popularization of technologies such as gyroscopes and accelerometers in mobile terminals, the advantages of mobile terminal hardware can be used to create a new VR video interaction mode, greatly improving the VR panoramic content interaction experience in the field of smart TV.

[0197] By acquiring data from the gyroscope, accelerometer and other devices on the mobile device, the direction or position of the mobile device in the real space is dynamically calculated, and the real-time data is transmitted to the smart TV through a private protocol of WiFi or Bluetooth. A 3D pointer is mapped in the VR three-dimensional space of the smart TV screen, realizing the remote linkage between the mobile device and the 3D pointer on the smart TV screen. Furthermore, the content can be input into the VR video on the smart TV screen through the soft keyboard input function of the mobile device, providing a flexible and accurate VR interactive experience.

[0198] The VR video control method provided in the embodiment of the present application is based on a mobile terminal, and provides three control modes including a gyroscope control mode, a joystick control mode and a touch control mode. It can also realize menu selection, perspective switching and other functions of VR videos such as VR panoramic roaming and VR games in a display terminal or a set-top box through virtual rays.

[0199] The following takes the display terminal as a smart TV as an example. Figure 7 An overall example description is given of the implementation steps of the VR video control method when it is applied to a mobile terminal and a display terminal. Figure 7 is a flowchart of the VR video control method provided in the embodiment of the present application, such as Figure 7As shown, the method may include five steps from S1 to S5, wherein S1 is a step of establishing a connection; S2 is a step of sending an instruction; S3 is a step of receiving an instruction; S4 is a step of parsing an instruction; and S5 is a step of calibrating an instruction.

[0200] Specifically, S1: connect the mobile terminal and the smart TV to the same network and establish a physical channel with WiFi. The mobile terminal sends a connection request to the device address and port of the smart TV by manual selection or code scanning to establish a port number (Socket) connection, and the smart TV implements the Sokect binding verification with the mobile terminal.

[0201] S2: After the mobile terminal and the smart TV are connected, the mobile terminal transmits the gyroscope parameters, joystick parameters, touch parameters, and key control instructions of the mobile terminal to the smart TV in real time. The functional diagrams of the three modes are as follows: Figures 2 to 4 as shown in .

[0202] S3: The smart TV receives the corresponding original parameters and instructions transmitted by the mobile terminal through the Sokect connection in real time and in sequence.

[0203] S4: The smart TV parses the received original parameters and instructions, establishes a data transmission channel through the parameters, completes the connection between the mobile terminal data and the smart TV instruction protocol, and prepares to start the data verification process.

[0204] S5: After establishing the connection through the command data, the smart TV uses the three-dimensional ellipsoid fitting algorithm to start verifying the original instruction set of the mobile terminal, so as to determine the three-dimensional sphere center of the VR video displayed by the smart TV, and according to the ray principle of the camera, the azimuth and pitch angle data of the mobile terminal gyroscope and the joystick parameters are mapped to the display screen of the smart TV, rendered as virtual rays, and respond to the user's corresponding operation events at the ray projection point, so as to realize VR video control and interaction between the mobile terminal and the smart TV.

[0205] In this embodiment, a new interactive control method is proposed to solve the problem that the traditional remote controller cannot be used to control the VR video when the VR video is played in a display terminal such as a smart TV or a set-top box. Based on a mobile terminal with a multi-touch screen and a built-in gyroscope, the mobile terminal controls the display device to display the VR video, thereby improving the controllability when controlling the VR video.

[0206] Figure 8 is a signaling flow chart of the VR video control method provided in the embodiment of the present application, such as Figure 8 As shown, the method includes:

[0207] Step 810: The mobile terminal determines a target control mode in response to a user operation, where the target control mode includes a gyroscope control mode, a joystick control mode, or a touch control mode.

[0208] Step 820: The mobile terminal obtains control parameters corresponding to the target control mode.

[0209] Step 830: The mobile terminal interactively controls the VR video on the display screen of the display terminal based on the control parameters.

[0210] The VR video control method provided in this embodiment has a specific implementation process and technical effects similar to those of the above embodiments. For details, please refer to the detailed introduction in the above embodiments, which will not be repeated here.

[0211] The following is a description of a VR video control device on a mobile terminal side provided in an embodiment of the present application. The VR video control device described below and the VR video control method described above can be referenced to each other.

[0212] Fig. 9 This is one of the structural diagrams of the VR video control device provided in the embodiment of the present application, referring to Fig. 9 As shown, the VR video control device 900 includes:

[0213] A first determining module 910, configured to determine a target control mode in response to a user operation, where the target control mode includes a gyroscope control mode, a joystick control mode, or a touch control mode;

[0214] An acquisition module 920 is used to acquire control parameters corresponding to a target control mode;

[0215] The control module 930 is used to interactively control the VR video on the display screen of the display terminal based on the control parameters.

[0216] In an exemplary embodiment, the acquisition module 920 is specifically configured to:

[0217] Based on the correspondence between the control mode and the control strategy, the target control strategy corresponding to the target control mode is determined; the target control strategy is used to characterize the parameter type to be obtained under the target control mode; based on the parameter type, the control parameters corresponding to the target control mode are obtained.

[0218] In an exemplary embodiment, the target control mode is a gyroscope control mode, and the parameter type includes a swing parameter type of a gyroscope in the mobile terminal; the acquisition module 920 is specifically used to:

[0219] When a field of view adjustment instruction input by a user is received, a first swing parameter of the gyroscope is acquired based on the swing parameter type of the gyroscope; and the first swing parameter is determined as a control parameter.

[0220] In an exemplary embodiment, the target control mode is a joystick control mode, and the parameter type includes a joystick parameter type; the acquisition module 920 is specifically used to:

[0221] Based on the joystick parameter type, the joystick moving distance and joystick moving direction input by the user through the joystick control are obtained; the joystick is displayed in the interface corresponding to the joystick control mode; and the joystick moving distance and joystick moving direction are determined as control parameters.

[0222] In an exemplary embodiment, the target control mode is a touch control mode, and the parameter type includes a touch parameter type; the acquisition module 920 is specifically used to:

[0223] Based on the touch parameter type, the touch distance and touch direction input by the user through the touch area are obtained; the touch area is displayed in an interface corresponding to the touch control mode; and the touch distance and touch direction are determined as control parameters.

[0224] In an exemplary embodiment, the acquisition module 920 is specifically configured to acquire a second swing parameter of a gyroscope in the mobile terminal, and determine the second swing parameter as a control parameter;

[0225] The control module 930 is specifically used to send the second swing parameter to the display terminal; the second swing parameter is used to instruct the display terminal to display a virtual ray determined based on the second swing parameter, and the virtual ray points to the first object to be selected in the VR video;

[0226] The acquisition module 920 is further used to acquire a first selection instruction input by a user for a first target object to be selected;

[0227] The control module 930 is further used to send a first selection instruction to the display terminal; the first selection instruction is used to instruct the display terminal to select a first target object to be selected.

[0228] In an exemplary embodiment, the acquisition module 920 is specifically used to acquire a user's click instruction for a target direction control when the target control mode includes a joystick control mode, and determine the click instruction as a control parameter; the target direction control is displayed in an interface corresponding to the joystick control mode;

[0229] The control module 930 is specifically used to send a click instruction to the display terminal; the click instruction is used to instruct the display terminal to determine the second object to be selected in the VR video based on the click instruction;

[0230] The acquisition module 920 is further used to acquire a second selection instruction input by the user for a second target object to be selected;

[0231] The control module 930 is further used to send a second selection instruction to the display terminal; the second selection instruction is used to instruct the display terminal to select a second target object to be selected.

[0232] In an exemplary embodiment, the gyroscope control mode, the joystick control mode and the touch control mode are switchable.

[0233] The device of this embodiment can be used to execute the method of any embodiment of the VR video control method embodiment on the mobile terminal side. Its specific implementation process and technical effects are similar to those in the embodiment of the VR video control method side. For details, please refer to the detailed introduction in the embodiment of the VR video control method side, which will not be repeated here.

[0234] The following describes a VR video control device on the display terminal side provided in an embodiment of the present application. The VR video control device described below and the VR video control method described above can refer to each other.

[0235] Fig.10 This is a second structural diagram of the VR video control device provided in the embodiment of the present application, referring to Fig.10 As shown, the VR video control device 1000 includes:

[0236] The receiving module 1010 is used to receive control parameters corresponding to the target control mode sent by the mobile terminal; the target control mode includes a gyroscope control mode, a joystick control mode or a touch control mode;

[0237] The control module 1020 is used to interactively control the VR video on the display screen of the display terminal based on the control parameters.

[0238] In an exemplary embodiment, the control module 1020 is specifically configured to:

[0239] Based on the control parameters, the first three-dimensional coordinates of the mobile terminal in the three-dimensional space are determined; the first three-dimensional coordinates are mapped to the display screen of the display terminal to obtain the first two-dimensional coordinates corresponding to the first three-dimensional coordinates; and the field of view of the VR video on the display screen is controlled based on the first two-dimensional coordinates.

[0240] In an exemplary embodiment, the control parameter includes a second swing parameter of the gyroscope obtained by the mobile terminal in the target control mode; the control module 1020 is specifically used to:

[0241] Display a virtual ray based on a second swing parameter; determine the object in the VR video pointed by the virtual ray as a first object to be selected; receive a first selection instruction sent by the mobile terminal; and in response to the first selection instruction, select a first target object to be selected corresponding to the first selection instruction.

[0242] In an exemplary embodiment, the control module 1020 is specifically configured to:

[0243] Determine the second three-dimensional coordinates of the mobile terminal in the three-dimensional space based on the second swing parameter; map the second three-dimensional coordinates to the display screen of the display terminal to obtain the second two-dimensional coordinates corresponding to the second three-dimensional coordinates; and display the virtual ray at the position of the second two-dimensional coordinates.

[0244] In an exemplary embodiment, the control parameter includes a user click instruction for a target direction control obtained by the mobile terminal in a target control mode; the target control mode includes a joystick control mode; the target direction control is displayed in an interface corresponding to the joystick control mode;

[0245] The control module 1020 is specifically used for:

[0246] In response to the click instruction, determine a second to-be-selected object corresponding to the click instruction; receive a second selection instruction sent by the mobile terminal; in response to the second selection instruction, select a second target to-be-selected object corresponding to the second selection instruction.

[0247] The device of this embodiment can be used to execute the method of any embodiment of the VR video control method embodiment on the display terminal side. Its specific implementation process and technical effects are similar to those in the embodiment on the VR video control method side. For details, please refer to the detailed introduction in the embodiment on the VR video control method side, which will not be repeated here.

[0248] Fig.11 is a schematic diagram of the physical structure of a mobile terminal provided in an embodiment of the present application, such as Fig.11 As shown, the mobile terminal may include: a processor 1110, a communication interface 1120, a memory 1130, a bus interface 1140, a gyroscope 1150, and a touch screen 1160. Among them, the processor 1110, the communication interface 1120, the memory 1130, the gyroscope 1150, and the touch screen 1160 complete mutual communication through the bus interface 1140. The memory 1130 is used to store computer programs; the gyroscope 1150 is used to obtain swing parameters; the touch screen 1160 is used to receive touch operations; the processor 1110 can call the logic instructions in the memory 1130 to execute the above-mentioned VR video control method, which includes: determining a target control mode in response to a user's operation, the target control mode including a gyroscope control mode, a rocker control mode, or a touch control mode; obtaining control parameters corresponding to the target control mode; and interactively controlling the VR video on the display screen of the display terminal based on the control parameters.

[0249] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0250] The embodiment of the present application also provides a mobile terminal, which may include: a processor, a communication interface, a memory, a bus interface, a transceiver, a gyroscope and a touch screen. Among them, the processor, the communication interface, the memory, the transceiver, the gyroscope, and the touch screen communicate with each other through the bus interface. The memory is used to store computer programs; the transceiver is used to send and receive data under the control of the processor; the gyroscope is used to obtain swing parameters; the touch screen is used to receive touch operations; the processor can call the logic instructions in the memory to execute the above-mentioned VR video control method, which includes: in response to the user's operation, determining the target control mode, the target control mode includes a gyroscope control mode, a rocker control mode or a touch control mode; obtaining the control parameters corresponding to the target control mode; and interactively controlling the VR video on the display screen of the display terminal based on the control parameters.

[0251] Fig.12 is a schematic diagram of the physical structure of the display terminal provided in the embodiment of the present application, such as Fig.12As shown, the display terminal may include: a processor 1210, a communication interface 1220, a memory 1230, a bus interface 1240, and a display 1250. The processor 1210, the communication interface 1220, the memory 1230, and the display 1250 communicate with each other through the bus interface 1240. The memory 1230 is used to store computer programs; the display 1250 is used to display VR videos; the processor 1210 can call the logic instructions in the memory 1230 to execute the above-mentioned VR video control method, which includes: receiving control parameters corresponding to the target control mode sent by the mobile terminal; the target control mode includes a gyroscope control mode, a joystick control mode, or a touch control mode; based on the control parameters, the VR video on the display screen of the display terminal is interactively controlled.

[0252] In addition, the logic instructions in the above-mentioned memory 1230 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0253] The embodiment of the present application also provides a display terminal, which may include: a processor, a communication interface, a memory, a bus interface, a transceiver and a display. Among them, the processor, the communication interface, the memory, the transceiver, and the display communicate with each other through the bus interface. The memory is used to store computer programs; the transceiver is used to send and receive data under the control of the processor; the display is used to display VR video; the processor can call the logic instructions in the memory to execute the above-mentioned VR video control method, which includes: receiving control parameters corresponding to the target control mode sent by the mobile terminal; the target control mode includes a gyroscope control mode, a rocker control mode or a touch control mode; based on the control parameters, the VR video on the display screen of the display terminal is interactively controlled.

[0254] On the other hand, an embodiment of the present application also provides a VR video control system, which includes a mobile terminal and a display terminal as provided in the above embodiments.

[0255] On the other hand, an embodiment of the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the steps of the VR video control method provided in the above embodiments, for example, including: determining a target control mode in response to a user's operation, the target control mode including a gyroscope control mode, a joystick control mode or a touch control mode; obtaining control parameters corresponding to the target control mode; and interactively controlling the VR video on the display screen of the display terminal based on the control parameters.

[0256] Alternatively, the computer can execute the steps of the VR video control method provided in the above embodiments, for example including: receiving control parameters corresponding to the target control mode sent by the mobile terminal; the target control mode includes a gyroscope control mode, a joystick control mode or a touch control mode; based on the control parameters, interactively controlling the VR video on the display screen of the display terminal.

[0257] On the other hand, an embodiment of the present application also provides a processor-readable storage medium, which stores a computer program, and the computer program is used to enable the processor to execute the steps of the VR video control method provided in the above embodiments, for example, including: determining a target control mode in response to a user's operation, the target control mode including a gyroscope control mode, a joystick control mode or a touch control mode; obtaining control parameters corresponding to the target control mode; and interactively controlling the VR video on the display screen of the display terminal based on the control parameters.

[0258] Alternatively, the computer can execute the steps of the VR video control method provided in the above embodiments, for example including: receiving control parameters corresponding to the target control mode sent by the mobile terminal; the target control mode includes a gyroscope control mode, a joystick control mode or a touch control mode; based on the control parameters, interactively controlling the VR video on the display screen of the display terminal.

[0259] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.

[0260] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0261] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0262] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A VR video control method, characterized in that: Applied to a mobile terminal, the method comprises: In response to the user's operation, determining a target control mode, wherein the target control mode includes a gyroscope control mode, a joystick control mode, or a touch control mode; Obtaining control parameters corresponding to the target control mode; The VR video on the display screen of the display terminal is interactively controlled based on the control parameters.

2. The VR video control method according to claim 1, characterized in that: The obtaining of the control parameters corresponding to the target control mode includes: Based on the correspondence between the control mode and the control strategy, determining the target control strategy corresponding to the target control mode; the target control strategy is used to characterize the type of parameters to be obtained under the target control mode; Based on the parameter type, a control parameter corresponding to the target control mode is acquired.

3. The VR video control method according to claim 2, characterized in that: The target control mode is a gyroscope control mode, and the parameter type includes a swing parameter type of a gyroscope in the mobile terminal; The acquiring, based on the parameter type, a control parameter corresponding to the target control mode includes: When receiving a field of view adjustment instruction input by a user, acquiring a first swing parameter of the gyroscope based on the swing parameter type of the gyroscope; The first swing parameter is determined as the control parameter.

4. The VR video control method according to claim 2, characterized in that: The target control mode is a joystick control mode, and the parameter type includes a joystick parameter type; The acquiring, based on the parameter type, a control parameter corresponding to the target control mode includes: Based on the joystick parameter type, obtaining the joystick movement distance and joystick movement direction input by the user through the joystick control; the joystick control is displayed in the interface corresponding to the joystick control mode; The joystick moving distance and the joystick moving direction are determined as the control parameters.

5. The VR video control method according to claim 2, characterized in that: The target control mode is a touch control mode, and the parameter type includes a touch parameter type; The acquiring, based on the parameter type, a control parameter corresponding to the target control mode includes: Based on the touch parameter type, acquiring the touch distance and touch direction input by the user through the touch area; the touch area is displayed in the interface corresponding to the touch control mode; The touch distance and the touch direction are determined as the control parameters.

6. The VR video control method according to any one of claims 1 to 5, characterized in that: The obtaining of the control parameters corresponding to the target control mode includes: Acquire a second swing parameter of the gyroscope in the mobile terminal, and determine the second swing parameter as the control parameter; The interactively controlling the VR video on the display screen of the display terminal based on the control parameter includes: sending the second swing parameter to the display terminal; the second swing parameter is used to instruct the display terminal to display a virtual ray determined based on the second swing parameter, the virtual ray pointing to the first object to be selected in the VR video; Acquire a first selection instruction input by a user for a first target object to be selected; The first selection instruction is sent to the display terminal; the first selection instruction is used to instruct the display terminal to select the first target object to be selected.

7. The VR video control method according to any one of claims 1 to 5, characterized in that: The obtaining of the control parameters corresponding to the target control mode includes: In the case where the target control mode includes a joystick control mode, obtaining a click instruction of a user for a target direction control, and determining the click instruction as the control parameter; the target direction control is displayed in an interface corresponding to the joystick control mode; The interactively controlling the VR video on the display screen of the display terminal based on the control parameter includes: Sending the click instruction to the display terminal; the click instruction is used to instruct the display terminal to determine the second object to be selected in the VR video based on the click instruction; Acquire a second selection instruction input by a user for a second target object to be selected; The second selection instruction is sent to the display terminal; the second selection instruction is used to instruct the display terminal to select the second target object to be selected.

8. A VR video control method, characterized in that: Applied to a display terminal, the method comprises: Receiving control parameters corresponding to a target control mode sent by a mobile terminal; the target control mode includes a gyroscope control mode, a joystick control mode or a touch control mode; Based on the control parameters, the VR video on the display screen of the display terminal is interactively controlled.

9. The VR video control method according to claim 8, characterized in that: The interactively controlling the VR video on the display screen of the display terminal based on the control parameter includes: Determining a first three-dimensional coordinate of the mobile terminal in a three-dimensional space based on the control parameter; Mapping the first three-dimensional coordinates onto a display screen of the display terminal to obtain first two-dimensional coordinates corresponding to the first three-dimensional coordinates; The field of view of the VR video on the display screen is controlled based on the first two-dimensional coordinates.

10. The VR video control method according to claim 8, characterized in that: The control parameter includes a second swing parameter of the gyroscope acquired by the mobile terminal in the target control mode; The interactively controlling the VR video on the display screen of the display terminal based on the control parameter includes: displaying a virtual ray based on the second swing parameter; Determine the object in the VR video pointed by the virtual ray as the first object to be selected; receiving a first selection instruction sent by the mobile terminal; In response to the first selection instruction, a first target to-be-selected object corresponding to the first selection instruction is selected.

11. The VR video control method according to claim 10, characterized in that: The displaying of the virtual ray based on the second swing parameter comprises: determining a second three-dimensional coordinate of the mobile terminal in the three-dimensional space based on the second swing parameter; Mapping the second three-dimensional coordinates onto the display screen of the display terminal to obtain second two-dimensional coordinates corresponding to the second three-dimensional coordinates; The virtual ray is displayed at the position of the second two-dimensional coordinate.

12. The VR video control method according to claim 8, characterized in that: The control parameter includes a user click instruction for a target direction control obtained by the mobile terminal in the target control mode; the target control mode includes a joystick control mode; the target direction control is displayed in an interface corresponding to the joystick control mode; The interactively controlling the VR video on the display screen of the display terminal based on the control parameter includes: In response to the click instruction, determining a second to-be-selected object corresponding to the click instruction; receiving a second selection instruction sent by the mobile terminal; In response to the second selection instruction, a second target to-be-selected object corresponding to the second selection instruction is selected.

Images