A control method and control device

Abstract

Embodiments of the present application disclose a control method and a control device. The method comprises: obtaining a first operation and operation information thereof; controlling a second electronic device to render at least part of to-be-output content based on at least the operation information; and presenting the content rendered by the second electronic device and content generated based on at least the first operation in a virtual display area.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of control, in particular to a control method and a control device. BACKGROUND

[0002] Virtual Reality (VR) technology is a computer simulation system that can create and experience a virtual world. It uses a computer to generate a simulated environment, allowing users to immerse themselves in the environment. With the continuous development of science and technology, VR technology has been recognized by more and more people. Using VR devices based on VR technology, users can have a more immersive sense of immersion.

[0003] In related technologies, the commonly used VR devices are mainly divided into host type VR devices and integrated VR devices. However, the host type VR device needs to use a cable to connect the VR headset device and the VR host, which limits the movement of the user and reduces the user experience. Although the integrated VR device can solve the problem of limited movement of the host type VR device, due to the volume limitation, the performance of the integrated VR device is limited, and the integrated VR device may appear lag and delay during use, resulting in poor user experience. SUMMARY

[0004] The technical solution of the present application is implemented as follows:

[0005] The control method provided by the present application comprises:

[0006] obtaining a first operation and operation information thereof, the first operation being used to input a control instruction to a first electronic device to at least control the display of at least one virtual display area displayed by an optical lens module of the first electronic device;

[0007] controlling at least part of the content to be output by a second electronic device based at least on the operation information, the second electronic device being a device that establishes a communication connection with the first electronic device;

[0008] presenting the content rendered by the second electronic device and the content generated based at least on the first operation in the virtual display area, wherein at least part of the content generated based at least on the first operation is generated by the first electronic device.

[0009] The control device provided by the present application comprises:

[0010] The obtaining module is configured to obtain a first operation and operation information thereof, the first operation being used to input a control instruction to a first electronic device to at least control the display of at least one virtual display area displayed by an optical lens module of the first electronic device;

[0011] a control module configured to control a second electronic device to render at least part of the content to be output based on at least the operation information, the second electronic device being a device that establishes a communication connection with the first electronic device;

[0012] a display module configured to present the content rendered by the second electronic device and the content generated based on at least the first operation in the virtual display area in a corresponding manner, wherein at least part of the content generated based on at least the first operation is rendered by the first electronic device.

[0013] Embodiments of the present application also provide an electronic device, comprising:

[0014] a memory storing executable control instructions;

[0015] a processor capable of implementing embodiments of the control method of the present application when executing the control instructions.

[0016] Embodiments of the present application provide a control method and a control device. According to the technical solution, first, the first operation and the operation information thereof are obtained, so that the display of at least one virtual display area displayed by the optical lens module of the first electronic device can be controlled based on the first operation. Then, at least part of the content to be output is rendered by the second electronic device based on at least the operation information. Finally, the content rendered by the second electronic device and the content generated based on at least the first operation are presented in the virtual display area in a corresponding manner. In this way, at least part of the content to be output is rendered by the second electronic device based on the operation information, and at least part of the content generated based on the first operation is rendered by the first electronic device, which reduces the rendering workload of the first electronic device, improves the processing efficiency of the first electronic device, and further ensures the fluency and continuity of content presentation in the virtual display area, thereby improving the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 a flowchart of a control method provided by an embodiment of the present application;

[0018] Figure 2 a flowchart of a method for controlling a second electronic device to render content to be output provided by an embodiment of the present application;

[0019] Figure 3 a VR architecture based on edge computing provided by an embodiment of the present application;

[0020] Figure 4 a flowchart of a data processing method based on edge computing provided by an embodiment of the present application;

[0021] Figure 5A schematic diagram of data interaction between a VR device and an edge computing device is provided for an embodiment of the present application.

[0022] Figure 6 A structural schematic diagram of a control device is provided for an embodiment of the present application.

[0023] Figure 7 A structural schematic diagram of a control device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0025] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0026] In the following description, “some embodiments / other embodiments” are described, which describe a subset of all possible embodiments, but it can be understood that “some embodiments / other embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0027] In the following description, the terms “first / second / third” are only to distinguish similar objects, and do not represent a specific order of the objects, and it can be understood that “first / second / third” can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0028] VR technology is a technology realized by using computers and images, and has been widely used in games, business, medical treatment, entertainment and other aspects. Based on VR technology, users can experience the most real feeling in the VR world. The realness of the VR technology simulation environment is difficult to distinguish from the real world, and it produces a sense of being in the scene. At the same time, VR has all the perceptual functions that humans have, such as auditory, visual and other perceptual systems. In addition, VR technology has a super strong simulation system, which truly realizes man-machine interaction, so that people can operate at will and get the most real feedback from the environment. It is due to the existence, multi-sensing and interaction of VR technology that it has been more recognized and widely used.

[0029] However, for the VR technology, the VR device is essential, through the VR device, the storage, rendering, display and the like of the VR data are realized, wherein the rendering workload of the VR data is large, in order to ensure that the user has a good experience in the process of using the VR device, it is necessary to ensure that the VR device has high processing performance when rendering the VR data.

[0030] The embodiment of the application provides a control method, which is applied to a VR device, an augmented reality (AR) device or a mixed reality (MR) device, can avoid delay and lag phenomenon of various devices, and improves the experience of a user. In the following, the control method provided by the embodiment of the application will be described, as shown in the figure, a flowchart of the control method provided by the embodiment of the application, the method comprises the following steps: Figure 1

[0031] S101, obtaining a first operation and operation information thereof.

[0032] It should be noted that the first operation is used to input a control instruction to the first electronic device, so as to control at least the display of at least one virtual display area displayed by the optical lens module of the first electronic device. The first electronic device can be a VR device, an AR device or an MR device, the first operation can be an action performed by a user in the process of interacting with VR / AR / MR content, for example, the user realizes corresponding actions such as lifting hands and kicking legs through the controls of the control device connected with the first electronic device, and the operation information of the first operation can be information corresponding to the first operation, for example, the intensity, speed, direction, amplitude and the like information collected by the first electronic device in the process of lifting hands. In practice, after obtaining the first operation, other operations can also be obtained. The optical lens module of the first electronic device can realize the display of VR / AR / MR content in one or more virtual display areas.

[0033] In some embodiments, the control instruction can be sent to the first electronic device through the first operation, and the display of the optical lens module of the first electronic device in one or more virtual display areas can be controlled, the virtual display area can be a display area corresponding to a foreground object in a VR / AR / MR picture, or a display area corresponding to a background environment in which the foreground object is located in the VR / AR / MR picture, wherein the foreground object can be, for example, a character in a VR game, and the background environment in which the foreground object is located can be a room, a street, a forest and the like in which the character in the VR game is located. The display area corresponding to the foreground object can be divided into one or more, and the display area corresponding to the background environment can also be divided into one or more.

[0034] S102, at least based on the operation information, controlling the second electronic device to render at least part of the to-be-output content. ​

[0035] It should be noted that the second electronic device is a device that establishes a communication connection with the first electronic device, and the second electronic device can be an edge computing device, such as an edge computing server, a desktop computer, an all-in-one machine, a PC terminal, a workstation, or the like. The first electronic device and the second electronic device can perform wireless transmission, for example, through a wireless connection mode such as 5G or WIFI. The content to be output can be VR / AR / MR picture content for showing to a user. In practice, the content to be output can be background environment data in a VR / AR / MR picture. For example, in a VR / AR / MR game, the content to be output can be a scene environment in which a character in the game is located, such as a room in which the character is located. The content to be output can also include a foreground object and an action performed by the foreground object caused by operation data (first operation, etc.). For example, in a VR / AR / MR game, a game character, and when a user performs an operation through a control device, the control game character performs a corresponding action. In addition, in an AR scene, the content to be output can also include content for rendering a real environment. At least part of the content to be output rendered by the second electronic device can be content (such as background data or environment data, and part of virtual object data) that needs to be rendered in the content to be output. The rendering of the content to be output can be processing the content to be output through 3D software (such as 3dsmax and Maya, etc.) to obtain more stereoscopic and realistic image content.

[0036] In some embodiments, the second electronic device can be one or more. When the second electronic device is multiple, a communication connection can be established between the first electronic device and the multiple second electronic devices. When controlling the rendering of at least part of the content to be output by the second electronic device based on the control information, different second electronic devices can be switched according to the load condition or position of the second electronic device, or multiple second electronic devices can be controlled at the same time to perform rendering of the part of the content to be output corresponding to each second electronic device.

[0037] In some embodiments, the rendering of at least part (such as a first part) of the content to be output can be completed by the second electronic device. Based on the operation information, the first electronic device can be controlled to render at least part (such as a second part, i.e., a remaining part of the content to be output excluding the first part) of the content to be output. When a user performs an operation through a control device, the content to be output rendered by the second electronic device can include background environment data in a VR / AR / MR picture. Based on the operation information obtained at the current time, the background environment in which the foreground object is located at the next time can be predicted, and the second electronic device is controlled to render the background environment data in which the foreground object is located at the next time.

[0038] In some embodiments, when the user performs an operation through the control device, the at least part of the content to be output rendered by the second electronic device can also include a plurality of background environment data sets in which the foreground object in the VR / AR / MR picture is located at a future time, i.e., data corresponding to a plurality of background environments adjacent to the background environment in which the foreground object is currently located, wherein each background environment data set corresponds to a background environment, and the future time can be the next time, the time after next, etc. Through the obtained operation information, one or more background environments corresponding to the foreground object at the future time can be predicted, so as to control the second electronic device to render the one or more background environments.

[0039] In some embodiments, when the user does not perform an operation through the control device such as a handle, the foreground object does not perform any action, and at this time, the at least part of the content to be output rendered by the second electronic device can include the background environment data and the foreground object in the VR / AR / MR picture, i.e., at this time, the second electronic device renders all of the content to be output. Based on the operation information obtained at the current time and the foreground object acted on by the operation information, the background environment in which the foreground object is located at the next time can be predicted, and then the second electronic device is controlled to render the foreground object and the background environment data in which the foreground object is located at the next time or a plurality of background environment data sets in which the foreground object is located at a future time.

[0040] It can be understood that by using the second electronic device to render at least part of the content to be output, the rendering workload of the first electronic device can be reduced, and the processing efficiency of the first electronic device can be improved. At the same time, when the second electronic device is an edge computing device, the second electronic device is close to the first electronic device, can provide a nearest end service, and after completing the rendering work of the at least part of the content to be output, a faster network service response is generated, the at least part of the content to be output after rendering is fed back to the first electronic device, and the real-time, security and privacy protection and other aspects are met, and smooth display of the first electronic device in the virtual display area is ensured.

[0041] S103, the content rendered by the second electronic device and the content generated based on at least the first operation are correspondingly presented in the virtual display area.

[0042] In some embodiments, at least part of the content generated based on at least the first operation is generated by the first electronic device. When the user performs an operation through the control device, the content generated based on the first operation can be an action corresponding to the first operation performed by the user through the control device (such as a handle, a remote controller, etc.) to control the foreground object; when the user does not perform an operation through the control device, the content generated based on the first operation can be the foreground object.

[0043] In some embodiments, when the user does not perform an operation through the control device, at least part of the content generated by the first operation rendered by the first electronic device can include a foreground object, and the background environment data corresponding to the foreground object can be rendered by the second electronic device; in other embodiments, when the user performs an operation through the control device, at least part of the content generated by the first operation rendered by the first electronic device can only include a control foreground object to perform an action corresponding to the first operation, and at this time the background environment data corresponding to the foreground object and the foreground object can be completed by the second electronic device; in other embodiments, when the user performs an operation through the control device, at least part of the content generated by the first operation rendered by the first electronic device can also be a foreground object and a control foreground object to perform an action corresponding to the first operation, and at this time the background environment data corresponding to the foreground object can be completed by the second electronic device.

[0044] After the first electronic device completes rendering of at least part of the content generated by the first operation, virtual content for displaying to the user can be obtained. After obtaining the content rendered by the second electronic device and the content generated by the first operation, the two parts of content can be displayed together in a virtual display area to form a complete VR / AR / MR picture and present it to the user.

[0045] In the embodiments of the present application, by obtaining the first operation and the operation information thereof, the display of at least one virtual display area displayed by the optical lens module of the first electronic device can be controlled based on the first operation; at least part of the content to be output rendered by the second electronic device is controlled based on the operation information; and the content rendered by the second electronic device and the content generated based on the first operation are correspondingly presented in the virtual display area. By controlling the second electronic device to render at least part of the content to be output based on the operation information, the rendering workload of the first electronic device is reduced, the processing efficiency of the first electronic device is improved, and the fluency and continuity of content presentation on the virtual display area are ensured, thereby improving the user experience.

[0046] In some embodiments of the present application, obtaining the first operation and the operation information thereof, i.e., step S101, can be achieved by at least the following four ways, which are described below.

[0047] The first implementation, obtaining device information of the first electronic device, and obtaining the first operation and the operation information thereof based on the device information through a sensor in the first electronic device or a sensor in a third electronic device different from the first electronic device.

[0048] It should be noted that the device information of the first electronic device can be a device type of an AR, VR or MR device, or various device configuration information, etc. The sensor can be a speed sensor, a force sensor, a direction sensor, a 6DOF sensor, a gravity sensor, an acceleration sensor, an image sensor, etc. The sensor can be arranged in the first electronic device, for example, a VR head-mounted device can be controlled by head movements, and a gyroscope in the head-mounted device can detect the direction and displacement of the object movement. The sensor can also be arranged in a third electronic device different from the first electronic device, and the third electronic device can be a controller such as a handle or a remote controller. When the sensor is arranged in the third electronic device, the third electronic device is communicatively connected to the first electronic device. In practice, the first electronic device can obtain the first operation collected by the third electronic device and the operation information corresponding to the first operation through wired or wireless communication.

[0049] After obtaining the device information of the first electronic device, the first operation obtained by the sensor in the first electronic device and the operation information of the first operation can be transmitted to the storage module of the first electronic device for storage based on the device information of the first electronic device, or the communication connection between the first electronic device and the third electronic device can be established based on the device information of the first electronic device, and the first operation collected by the sensor in the third electronic device and the operation information corresponding to the first operation can be sent to the storage module of the first electronic device for storage.

[0050] In the second implementation, the relative position relationship between the first electronic device and the target operation body is obtained, and the first operation and the operation information thereof are obtained by the sensor in the first electronic device or the sensor in the third electronic device different from the first electronic device based on the relative position relationship.

[0051] It should be noted that the target operation body can be a user, and the user can interact with the VR / AR / MR content. The relative position relationship can be the position relationship between the user and the first electronic device, and the position relationship can be distance, direction, etc. After determining the relative position relationship between the first electronic device and the target operation body, the first operation information executed by the target operation body and the operation information corresponding to the first operation can be obtained by the sensor in the first electronic device or the sensor in the third electronic device different from the first electronic device based on the relative position relationship between the first electronic device and the target operation body.

[0052] In the third implementation, the use parameter of the first electronic device is obtained, and the first operation and the operation information thereof are obtained by the sensor in the first electronic device or the sensor in the third electronic device different from the first electronic device based on the use parameter.

[0053] It should be noted that the use parameter of the first electronic device can be a use state, a use mode or a use scenario of the first electronic device, the use state of the first electronic device can be that the first electronic device is being used, paused or not used, the use mode can be head control, eye control or remote control device control, and the use scenario can be a shooting game scenario, an adventure game scenario, etc. After obtaining the use parameter of the first electronic device, the first operation and the operation information corresponding to the first operation can be obtained based on the use parameter of the first electronic device, for example, it is determined that the use state of the first electronic device is not used, the use mode is remote control device control, and the use scenario is a shooting game scenario.

[0054] In a fourth implementation manner, the identity information of the operator of the first electronic device is obtained, and the first operation and the operation information corresponding to the first operation are obtained through the sensor in the first electronic device or the sensor in the third electronic device different from the first electronic device based on the identity information.

[0055] The identity information of the operator of the first electronic device can be identity information of a user, such as height, weight, etc. of the user. The identity information of the operator corresponds to the operation (for example, the first operation based on the remote control device or other operations except the first operation) performed by the operator. After obtaining the identity information of the operator, the first operation and the operation information corresponding to the first operation can be obtained from the sensor in the first electronic device or the sensor in the third electronic device different from the first electronic device based on the identity information of the operator.

[0056] As shown in Figure 2 Fig. 1 is a flowchart of a method for controlling a second electronic device to render at least part of to-be-output content according to an embodiment of the present application. In some embodiments of the present application, at least based on the operation information, the second electronic device is controlled to render at least part of the to-be-output content, that is, step S102 can be implemented by steps S1021A to S1022A described below. Each step is described below.

[0057] S1021A, determining the to-be-output content at the second time based on the operation information.

[0058] It should be noted that the second time is later than the first time, and the first time is the time when the first operation is obtained. The to-be-output content at the second time can include background environment data of a foreground object in a VR / AR / MR picture at the second time. At the first time, the operation and the operation information corresponding to the first operation can be obtained based on the first electronic device. The operation information can predict the background environment of the foreground object at the second time, that is, based on the operation information, the to-be-output content at the second time can be determined.

[0059] Exemplarily, in the VR game, the character in the game is the foreground object, the current scene where the character is located at the first moment is the background environment, in practice, the corresponding relationship between the operation information of the character and the scene where the character is located is the design rule of the VR game, which can be obtained in advance. When the operation information is obtained at the first moment, the scene where the character is located at the second moment (the next scene relative to the current scene) can be determined based on the operation information. For example, the character is located in room A at the first moment, and based on the operation information (such as speed, direction, etc.) of the character at the first moment, it can be determined that the next scene where the character will enter is room B, that is, the character will be in room B at the second moment.

[0060] S1022A, control the second electronic device to render at least part of the to-be-rendered content corresponding to the to-be-output content of the second moment at the third moment.

[0061] It should be noted that the third moment is not later than the second moment, that is, at least part of the to-be-rendered content corresponding to the to-be-output content of the second moment is controlled to be rendered by the second electronic device before the second moment, the to-be-output content of the second moment corresponds to the second operation, and the action performed by the foreground object controlled by the second operation and the rendered to-be-output content of the second moment are displayed in the same VR / AR / MR picture.

[0062] It can be understood that the to-be-rendered content of the to-be-output content of the second moment is rendered by the second electronic device before the second moment, so that the to-be-output content of the second moment after rendering is obtained before the second moment, and the to-be-output content of the second moment after rendering is displayed together with the content corresponding to the second operation control instruction in the virtual reality area of the first electronic device after the second moment, ensuring smooth display of the first electronic device.

[0063] In some embodiments, the above step S1021A “determining the to-be-output content of the second moment based on the operation information” can be implemented by the following steps S21A1 to S21A3.

[0064] S21A1, predicting the second operation of the second moment based on the operation information.

[0065] It should be noted that the second operation is the operation obtained by the electronic device at the second moment, the second operation is the operation after the first operation, and the second operation has a similar effect to the first operation, which is used to input a control instruction to the first electronic device to control the display of at least one virtual area of the optical lens module of the first electronic device. The second operation can also be an action performed by the user in the process of interacting with the VR / AR / MR content, for example, the user performs corresponding actions such as lifting hands and kicking legs through the controls of the control device connected to the first electronic device.

[0066] In some embodiments, the second operation can be the same as the first operation, for example, the first operation and the second operation are both the same control instruction sent by the user to the first electronic device using the same control on the remote control. In other embodiments, the first operation and the second operation can also be different, for example, the first operation is that the user uses control A on the remote control to send control instruction a to the first electronic device, and the second operation is that the user uses control B on the remote control to send control instruction b to the first electronic device. By obtaining the first operation at the first time and the operation information corresponding to the first operation, the second operation corresponding to the second time can be predicted, for example, according to the force, speed, direction, formation, etc. of the first operation, the second operation that the user can perform at the second time can be predicted. In practice, for example, in a VR game, one or more operations performed at the first time and one or more operations performed at the second time are set during the development of the VR game. According to the first operation performed at the first time and the operation information corresponding to the first operation, one or more operations that can be performed at the second time, i.e., the second operation, can be predicted.

[0067] S21A2, determining a first relationship between the second operation and the first operation, and a second relationship between the first operation and the current output content.

[0068] In some embodiments, after the operation body or the user performs the first operation at the first time, the operation that can be performed at the second time (the second operation) based on the first operation has continuity with the first operation, for example, the first operation is to raise the hand, and according to the operation information of the first operation, the second operation can be to turn left or turn right. The first relationship between the first operation and the second operation can be a corresponding relationship between the first operation and the second operation, and one first operation can correspond to multiple second operations. The current output content can be the background environment in which the foreground object is located when the foreground object performs the corresponding action based on the first operation. The corresponding relationship between the first operation and the current output content is the second relationship.

[0069] S21A3, determining the to-be-output content at the second time based on the first relationship and the second relationship.

[0070] It should be noted that the to-be-output content at the second time can be the output content corresponding to the second operation obtained at the second time, for example, the second operation obtained at the second time is to turn left, and the to-be-output content at the second time is the data corresponding to the background environment in which the foreground object is located when the foreground object performs the operation of turning left. After the first relationship between the second operation and the first operation and the second relationship between the first operation and the current output content are determined, the to-be-output content at the second time can be determined. In practice, since there can be multiple operations at the second time, the to-be-output content at the second time determined based on the first relationship and the second relationship can be a data set composed of multiple background environment corresponding data.

[0071] In some embodiments of the present application, after step S21A3, the following step S21A4 can also be performed.

[0072] S21A4, determining a target object in the to-be-output content on which the first operation or the second operation acts, rendering a to-be-rendered object corresponding to the target object at a corresponding time.

[0073] It should be noted that the target object in the to-be-output content on which the operation or the user interacts can be a foreground object, and the first operation and the second operation can control the target object to perform an action corresponding to the respective operation. The to-be-rendered object corresponding to the target object can be the target object itself or a part of the target object. The rendering of the to-be-rendered object corresponding to the target object at the first time and the to-be-rendered object corresponding to the target object at the second time can be completed by the first electronic device.

[0074] In some embodiments of the present application, in addition to being implemented by the above steps S1021A to S1022A, the above step S102“controlling the second electronic device to render at least part of the to-be-output content based on at least the operation information” can also be implemented by one of the following steps S1021B, S1021C or S1021D, which will be described below.

[0075] Step S1021B, sending the to-be-rendered content determined based on the operation information to the second electronic device at the first time, to control the second electronic device to render at least part of the to-be-output content corresponding to the second time at or before the second time.

[0076] It should be noted that the first time is the time when the first operation is obtained, and the second time is later than the first time. After obtaining the to-be-rendered content (to-be-output content at the second time) determined based on the operation information at the first time, the to-be-rendered content can be immediately sent to the second electronic device to control the second electronic device to render at least part of the to-be-output content at the second time at or before the second time. After controlling the second electronic device to render the to-be-output content at the second time at or before the second time, the second electronic device is controlled to immediately feed back the to-be-output content at the second time after rendering to the first electronic device, so as to ensure that the first electronic device can display the to-be-output content at the second time in time at the second time.

[0077] Step S1021C, sending at least the operation information to the second electronic device at the first time, so that the second electronic device renders at least part of the to-be-output content corresponding to the second time at or before the second time based on the operation information.

[0078] It should be noted that the first time is the time when the first operation is obtained, and the second time is later than the first time. By sending the operation information corresponding to the first operation to the second electronic device at the first time, the second electronic device can determine the second operation based on the operation information corresponding to the first operation, and determine the to-be-output content corresponding to the second time of the second operation, so as to control the second electronic device to render at least part of the to-be-output content corresponding to the second time at or before the second time.

[0079] Step S1021D, controlling the third electronic device to send the collected operation information to the second electronic device, so that the second electronic device renders at least part of the to-be-output content corresponding to the second time at or before the second time based on the operation information.

[0080] In some embodiments, the first time is the time when the first operation is obtained, and the second time is later than the first time. The third electronic device can be a control device different from the first electronic device and the second electronic device, such as a remote controller or a handle, etc. The operation information corresponding to the first operation can be collected through the sensor in the third electronic device, and the collected operation information is sent to the second electronic device, so that the second electronic device renders at least part of the to-be-output content corresponding to the second time at or before the second time based on the operation information.

[0081] In some embodiments of the present application, after step S1021B, or step S1021C, or step S1021D is executed, the following step S201 or step S202 can also be executed.

[0082] S201, sending a data request of the to-be-output content to the second electronic device to control the second electronic device to feed back at least part of the to-be-output content rendered by the second electronic device.

[0083] After controlling the second electronic device to render at least part of the to-be-output content based on the control information, the first electronic device can send a data request of the to-be-output content to the second electronic device, so that based on the data request of the to-be-output content, the first electronic device receives the to-be-output content rendered by the second electronic device.

[0084] In some embodiments, there can be multiple second electronic devices for rendering the to-be-output content. When there are multiple second electronic devices for rendering the to-be-output content, the data request of the to-be-output content needs to be sent to the multiple second electronic devices. After receiving the data request, the multiple second electronic devices feed back the corresponding part of the to-be-output content rendered by each of the second electronic devices to the first electronic device.

[0085] S202, receiving at least part of the to-be-output content actively fed back by the second electronic device.

[0086] The second electronic device can also actively feed back the rendered to-be-output content. In some embodiments, the second electronic device sends the rendered content to the first electronic device immediately after the rendering is completed. In some other embodiments, the second electronic device actively monitors the operations (e.g., the first operation, the second operation, or other operations) obtained by the first electronic device, and sends the rendered content corresponding to the operations to the first electronic device. In some other embodiments, the second electronic device calculates or predicts the real actions performed by the user or the operation body (e.g., the user performs the corresponding action of a control on a remote control through the control), based on the obtained operation information, and sends the rendered content corresponding to the calculated or predicted real actions to the first electronic device.

[0087] It can be understood that, by actively feeding back the processed to-be-output content to the first electronic device through the second electronic device, the number of communications between the first electronic device and the second electronic device can be reduced, thereby accelerating the feedback speed of the second electronic device to the first electronic device, so that the first electronic device can timely present the processed to-be-output content and the content generated based on the first operation in the virtual display area after receiving the feedback of the second electronic device, and ensure the smoothness and continuity of the display in the virtual display area.

[0088] In some embodiments of the present application, the content rendered by the second electronic device and the content generated based on the first operation are presented in the virtual display area, that is, step S103 can be implemented through step S103A or step S103B, which are described below.

[0089] S103A, synchronously display the content rendered by the second electronic device and the content generated based on the first operation in the virtual display area.

[0090] The content rendered by the second electronic device is at least part of the to-be-output content, and the content generated based on the first operation is the content obtained by the first electronic device after rendering the content corresponding to the first operation, including the actions of the target object controlled by various operations (the first operation, the second operation, etc.). The synchronous display can be simultaneous display of the content rendered by the second electronic device and the content generated based on the first operation in the virtual display area. After receiving the rendered to-be-output content from the second electronic device, the first electronic device determines the content matching the first operation in the at least part of the rendered to-be-output content, and displays the content matching the first operation together with the content generated based on the first operation in the virtual display area.

[0091] S103B, display the content rendered by the second electronic device and the content generated based on the first operation in the first virtual display area and the second virtual display area, respectively.

[0092] In some embodiments, the first virtual display area and the second virtual display area are in the same plane or different planes, the content displayed by the first virtual display area can be the content rendered by the second electronic device, for example, the content rendered after the foreground object is rendered in the background environment, and the content displayed by the second virtual display area can be the content generated based on the first operation, for example, the content rendered after the action performed by the foreground object controlled by the first operation or the second operation.

[0093] In some embodiments of the present application, the above step S103 "corresponding display of the content rendered by the second electronic device and the content generated based on the first operation in the virtual display area" can be implemented by S103A or S103B, and can also be implemented by S103C.

[0094] S103C, corresponding display of the content rendered by the second electronic device and the content generated based on the first operation in the first virtual display area and the first display area.

[0095] In some embodiments, the first display area can be a display area driven by the second electronic device, and the display area driven by the second electronic device can be a display screen configured by the second electronic device, or a projection display area driven by the second electronic device.

[0096] In some embodiments, the content rendered by the second electronic device can be displayed in the first virtual display area, and the content generated based on the first operation can be displayed in the first display area; or the content rendered by the second electronic device and the content generated based on the first operation can be displayed in the first virtual display area; or the content rendered by the second electronic device and the content generated based on the first operation can be displayed in the first display area.

[0097] In some embodiments, the first virtual display area and the first display area are in the same plane or different planes, and in practice, the first virtual display area can be displayed in any position in the same plane of the second display area, for example, the upper left corner or the lower right corner of the first display area, and the content displayed by the first virtual display area can also be in different planes of the second display area, for example, the front left or the back right of the first display area.

[0098] Exemplarily, in the scenario of cooperation between the AR device and the edge computing device, the virtual content generated after the AR device renders the content generated based on the first operation can be displayed in the first virtual display area, and the content rendered by the edge computing device can be displayed on the display screen or the projection display area driven by the second device.

[0099] The embodiment of the application obtains the first operation and operation information thereof, so that the display of at least one virtual display area displayed by the optical lens module of the first electronic device can be controlled based on the first operation; at least part of the content to be output is controlled to be rendered by the second electronic device based on the operation information; and the content rendered by the second electronic device and the content generated based on the first operation are correspondingly presented in the virtual display area. The operation information is used to control the second electronic device to render at least part of the content to be output, thereby reducing the rendering workload of the first electronic device, improving the processing efficiency of the first electronic device, and further ensuring the fluency and continuity of content presentation in the virtual display area, and improving the user experience. In the following, the implementation process of the embodiment of the application in the actual application scenario is introduced.

[0100] The embodiment of the application provides a data processing method based on edge computing, which is implemented based on an edge computing-based VR architecture 1. Figure 3 A VR architecture based on edge computing is provided for the embodiment of the application, as shown in Figure 3 The edge computing-based VR architecture 1 includes an edge computing device 11 and a VR (VR or MR) device 12.

[0101] The edge computing device 11 and the VR (AR or MR) device 12 are communicatively connected through a wireless transmission mode such as 5G or WIFI, and the edge computing device 11 and the VR (AR or MR) device 12 perform collaborative rendering. The edge computing device 11 performs rendering of background environment data, the VR (AR or MR) device 12 performs rendering of foreground interaction data, and the background environment data obtained after rendering by the edge computing device 11 and the foreground interaction data obtained after rendering by the VR (AR or MR) device 12 are synchronously displayed.

[0102] As shown in Figure 4 A flowchart of a data processing method based on edge computing is provided for the embodiment of the application, and the data processing method based on the edge computing device provided by the embodiment of the application can be implemented based on the edge computing-based VR architecture through S301 to S304.

[0103] S301, the VR device (first electronic device) processes (renders) the foreground interaction data (content generated by the first operation) of the VR content in real time.

[0104] In some embodiments, the VR content includes foreground interaction data and background environment data (at least part of the content to be output). The foreground interaction data can be content corresponding to the interaction of the user with the VR content, for example, content generated when the user controls the foreground object in the VR picture to perform corresponding operations through a controller such as a handle. The foreground interaction data is real-time and randomly unpredictable, but the rendering workload of the foreground interaction data is small, so the rendering of the foreground interaction data can be processed by the VR device in real time to ensure the performance of the VR device.

[0105] S302, the edge computing device (the second electronic device) processes the background environment data of the VR content (at least part of the content to be output) in advance.

[0106] The background environment is continuous and predictable, and can be predicted in advance, but the rendering workload of the background environment data is large, so the rendering of the background environment data can be completed by the edge computing device.

[0107] In some embodiments, the edge computing device includes one or more, and each of the one or more edge computing devices has a corresponding identification code. The VR device can obtain the corresponding identification code of each of the one or more edge computing devices, and establish a wireless connection between the VR device and the one or more edge computing devices based on the identification code of the one or more edge computing devices (the second electronic device is in communication connection with the first electronic device).

[0108] Figure 5 A data interaction schematic diagram between a VR device and an edge computing device provided by an embodiment of the present application is shown in FIG. 1. Figure 5 As shown in FIG. 1, when the edge computing device includes multiple edge computing devices, the VR operating system or the VR platform in the VR device can determine at least one available edge computing device from the multiple edge computing devices according to the load information of each of the multiple edge computing devices, determine the load capacity of the at least one available edge computing device, split the background environment source data corresponding to the background environment data into at least one background source data subset according to the load capacity of each of the available edge computing devices, so that each available edge computing device with the load capacity can process the corresponding background source data subset, and send the at least one background source data subset to the corresponding edge computing device. Each edge computing device receives the corresponding background source data subset and performs corresponding rendering processing.

[0109] S303, the edge computing device transmits the proximate background environment data obtained after processing (the content rendered by the second electronic device) to the VR device in advance.

[0110] In some embodiments, as shown in FIG. 1, Figure 5As shown, the VR device can send a data acquisition request to the edge computing device to obtain the processed background environment data fed back by the edge computing device. In practice, the VR device can obtain the current scene corresponding to the foreground object in the foreground interaction data, based on the preset rule of the foreground object, the VR content corresponding scene and the current scene corresponding to the foreground object. Then, at least one to-be-output scene corresponding to the foreground object is determined, and the to-be-output background source data corresponding to each to-be-output scene is determined, i.e. the background source data corresponding to the adjacent background environment data. Next, from one or more edge computing devices, the edge computing device processing the to-be-output background source data is determined as a candidate edge computing device, which can be one or more. Then, the data acquisition request is sent to one or more candidate edge computing devices to obtain the to-be-output background data processed from the to-be-output background source data, i.e. the adjacent background environment data processed by the edge electronic device.

[0111] S304, after the VR device receives the processed background environment data (the content rendered by the second electronic device) fed back by the edge computing device, the processed foreground interaction data and the processed background environment data are displayed in coordination.

[0112] After the VR device receives the background environment data sent by the edge computing device, the background environment data corresponding to the foreground interaction content can be determined, and the background environment data corresponding to the current foreground interaction content and the foreground interaction data are displayed in synchronization (the content rendered by the second electronic device and the content generated based on at least the first operation are presented in the virtual display area).

[0113] In some embodiments, when there are multiple candidate edge computing devices, multiple pieces of processed background environment data can be obtained, for example, as shown in Figure 5 As shown, the edge computing device A can process the background source data subset a to obtain one piece of background environment data, the edge computing device B can process the background source data subset b to obtain another piece of background environment data, and the VR device can determine the background environment data matching the foreground interaction in the multiple pieces of background environment data after receiving the multiple pieces of background environment data, and display the foreground interaction data and the background environment data in synchronization.

[0114] In some embodiments, the background environment data (background source data or background source data subset) that needs to be processed and sent by the VR device to the edge computing device can be compressed before being sent to the edge computing device, for example, as shown in Figure 5As shown, the background environment source data to be sent to the edge computing device is compressed by the VR device, and after the edge computing device receives the compressed data, the edge computing device performs rendering processing based on the edge computing device after decompressing the compressed data. Correspondingly, the processed background environment data fed back by the edge computing device to the VR device can also be obtained after compression processing by the intelligent acceleration platform of the edge computing device. After the compression processing, the background environment data can ensure the safe and efficient transmission of data between the VR device and the edge computing device.

[0115] In the embodiments of the present application, the foreground interaction data of the VR content is processed in real time by the VR device, and the background environment data of the VR content is processed in advance by the edge computing device. The edge computing device transmits the processed proximal background environment data to the VR device in advance. After the VR device receives the background environment data fed back by the edge computing device, the VR device performs cooperative display, solves the demand of the VR device for low delay, realizes the ultra-high-definition real-time interaction of the VR device, and solves the problem of insufficient performance of the current integrated VR.

[0116] The present application also provides a control device, Figure 6 A structural schematic diagram of the control device provided in the embodiments of the present application is shown in Figure 6 As shown, the control device 600 comprises: an acquisition module 601, configured to obtain a first operation and operation information thereof, the first operation being used to input a control instruction to a first electronic device to control at least a virtual display area displayed by an optical lens module of the first electronic device;

[0117] a control module 602, configured to control at least part of to-be-output content to be rendered by a second electronic device based on at least the operation information, the second electronic device being a device that establishes a communication connection with the first electronic device;

[0118] a display module 603, configured to present the content rendered by the second electronic device and the content generated based on at least the first operation in the virtual display area correspondingly, wherein at least part of the content generated based on at least the first operation is generated by the first electronic device.

[0119] In some embodiments of the present application, the obtaining module 601 is further configured to obtain device information of the first electronic device, obtain the first operation and the operation information thereof based on the device information through a sensor in the first electronic device or a sensor in a third electronic device different from the first electronic device; or, obtain a relative positional relationship between the first electronic device and the target operation body, obtain the first operation and the operation information thereof based on the relative positional relationship through the sensor in the first electronic device or the sensor in the third electronic device different from the first electronic device; or, obtain a use parameter of the first electronic device, obtain the first operation and the operation information thereof based on the use parameter through the sensor in the first electronic device or the sensor in the third electronic device different from the first electronic device; or, obtain identity information of an operation body of the first electronic device, obtain the first operation and the operation information thereof based on the identity information through the sensor in the first electronic device or the sensor in the third electronic device different from the first electronic device.

[0120] In some embodiments of the present application, the control module 602 is further configured to determine the to-be-output content at a second time based on the operation information, the second time being later than a first time, the first time being the time when the first operation is obtained; control the second electronic device to render at least part of the to-be-rendered content corresponding to the to-be-output content at the second time at a third time, the third time being not later than the second time.

[0121] In some embodiments of the present application, the control module 602 is further configured to predict a second operation at a second time based on the operation information; determine a first relationship between the second operation and the first operation, and a second relationship between the first operation and the current output content; determine the to-be-output content at the second time based on the first relationship and the second relationship.

[0122] In some embodiments of the present application, the control module 602 is further configured to send the content to be rendered determined based on the operation information to the second electronic device at a first time point, so as to control the second electronic device to render at least part of the content to be output corresponding to a second time point at or before the second time point, the first time point being the time point at which the first operation is obtained, and the second time point being later than the first time point; or, send at least the operation information to the second electronic device at a first time point, so as to control the second electronic device to render at least part of the content to be output corresponding to a second time point based on the operation information at or before the second time point, the first time point being the time point at which the first operation is obtained, and the second time point being later than the first time point; or, control a third electronic device to send the collected operation information to the second electronic device, so as to control the second electronic device to render at least part of the content to be output corresponding to a second time point based on the operation information at or before the second time point, the first time point being the time point at which the first operation is obtained, and the second time point being later than the first time point.

[0123] In some embodiments of the present application, the control module 602 is further configured to send a data request of the content to be output to the second electronic device, so as to control the second electronic device to feed back at least part of the content to be output rendered by the second electronic device; or, receive at least part of the content to be output fed back by the second electronic device.

[0124] In some embodiments of the present application, the control module 602 is further configured to determine a target object in the content to be output on which the first operation or the second operation acts, and render a content to be rendered corresponding to the target object at a corresponding time point.

[0125] In some embodiments of the present application, the display module 603 is further configured to synchronously display the content rendered by the second electronic device and the content generated based on the first operation in the virtual display area; or, display the content rendered by the second electronic device and the content generated based on the first operation in a first virtual display area and a second virtual display area respectively, wherein the first virtual display area and the second virtual display area are in the same plane or different planes.

[0126] In some embodiments of the present application, the display module 603 is further configured to display the content rendered by the second electronic device and the content generated based on the first operation in a first virtual display area and a first display area respectively, wherein the first display area is a display area driven by the second electronic device, and the first virtual display area and the first display area are in the same plane or different planes.

[0127] The embodiments of the present application further provide a control device, Figure 7A structural schematic diagram of a control device provided in an embodiment of the present application is shown in FIG. 7. As shown in FIG. 7, the control device 700 includes a memory 701 for storing executable control instructions, and a processor 702 for executing the executable control instructions stored in the memory to implement the method provided in the embodiment of the present application, for example, to implement the control method provided in the embodiment of the present application. Figure 7

[0128] In the embodiment of the present application, the processor 702 can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, or a microprocessor. It can be understood that, for different devices, the electronic device used to implement the function of the processor can also be other devices, and the embodiment of the present application does not make a specific limitation in this regard.

[0129] In the embodiment of the present application, the processor 702 is configured to obtain a first operation and operation information thereof, so that the display of at least one virtual display area displayed by the optical lens module of the first electronic device can be controlled based on the first operation; at least based on the operation information, control the second electronic device to render at least part of the content to be output; and present the content rendered by the second electronic device and the content generated based on the first operation in the virtual display area.

[0130] The embodiment of the present application also provides a computer readable storage medium storing executable control instructions for causing the processor 702 to execute the method provided in the embodiment of the present application, for example, the executable control method provided in the embodiment of the present application. For example, the control method instructions in the embodiment can be stored on a storage medium such as an optical disc, a hard disk, a U disk, etc. When the program instructions corresponding to a control method in the storage medium are read by an electronic device or executed, the control method as described in any of the above embodiments can be implemented.

[0131] ​Those skilled in the art will appreciate that embodiments of the application can be readily used on a variety of computer systems. Persons skilled in the relevant art will also recognize the interchangeability of various hardware and software components that can be used in a computer system depending on the particular implementation desired. Moreover, methods of the application can typically be performed on computing devices from a wide variety of vendors. Therefore, the drawings and descriptions are meant to be only exemplary, and are not intended to limit the scope of the application to any one type of computing device or system.

[0132] The present application is described in reference to the drawings, which are as follows: Figure 1 Figure 1

[0133] Figure 1 Figure 1

[0134] Figure 1 Figure 1

[0135] The above description is intended to be illustrative and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of the application should, therefore, be determined not with reference to the above description, but instead with reference to the appended claims, along with their full scope of equivalents.​​​​​​​​

Claims

1. A control method, comprising: obtaining a first operation and operation information thereof, the first operation being used to input a control instruction to a first electronic device to control at least display of at least one virtual display area displayed by an optical lens module of the first electronic device; the virtual display area displaying a foreground object and corresponding background environment data thereof, the first operation being an action of interacting with the foreground object and / or the background environment data; controlling a second electronic device to render at least part of to-be-output content based at least on the operation information, the second electronic device being a device that establishes a communication connection with the first electronic device; presenting the content rendered by the second electronic device and content generated based at least on the first operation in the virtual display area, wherein at least part of the content generated based at least on the first operation is generated by the first electronic device, and the content rendered by the second electronic device includes the background environment data or the background environment data and the content generated based on the first operation except the part generated by the first electronic device. 2.The method of claim 1, wherein the obtaining a first operation and operation information thereof comprises: obtaining device information of the first electronic device, and obtaining the first operation and operation information thereof based on the device information through a sensor in the first electronic device or a sensor in a third electronic device different from the first electronic device; or, obtaining a relative position relationship between the first electronic device and a target operation body, and obtaining the first operation and operation information thereof based on the relative position relationship through a sensor in the first electronic device or a sensor in a third electronic device different from the first electronic device; or, obtaining a use parameter of the first electronic device, and obtaining the first operation and operation information thereof based on the use parameter through a sensor in the first electronic device or a sensor in a third electronic device different from the first electronic device; or, obtaining identity information of an operation body of the first electronic device, and obtaining the first operation and operation information thereof based on the identity information through a sensor in the first electronic device or a sensor in a third electronic device different from the first electronic device. 3.The method of claim 1 or 2, wherein the controlling a second electronic device to render at least part of to-be-output content based at least on the operation information comprises: determining to-be-output content at a second time based on the operation information, the second time being later than a first time, and the first time being a time at which the first operation is obtained; controlling the second electronic device to render at least part of to-be-rendered content corresponding to the to-be-output content at the second time at a third time, the third time being not later than the second time.

4. The method of claim 3, wherein, The determining to-be-output content at a second time based on the operation information comprises: predicting a second operation at the second time based on the operation information; determining a first relationship between the second operation and the first operation, and a second relationship between the first operation and current output content; determining the to-be-output content at the second time based on the first relationship and the second relationship.

5. The method of claim 1 or 2, wherein the controlling the second electronic device to render at least part of the to-be-output content based on at least the operation information comprises: sending the to-be-rendered content determined based on the operation information to the second electronic device at a first time, to control the second electronic device to render at least part of the to-be-output content corresponding to a second time at or before the second time, the first time being a time when the first operation is obtained, and the second time being later than the first time; or sending at least the operation information to the second electronic device at a first time, to control the second electronic device to render at least part of the to-be-output content corresponding to a second time based on the operation information at or before the second time, the first time being a time when the first operation is obtained, and the second time being later than the first time; or controlling a third electronic device to send the collected operation information to the second electronic device, to control the second electronic device to render at least part of the to-be-output content corresponding to a second time based on the operation information at or before the second time, the first time being a time when the first operation is obtained, and the second time being later than the first time.

6. The method of claim 5, further comprising: sending a data request of the to-be-output content to the second electronic device, to control the second electronic device to feed back at least part of the to-be-output content rendered thereby; or receiving at least part of the to-be-output content actively fed back by the second electronic device.

7. The method of claim 4, further comprising: determining a target object in the to-be-output content on which the first operation or the second operation acts, and rendering a to-be-rendered object corresponding to the target object at a corresponding time.

8. The method of claim 1, 2, 6 or 7, wherein the corresponding presentation of the content rendered by the second electronic device and the content generated based on the first operation in the virtual display area comprises: synchronous display of the content rendered by the second electronic device and the content generated based on the first operation in the virtual display area; or separate display of the content rendered by the second electronic device and the content generated based on the first operation in a first virtual display area and a second virtual display area, wherein the first virtual display area and the second virtual display area are in the same plane or different planes.

9. The method of claim 1, 2, 6 or 7, wherein the corresponding presentation of the content rendered by the second electronic device and the content generated based on the first operation in the virtual display area comprises: corresponding display of the content rendered by the second electronic device and the content generated based on the first operation in a first virtual display area and a first display area, wherein the first display area is a display area driven by the second electronic device, and the first virtual display area and the first display area are in the same plane or different planes.

10. A control apparatus, comprising ​ ​ An obtaining module is configured to obtain a first operation and operation information of the first operation, the first operation being used to input a control instruction to a first electronic device to control at least display of at least one virtual display area displayed by an optical lens module of the first electronic device; the virtual display area displays a foreground object and corresponding background environment data of the foreground object, and the first operation is an action of interacting with the foreground object and / or the background environment data; A control module is configured to control a second electronic device to render at least part of to-be-output content based on at least the operation information, the second electronic device being a device that establishes a communication connection with the first electronic device; A display module is configured to present the content rendered by the second electronic device and content generated based on at least the first operation in the virtual display area, wherein at least part of the content generated based on at least the first operation is generated by the first electronic device; the content rendered by the second electronic device includes the background environment data, or the background environment data and the content generated based on the first operation except the part generated by the first electronic device.

Images