Equipment control method, system and device, equipment and storage medium

Abstract

The embodiment of the invention discloses an equipment control method and device, computer equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: displaying a first virtual object in a first virtual scene; under the condition that the pose of the first virtual object in the first virtual scene is changed, obtaining pose data after the first virtual object is changed; sending a first control instruction to at least one second device, wherein the first control instruction comprises the pose data; wherein the second device displays a second virtual scene, the second virtual scene comprises a second virtual object, the scene layout of the second virtual scene is the same as the scene layout of the first virtual scene, and the first control instruction is used for instructing the second device to control the position and orientation of the second virtual object in the second virtual scene according to the pose data. According to the invention, the first equipment can synchronously control the content displayed by the second equipment, and the equipment control accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a device control method, apparatus, computer device, and storage medium. Background Technology

[0002] Group control refers to the technology of centrally controlling and managing a group of slave devices through a master device.

[0003] In related technologies, the host screen of the host device and the slave screen of the slave device are projected onto the computer. Based on the ratio between the screen resolution of the host device and the screen resolution of the slave device, the position of the trigger operation on the host screen is determined and mapped to the position on the slave screen, thereby simulating the trigger operation executed at that position, so as to realize synchronous control of the slave device through the host device.

[0004] However, in the above method, due to the difference in screen resolution between the host device and the slave device, the position mapped based on the ratio between the screen resolutions is not accurate enough, resulting in low precision of the device control method. Summary of the Invention

[0005] This application provides a device control method, apparatus, computer equipment, and storage medium, which enables a first device to synchronously control the content displayed by a second device, thereby improving the accuracy of device control. The technical solution is as follows:

[0006] On one hand, a device control method is provided, executed by a first device, the method comprising:

[0007] Display the first virtual object located in the first virtual scene;

[0008] When the pose of the first virtual object changes in the first virtual scene, the pose data of the first virtual object after the change is obtained, and the pose data represents the position and orientation of the first virtual object in the first virtual scene;

[0009] Send a first control command to at least one second device, the first control command including the pose data, the first device being used to perform synchronous control of the at least one second device;

[0010] The second device displays a second virtual scene, which includes a second virtual object. The scene layout of the second virtual scene is the same as that of the first virtual scene. The first control command is used to instruct the second device to control the position and orientation of the second virtual object in the second virtual scene according to the pose data.

[0011] Optionally, determining the operation type of the triggering operation when a triggering operation on the target functional control is detected includes:

[0012] If a trigger operation on the target functional control is detected, the start position and end position of the trigger operation are determined.

[0013] If the distance between the start position and the end position of the operation is greater than a distance threshold, the operation type of the triggered operation is determined to be a sliding operation;

[0014] If the distance between the start position and the end position of the operation is not greater than the distance threshold, the operation type of the triggered operation is determined to be a click operation.

[0015] Optionally, the operation characteristics of the triggering operation include at least one of the following: operation duration, operation type, operation start position, or operation end position.

[0016] On the one hand, a device control method is provided, executed by a second device, the method comprising:

[0017] Display the second virtual object located in the second virtual scene;

[0018] The device receives a first control command sent by a first device, the first control command including pose data; the first device is used to synchronously control the second device, the first device displays a first virtual scene, the first virtual scene includes a first virtual object, the scene layout of the second virtual scene is the same as the scene layout of the first virtual scene, and the pose data represents the position and orientation of the first virtual object after changes in the first virtual scene;

[0019] In response to the first control command, the position and orientation of the second virtual object in the second virtual scene are controlled according to the pose data.

[0020] Optionally, the first control instruction further includes a target number of frames, which represents the number of frames between the k-th frame and the i-th frame. The k-th frame refers to the frame displayed by the first device when the control instruction was last received, and the i-th frame refers to the frame displayed by the first device when the first control instruction was received.

[0021] The step of responding to the first control command and controlling the position and orientation of the second virtual object in the second virtual scene according to the pose data includes:

[0022] In response to the first control command, within several screen frames of the target frame, the second virtual object is controlled to move in the second virtual scene to the position and orientation indicated by the pose data.

[0023] On the one hand, a device control system is provided, which includes a management device, a first device and at least one second device, wherein the first device is used to synchronously control the at least one second device, and the first device and the at least one second device are respectively connected to the management device.

[0024] The first device is configured to send a first control command to the management device when it detects a change in the pose of the first virtual object in the first virtual scene. The first control command includes pose data, which represents the position and orientation of the first virtual object after the change in the first virtual scene.

[0025] The management device is configured to receive the first control command and forward the first control command to the at least one second device.

[0026] The at least one second device is configured to respond to the first control command and control the position and orientation of the second virtual object in the second virtual scene according to the pose data, wherein the scene layout of the second virtual scene is the same as that of the first virtual scene.

[0027] On the other hand, a device control apparatus is provided, disposed in a first device, the apparatus comprising:

[0028] The display module is used to display the first virtual object located in the first virtual scene;

[0029] The first acquisition module is used to acquire the changed pose data of the first virtual object when the pose of the first virtual object in the first virtual scene changes, wherein the pose data represents the position and orientation of the first virtual object in the first virtual scene;

[0030] A first transmitting module is configured to transmit a first control command to at least one second device, the first control command including the pose data, and the first device is configured to perform synchronous control on the at least one second device.

[0031] The second device displays a second virtual scene, which includes a second virtual object. The scene layout of the second virtual scene is the same as that of the first virtual scene. The first control command is used to instruct the second device to control the position and orientation of the second virtual object in the second virtual scene according to the pose data.

[0032] Optionally, the first acquisition module is configured to:

[0033] If the change in pose data of the first virtual object between the i-th frame and the k-th frame is greater than the change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed. The k-th frame is the frame before the i-th frame, where i and k are positive integers.

[0034] The pose data of the first virtual object in the i-th frame is determined as the pose data of the first virtual object after the change.

[0035] Optionally, the pose data includes position parameters and orientation parameters; the first acquisition module is configured to perform any of the following:

[0036] The position change amount is obtained. If the position change amount is greater than the position change amount threshold, it is determined that the pose of the first virtual object in the first virtual scene has changed. The position change amount represents the difference between the position parameters of the i-th frame and the position parameters of the k-th frame.

[0037] If the orientation change amount is greater than the orientation change amount threshold, it is determined that the pose of the first virtual object in the first virtual scene has changed. The orientation change amount represents the difference between the orientation parameter of the i-th frame and the orientation parameter of the k-th frame.

[0038] The position change and the orientation change are obtained. If the position change is greater than the position change threshold and the orientation change is greater than the orientation change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed.

[0039] Optionally, the first acquisition module is configured to perform at least one of the following:

[0040] The kth frame refers to the frame preceding the i-th frame. If the change between the pose data of the i-th frame and the pose data of the k-th frame is greater than a first change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed.

[0041] The kth frame refers to the frame from which the control command was last sent. If the change between the pose data of the i-th frame and the pose data of the k-th frame is greater than a second change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed; wherein, the second change threshold is greater than the first change threshold.

[0042] Optionally, the first acquisition module is further configured to:

[0043] When the pose of the first virtual object in the first virtual scene changes, the target frame number is determined. The target frame number refers to the number of frames from the kth frame to the ith frame. The kth frame refers to the frame when the control command was last sent, and the ith frame refers to the frame when the first control command was sent.

[0044] The first control instruction further includes the target number of frames. The first control instruction is used to instruct the second device to control the position and orientation of the second virtual object in the second virtual scene according to the pose data within the target number of frames.

[0045] Optionally, the first device and the second device run a target application, the first virtual scene and the second virtual scene are virtual scenes in the target application, and the target application includes a pose control plugin; the first acquisition module is used for:

[0046] The pose control plugin detects the pose of the first virtual object in the first virtual scene. When the pose control plugin detects a change in the pose of the first virtual object, it acquires the pose data of the first virtual object after the change.

[0047] Optionally, the first virtual scene displays multiple functional controls; the device further includes:

[0048] The second acquisition module is used to acquire control trigger data when a trigger operation on a target functional control is detected. The control trigger data includes the control characteristics of the target functional control and the operation characteristics of the trigger operation. The target functional control is any one of the plurality of functional controls.

[0049] The second sending module is used to send a second control command to the at least one second device, the second control command including the control trigger data;

[0050] The second virtual scene in the second device displays the plurality of functional controls, and the second control instruction is used to instruct the second device to simulate the triggering operation of the target functional control according to the control trigger data.

[0051] Optionally, the second acquisition module is used for:

[0052] If a trigger operation on the target functional control is detected, the operation type of the trigger operation is determined;

[0053] If the operation type of the triggered operation does not belong to the preset operation type, the control trigger data is obtained. The preset operation type refers to the operation type that does not need to be synchronized with the at least one second device.

[0054] Optionally, the second acquisition module is used for:

[0055] If a trigger operation on the target functional control is detected, the start position and end position of the trigger operation are determined.

[0056] If the distance between the start position and the end position of the operation is greater than a distance threshold, the operation type of the triggered operation is determined to be a sliding operation;

[0057] If the distance between the start position and the end position of the operation is not greater than the distance threshold, the operation type of the triggered operation is determined to be a click operation.

[0058] Optionally, the first device and the second device run a target application, and the plurality of functional controls are provided by the target application; the second acquisition module is used for:

[0059] Upon detecting the trigger operation, determine the coordinates of the start position of the trigger operation in the interface coordinate system of the target application;

[0060] Obtain the coordinates of the plurality of functional controls in the interface coordinate system of the target application, wherein the coordinates of the functional controls represent the display position of the functional controls on the interface of the target application;

[0061] The function control that matches the coordinates of the operation start position is identified as the target function control to be triggered.

[0062] Optionally, the operation characteristics of the triggering operation include at least one of the following: operation duration, operation type, operation start position, or operation end position.

[0063] Optionally, the first device and the second device run a target application, the first virtual scene and the second virtual scene are virtual scenes in the target application, and the target application includes a control plugin; the second acquisition module is used to:

[0064] The control plugin detects the multiple functional controls on the first virtual scene, and when the control plugin detects a trigger operation on the target functional control, it acquires the control trigger data.

[0065] On the other hand, a device control apparatus is provided, configured in a second device, the apparatus comprising:

[0066] The display module is used to display the second virtual object located in the second virtual scene;

[0067] A first receiving module is used to receive a first control command sent by a first device, the first control command including pose data; the first device is used to perform synchronous control on the second device, the first device displays a first virtual scene, the first virtual scene includes a first virtual object, the scene layout of the second virtual scene is the same as the scene layout of the first virtual scene, and the pose data represents the position and orientation of the first virtual object after changes in the first virtual scene;

[0068] The first control module is used to respond to the first control command and control the position and orientation of the second virtual object in the second virtual scene according to the pose data.

[0069] Optionally, the first control instruction further includes a target number of frames, which represents the number of frames between the k-th frame and the i-th frame. The k-th frame refers to the frame displayed by the first device when the control instruction was last received, and the i-th frame refers to the frame displayed by the first device when the first control instruction was received.

[0070] The first control module is used for:

[0071] In response to the first control command, within several screen frames of the target frame, the second virtual object is controlled to move in the second virtual scene to the position and orientation indicated by the pose data.

[0072] Optionally, the first device and the second device run a target application, the first virtual scene and the second virtual scene are virtual scenes in the target application, and the target application includes a pose control plugin;

[0073] The first control module is used for:

[0074] In response to the first control command, the position and orientation of the second virtual object in the second virtual scene are controlled by the pose control plugin according to the pose data.

[0075] Optionally, the second virtual scene displays multiple functional controls; the device further includes:

[0076] The second receiving module is used to receive a second control instruction sent by the first device. The second control instruction includes control trigger data, which represents a trigger operation detected by the first device on a target functional control. The target functional control is any one of the plurality of functional controls. The control trigger data includes the control characteristics of the target functional control and the operation characteristics of the trigger operation.

[0077] The second control module is used to respond to the second control command and simulate the triggering operation of the target function control according to the control triggering data.

[0078] Optionally, the first device and the second device run a target application, the first virtual scene and the second virtual scene are virtual scenes in the target application, and the target application includes a control plugin;

[0079] The second control module is used for:

[0080] In response to the second control command, the control plugin simulates the triggering operation of the target function control according to the control trigger data.

[0081] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to perform the operations performed by the device control method as described above.

[0082] On the other hand, a computer-readable storage medium is provided that stores at least one computer program, which is loaded and executed by a processor to perform the operations performed by the device control method as described above.

[0083] On the other hand, a computer program product is provided, including a computer program loaded and executed by a processor to perform the operations performed by the device control method as described above.

[0084] The solution provided in this application embodiment has the same scene layout for the first virtual scene displayed by the first device and the second virtual scene displayed by the second device. After the first device controls the first virtual object to move in the first virtual scene, it synchronizes the changed pose data of the first virtual object to the second device. Since the scene layout is the same, the position and orientation indicated by the same pose data in the first and second virtual scenes are consistent. Therefore, the second device controls the position and orientation of the second virtual object in the second virtual scene according to the pose data, ensuring that the second virtual object remains synchronized with the first virtual object without deviation. This achieves synchronous control of the content displayed by the second device by the first device, improving the accuracy of device control. Attached Figure Description

[0085] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1 This is a schematic diagram of a device control system provided in an embodiment of this application;

[0087] Figure 2 This is a flowchart of a device control method provided in an embodiment of this application;

[0088] Figure 3 This is a flowchart of another device control method provided in an embodiment of this application;

[0089] Figure 4 This is a flowchart of another device control method provided in an embodiment of this application;

[0090] Figure 5 This is a flowchart of another device control method provided in an embodiment of this application;

[0091] Figure 6 This is a flowchart of another device control method provided in an embodiment of this application;

[0092] Figure 7 This is a flowchart of another device control method provided in an embodiment of this application;

[0093] Figure 8 This is a flowchart of another device control method provided in an embodiment of this application;

[0094] Figure 9 This is a flowchart of another device control method provided in an embodiment of this application;

[0095] Figure 10 This is an architecture diagram of a target application provided in an embodiment of this application;

[0096] Figure 11 This is a flowchart of another device control method provided in an embodiment of this application;

[0097] Figure 12 This is a schematic diagram of a control and management interface provided in an embodiment of this application;

[0098] Figure 13 This is a flowchart of another device control method provided in an embodiment of this application;

[0099] Figure 14 This is a schematic diagram of the structure of a device control apparatus provided in an embodiment of this application;

[0100] Figure 15 This is a schematic diagram of another device control device provided in an embodiment of this application;

[0101] Figure 16 This is a schematic diagram of another device control device provided in an embodiment of this application;

[0102] Figure 17 This is a schematic diagram of another device control device provided in an embodiment of this application;

[0103] Figure 18 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0104] Figure 19 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0105] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0106] It is understood that the terms "first," "second," etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, a first device may be referred to as a second device, and similarly, a second device may be referred to as a first device.

[0107] "At least one" refers to one or more devices. For example, at least one device can be one device, two devices, three devices, or any integer number of devices greater than or equal to one. "Multiple" refers to two or more devices. For example, multiple devices can be two devices, three devices, or any integer number of devices greater than or equal to two. "Each" refers to each of the at least one devices. For example, each device refers to each of the multiple devices. If the multiple devices are three devices, then each device refers to each of the three devices.

[0108] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between the user terminal and other devices) involved in this application have been fully authorized by the user or relevant parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the virtual objects, pose data, and control trigger data involved in this application were all obtained with full authorization.

[0109] For ease of understanding, the terms used in the embodiments of this application will be explained below.

[0110] Virtual scene: A virtual scene displayed (or provided) when an application runs on a terminal. This virtual scene can be a simulation of the real world, a semi-simulated / semi-fictional environment, or a purely fictional environment. A virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene; this application embodiment does not limit the dimension of the virtual scene. For example, a virtual scene may include the sky, land, ocean, etc., and the land may include environmental elements such as deserts and cities. Users can control virtual objects to move within this virtual scene. Optionally, the virtual scene can provide a battle environment for virtual objects, containing virtual resources available for use by the virtual objects. These virtual resources include virtual items needed for battle, virtual medicines needed for treatment, virtual items needed for upgrades, and virtual coins needed for trading.

[0111] Virtual objects refer to movable objects in a virtual scene. These movable objects can be virtual characters, virtual animals, virtual sprites, anime characters, etc. A virtual object can be a virtual avatar representing the user within the virtual scene. A virtual scene can include multiple virtual objects, each with its own shape and volume, occupying a portion of the virtual scene's space. Optionally, when the virtual scene is a three-dimensional virtual scene, the virtual object can be a three-dimensional model. This three-dimensional model can be a three-dimensional character constructed based on three-dimensional human skeleton technology. The same virtual object can display different appearances by wearing different skins. Optionally, virtual objects can also be implemented using 2.5D or 2D models; this application embodiment does not limit this.

[0112] UE games refer to games developed using Unreal Engine. Unreal Engine provides a wealth of tools and resources, including advanced graphics rendering technology, a powerful physics simulation system, and a user-friendly development interface, enabling developers to easily create high-quality games and applications. Unreal Engine is renowned worldwide for its high-quality rendering effects and powerful physics engine. It employs advanced technologies such as global illumination and real-time dynamic shadows to create incredibly realistic game visuals and effects.

[0113] UMG (Unreal Motion Graphics UIDesigner): UMG is a user interface design tool based on Unreal Engine that allows developers to visually create user interfaces in games. The main components of UMG are UI (User Interface) controls, which can be programmed using Blueprints or C++ (a programming language).

[0114] The device control method provided in this application can be used in computer devices. Optionally, the computer device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited to these. The embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0115] Figure 1 This is a schematic diagram of a device control system provided in an embodiment of this application. See also... Figure 1 The equipment control system includes: a first device 101 and at least one second device 102. Figure 1 (Taking three devices as an example) and management device 103. The first device 101 and at least one second device 102 are respectively connected to the management device 103 via a communication connection, which can be made via a wireless or wired network.

[0116] In this embodiment, a first device 101 provides a first virtual scene, which includes a first virtual object. A second device 102 provides a second virtual scene, which includes a second virtual object. The scene layout of the first virtual scene and the scene layout of the second virtual scene are the same. When the pose of the first virtual object in the first virtual scene changes, the first device 101 sends a control command including the pose data of the first virtual object to a management device 103. The management device 103 forwards the control command to the second device 102. The second device 102 controls the pose of the second virtual object in the second virtual scene according to the pose data, thereby realizing the synchronous control of at least one second device 102 by the first device 101.

[0117] In one possible implementation, the first device 101, the second device 102, and the management device 103 include, but are not limited to, smartphones, tablets, laptops, desktop computers, smart voice interaction devices, smart home appliances, in-vehicle terminals, aircraft, VR (Virtual Reality) devices, AR (Augmented Reality) devices, etc.

[0118] In one possible implementation, the first device 101 and at least one second device 102 each run a target application, which can be a game application, a social application, an online payment application, an online shopping application, a medical service application, a video application, or the like. When a device runs a target application, the user interface of the target application is displayed on the device's screen.

[0119] Optionally, the target application includes a rendering engine, such as Unity3D or Unreal Engine (UE). The target application provides virtual scenes and virtual objects, and the rendering engine can render the virtual objects to different positions and orientations in the virtual scene.

[0120] It should be noted that the above-described equipment control system is only an example. The method provided in this application embodiment can also be executed by the first device 101 and the second device 102. That is, the first device 101 directly sends the control command to the second device 102. This application embodiment does not limit this.

[0121] Figure 2 This is a flowchart of a device control method provided in an embodiment of this application. This embodiment is executed by a first device. See also... Figure 2 The method includes:

[0122] 201. The first device displays the first virtual object located in the first virtual scene.

[0123] The group control scheme in this application involves a first device and at least one second device. Group control refers to a technology that centrally controls and manages a group of slave devices through a master device. A master device is a device that controls and manages slave devices, and a slave device is a device controlled and managed by the master device. Specifically, the first device is the master device, and the at least one second device is a slave device; the first device is used to control the at least one second device.

[0124] The first device provides a first virtual scene, which includes a first virtual object. The first device displays the first virtual object located in the first virtual scene. In one possible implementation, the first virtual object is a virtual object controlled by an account logged into the first device.

[0125] 202. When the pose of the first virtual object changes in the first virtual scene, the first device acquires the pose data of the first virtual object after the change. The pose data represents the position and orientation of the first virtual object in the first virtual scene.

[0126] The first device detects whether the pose of the first virtual object in the first virtual scene has changed. The pose of the first virtual object refers to its position and orientation; that is, it detects whether the position and orientation of the first virtual object in the first virtual scene have changed. If the first device detects a change in the pose of the first virtual object, it acquires the changed pose data of the first virtual object.

[0127] In one possible implementation, the pose data includes position parameters and orientation parameters. The position parameters represent the position of the first virtual object in the first virtual scene; for example, the position parameters can be the coordinates of the first virtual object in the world coordinate system of the first virtual scene. The orientation parameters represent the orientation of the first virtual object in the first virtual scene; for example, the orientation parameters can include the pitch angle, yaw angle, and roll angle of the first virtual object in the first virtual scene.

[0128] 203. The first device sends a first control command to at least one second device, the first control command including pose data, and the first device is used to perform synchronous control on at least one second device.

[0129] The second device displays a second virtual scene, which includes a second virtual object. The layout of the second virtual scene is the same as that of the first virtual scene. The second virtual scene in the second device and the first virtual scene in the first device can be the same virtual scene or different virtual scenes. Similarly, the second virtual scenes in different second devices can be the same virtual scene or different virtual scenes.

[0130] After acquiring the pose data of the first virtual object, the first device generates a first control instruction including the pose data and sends the first control instruction to at least one second device. The first control instruction is used to instruct the second device to control the position and orientation of the second virtual object in the second virtual scene according to the pose data, so that the pose of the second virtual object in the second device is consistent with the pose of the first virtual object in the first device, thereby realizing synchronous control of at least one second device through the first device.

[0131] The method provided in this application embodiment has the same scene layout for a first virtual scene displayed by a first device and a second virtual scene displayed by a second device. After the first device controls a first virtual object to move in the first virtual scene, it synchronizes the changed pose data of the first virtual object to the second device. Since the scene layout is the same, the position and orientation indicated by the same pose data in the first and second virtual scenes are consistent. Therefore, the second device controls the position and orientation of the second virtual object in the second virtual scene based on the pose data, ensuring that the second virtual object remains synchronized with the first virtual object without deviation. This achieves synchronous control of the content displayed by the second device by the first device, improving the accuracy of device control.

[0132] Figure 3 This is a flowchart of another device control method provided in this application embodiment. This application embodiment is executed by a second device. See also... Figure 3 The method includes:

[0133] 301. The second device displays the second virtual object located in the second virtual scene.

[0134] The second device provides a second virtual scene, which includes a second virtual object, and the second device displays the second virtual object located in the second virtual scene. In one possible implementation, the second virtual object is a virtual object controlled by the account logged into the second device.

[0135] 302. The second device receives a first control command sent by the first device, the first control command including pose data.

[0136] The first device is used to synchronously control the second device. The first device displays a first virtual scene, which includes a first virtual object. The scene layout of the second virtual scene is the same as that of the first virtual scene.

[0137] The second device receives a first control command including pose data, which represents the changed position and orientation of the first virtual object in the first virtual scene.

[0138] 303. The second device responds to the first control command and controls the position and orientation of the second virtual object in the second virtual scene according to the pose data.

[0139] Since the pose data is used to indicate the position and orientation of the first virtual object in the first virtual scene, and the scene layout of the second virtual scene is the same as that of the first virtual scene, the second device can adjust the position and orientation of the second virtual object in the second virtual scene according to the pose data, so that the pose of the second virtual object in the second virtual scene is consistent with the pose of the first virtual object in the first virtual scene, thereby realizing synchronous control of the second device by the first device.

[0140] The method provided in this application embodiment has the same scene layout for a first virtual scene displayed by a first device and a second virtual scene displayed by a second device. After the first device controls a first virtual object to move in the first virtual scene, it synchronizes the changed pose data of the first virtual object to the second device. Since the scene layout is the same, the position and orientation indicated by the same pose data in the first and second virtual scenes are consistent. Therefore, the second device controls the position and orientation of the second virtual object in the second virtual scene based on the pose data, ensuring that the second virtual object remains synchronized with the first virtual object without deviation. This achieves synchronous control of the content displayed by the second device by the first device, improving the accuracy of device control.

[0141] The above Figure 2 and Figure 3 The diagram shown is only the basic process of this application. The following is a further explanation of the solution provided in this application based on a specific implementation method. Figure 4 This is a flowchart of another device control method provided in this application embodiment. This application embodiment is executed by a first device and a second device. See also... Figure 4 The method includes:

[0142] 401. The first device displays the first virtual object located in the first virtual scene.

[0143] The first device provides a first virtual scene, which includes a first virtual object. The first device displays the first virtual object located in the first virtual scene. The first virtual object is capable of movement within the first virtual scene, such as walking, climbing, or running.

[0144] In one possible implementation, the first device runs a target application, such as a game application. This target application includes a rendering engine, such as UE (User Experience). The first virtual scene is provided by the target application, and the first virtual object is a pawn (a manipulated object) in the rendering engine, which can be a character or a vehicle, etc. The pawn can perform corresponding actions based on input from the player or AI (Artificial Intelligence). Optionally, the rendering engine provides a PlayerController (object controller) to process input from the player or AI and translate it into actions performed by the pawn. The PlayerController can also control the camera viewpoint in the first virtual scene.

[0145] 402. The second device displays the second virtual object located in the second virtual scene.

[0146] The second device provides a second virtual scene, which includes a second virtual object. The second device displays the second virtual object located in the second virtual scene. The second virtual object is capable of movement within the second virtual scene, such as walking, climbing, or running.

[0147] It should be noted that the scene layout of the second virtual scene is the same as that of the first virtual scene. This can be understood as the world coordinate system of the first virtual scene being the same as that of the second virtual scene. The first and second virtual scenes can be the same virtual scene, or they can be two different virtual scenes.

[0148] In one possible implementation, the second device runs a target application that includes a rendering engine. The second virtual scene is provided by the target application, and the second virtual object is a Pawn in the rendering engine. Optionally, the rendering engine provides a PlayerController. Optionally, the target application running on the second device is the same as the target application running on the first device.

[0149] 403. When the pose of the first virtual object changes in the first virtual scene, the first device acquires the pose data of the first virtual object after the change, and the pose data represents the position and orientation of the first virtual object in the first virtual scene.

[0150] The first device detects whether the pose of the first virtual object in the first virtual scene has changed. If the first device detects that the pose of the first virtual object has changed, it acquires the pose data of the first virtual object after the change.

[0151] In one possible implementation, if the change in pose data of the first virtual object between the i-th frame and the k-th frame exceeds a change threshold, the first device determines that the pose of the first virtual object in the first virtual scene has changed. The k-th frame refers to the frame preceding the i-th frame, where i and k are positive integers. The pose data of the first virtual object in the i-th frame is then determined as the changed pose data of the first virtual object.

[0152] The i-th frame can be understood as the current frame, and the k-th frame can be understood as a historical frame. The pose data of the i-th frame is the current pose data, i.e., the latest pose data, while the pose data of the k-th frame is the historical pose data. The first device compares the current pose data with the historical pose data. If the change between the two is greater than a change threshold, it indicates that the difference between the current pose and the historical pose of the first virtual object is large, meaning that the pose of the first virtual object has changed. The change threshold is a pre-set threshold.

[0153] For example, the k-th frame can be a frame preceding the i-th frame and adjacent to the i-th frame, meaning the k-th frame is the frame preceding the i-th frame, and i equals k plus 1. For example, the k-th frame can be a frame preceding the i-th frame and not adjacent to the i-th frame, meaning i is greater than k plus 1.

[0154] For example, each time the first device renders a frame, it acquires the pose data of the first virtual object in that frame, compares the currently acquired pose data with the historical pose data, and determines whether the pose of the first virtual object has changed.

[0155] In this implementation, the change in pose data of the current frame is compared with the change threshold between the current frame's pose data and historical frame pose data to determine whether the pose of the first virtual object has changed. This avoids misjudgments caused by slight jitter and improves accuracy. Furthermore, the second device is only controlled synchronously when the pose of the first virtual object changes significantly, reducing the response frequency to small movements and thus lowering computational resource consumption and system load.

[0156] Optionally, the pose data includes position parameters and orientation parameters. Therefore, step 403 can be implemented in any of the following three ways.

[0157] (1) Obtain the position change amount. If the position change amount is greater than the position change amount threshold, it is determined that the pose of the first virtual object in the first virtual scene has changed. The position change amount represents the difference between the position parameters of the i-th frame and the position parameters of the k-th frame.

[0158] The position parameter can be the coordinates of the first virtual object in the world coordinate system of the first virtual scene. For example, the position parameter is (x, y, z), where x represents the X-axis coordinate in the world coordinate system, y represents the Y-axis coordinate in the world coordinate system, and z represents the Z-axis coordinate in the world coordinate system.

[0159] The first device compares the position change with a preset position change threshold. If the position change is greater than the position change threshold, it means that the position of the first virtual object in the first virtual scene has changed. When the position of the first virtual object changes, it is considered that the pose of the first virtual object has changed.

[0160] (2) Obtain the orientation change amount. If the orientation change amount is greater than the orientation change amount threshold, it is determined that the pose of the first virtual object in the first virtual scene has changed. The orientation change amount represents the difference between the orientation parameter of the i-th frame and the orientation parameter of the k-th frame.

[0161] The orientation parameters can include the pitch angle, yaw angle, and roll angle of the first virtual object in the first virtual scene. For example, the orientation parameters are (Pitch, Yaw, Roll), where Pitch represents the pitch angle, Yaw represents the yaw angle, and Roll represents the roll angle.

[0162] The first device compares the amount of orientation change with a preset orientation change threshold. If the amount of orientation change is greater than the orientation change threshold, it indicates that the orientation of the first virtual object in the first virtual scene has changed. When the orientation of the first virtual object changes, it is considered that the pose of the first virtual object has changed.

[0163] (3) Obtain the position change and orientation change. If the position change is greater than the position change threshold and the orientation change is greater than the orientation change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed.

[0164] The first device compares the position change with a preset position change threshold and the orientation change with a preset orientation change threshold. If the position change is greater than the preset position change threshold and the orientation change is greater than the orientation change threshold, it indicates that the position and orientation of the first virtual object in the first virtual scene have changed. When both the position and orientation of the first virtual object have changed, it is considered that the pose of the first virtual object has changed.

[0165] This implementation, by separately detecting changes in position and orientation and setting thresholds for each, allows for a more detailed assessment of pose changes in virtual objects. This helps distinguish between minor and significant changes in position or orientation, thus improving accuracy. Furthermore, it supports the detection of individual position or orientation changes, as well as simultaneous detection of both, adapting to different usage scenarios and application requirements.

[0166] Optionally, the k-th frame can be the frame preceding the i-th frame, or the frame from the last time a control command was sent. Therefore, step 403 includes at least one of the following two implementation methods.

[0167] (1) The kth frame refers to the frame before the ith frame. If the change between the pose data of the ith frame and the pose data of the kth frame is greater than the first change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed.

[0168] If the k-th frame is the previous frame of the i-th frame, then the change between the pose data of the i-th frame and the pose data of the k-th frame is compared with a first change threshold.

[0169] (2) The kth frame refers to the frame when the control command was sent last time. If the change between the pose data of the i-th frame and the pose data of the kth frame is greater than the second change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed; wherein, the second change threshold is greater than the first change threshold.

[0170] If the k-th frame is the frame from which the control command was last sent, then the change between the pose data of the i-th frame and the pose data of the k-th frame is compared with a second change threshold. Here, "last sent control command" refers to the control command, including pose data, sent to at least one second device when a change in the pose of the first virtual object was detected.

[0171] In this implementation, the current pose data can be compared with the pose data of the previous frame. This helps to detect pose changes of virtual objects in a timely manner and improves the real-time performance of device synchronization control. Alternatively, the current pose data can be compared with the pose data at the time the control command was last sent. This avoids situations where small changes occur in each frame, leading to prolonged periods without detecting pose changes.

[0172] In one possible implementation, the first device runs a target application, and the first virtual scene is a virtual scene within the target application. The target application includes a pose control plugin. The first device detects the pose of a first virtual object in the first virtual scene through the pose control plugin. When the pose control plugin detects a change in the pose of the first virtual object, it acquires the changed pose data of the first virtual object.

[0173] The pose control plugin is a built-in plugin in the target application. This plugin can be used to detect whether the pose of a first virtual object in the target application has changed. The target application running on the first device can invoke the pose control plugin to detect the pose of the first virtual object and acquire pose data.

[0174] For example, the target application is a game application, which includes a rendering engine. The pose control plugin is a built-in plugin within the rendering engine. It can be understood as integrating pose detection functionality into the rendering engine as a plugin, and packaging it into the target application through the rendering engine, thereby extending the pose detection functionality of the target application. Therefore, any application that includes this rendering engine can use this pose control plugin to implement the device control scheme executed by the first device.

[0175] In this implementation, a pose control plugin is built into the target application. This plugin detects and controls the pose of virtual objects within the target application, enabling rapid and accurate acquisition of pose data and ensuring the real-time nature of the acquired pose data. Furthermore, it expands the functionality of the target application, allowing for device synchronization control methods to be implemented in any type of target application, adapting to different usage scenarios and application requirements.

[0176] 404. The first device sends a first control command to at least one second device, the first control command including pose data.

[0177] The first device is used to synchronously control at least one second device. After acquiring the pose data of the first virtual object, the first device generates a first control command including the pose data and sends the first control command to at least one second device. The first control command is used to instruct the second device to control the position and orientation of the second virtual object in the second virtual scene according to the pose data.

[0178] In one possible implementation, in step 403 above, when the pose of the first virtual object in the first virtual scene changes, the first device further determines a target frame number. The target frame number refers to the number of frames between the k-th frame and the i-th frame, where the k-th frame refers to the frame from which the control command was last sent, and the i-th frame refers to the frame from which the first control command was sent. Then, the first control command also includes the target frame number, and the first control command instructs the second device to control the position and orientation of the second virtual object in the second virtual scene according to the pose data within the target frame number of frames.

[0179] In this embodiment of the application, the number of frames between the kth frame when the control command was last sent and the ith frame when the first control command was sent this time is the target frame number. It can be understood that the pose of the first virtual object has changed this time after the target frame number of frames has passed. That is, within the target frame number, the first virtual object changes from the previous pose to the current pose.

[0180] To maintain consistency between the second virtual object and the first virtual object, the second device also adjusts the position and orientation of the second virtual object in the second virtual scene to match the position and orientation indicated by the pose data within several frames of the target frame. Therefore, it can be understood that the second virtual object, at its current position and orientation, moves a certain distance along the movement path from its current position and orientation to the pose and orientation indicated by the pose data, with each frame refreshed, until several frames from the target frame, the second virtual object moves to the position and orientation indicated by the pose data.

[0181] For example, the current pose is denoted as pose A, and the pose indicated by the pose data is denoted as pose data indication B. Based on the target frame number, the average interpolation is performed on the movement path from pose A to pose B to obtain the target frame number - 2 poses. Starting from pose A, every time a frame is refreshed, the second device displays the second virtual object as the next pose of the current pose, thus forming the effect that the second virtual object gradually changes from the current pose A to pose B within several target frame frames.

[0182] In this implementation, the first virtual object of the first device undergoes pose changes within a target number of frames, and the second virtual object of the second device also undergoes pose changes within the target number of frames. This ensures that the pose change process of the second virtual object is consistent with that of the first virtual object, thereby improving the coordination effect of the synchronous control devices. Furthermore, gradually adjusting the pose of the second virtual object within a specified number of frames avoids sudden and drastic changes, resulting in smoother changes in the position and orientation of the virtual object and improving the control effect.

[0183] Figure 5 This is a flowchart of another device control method provided in an embodiment of this application, such as... Figure 5 As shown, the first device begins pose group control, which refers to synchronously controlling at least one second device by sending pose data. The first device identifies a first virtual object under its control and determines whether the pose of the first virtual object has changed. If the pose of the first virtual object has changed, the first device synchronizes the pose data of the first virtual object to at least one second device. The first device determines whether to end the pose group control. If not, it repeats the above steps until the pose group control ends.

[0184] 405. The second device receives a first control command sent by the first device, the first control command including pose data.

[0185] The second device receives a first control command including pose data, which represents the changed position and orientation of the first virtual object in the first virtual scene.

[0186] 406. The second device responds to the first control command and controls the position and orientation of the second virtual object in the second virtual scene according to the pose data.

[0187] Since the pose data is used to indicate the position and orientation of the first virtual object in the first virtual scene, and the scene layout of the second virtual scene is the same as that of the first virtual scene, the second device can adjust the position and orientation of the second virtual object in the second virtual scene according to the pose data, so that the pose of the second virtual object in the second virtual scene is consistent with the pose of the first virtual object in the first virtual scene.

[0188] In one possible implementation, as described in step 404 above, the first control command further includes a target frame number, which represents the number of frames from the k-th frame to the i-th frame. The k-th frame refers to the frame displayed by the first device when the control command was last received, and the i-th frame refers to the frame displayed by the first device when the first control command was received. Then, in response to the first control command, the second device controls the second virtual object to move within the target frame number to the position and orientation indicated by the pose data in the second virtual scene.

[0189] In this implementation, the first virtual object of the first device undergoes pose changes within a target number of frames, and the second virtual object of the second device also undergoes pose changes within the target number of frames. This ensures that the pose change process of the second virtual object is consistent with that of the first virtual object, thereby improving the coordination effect of the synchronous control devices. Furthermore, gradually adjusting the pose of the second virtual object within a specified number of frames avoids sudden and drastic changes, resulting in smoother changes in the position and orientation of the virtual object and improving the control effect.

[0190] In one possible implementation, the second device runs a target application, and the second virtual scene is a virtual scene within the target application. The target application includes a pose control plugin. Responding to a first control command, the second device, through the pose control plugin, controls the position and orientation of a second virtual object within the second virtual scene according to pose data.

[0191] The pose control plugin is a built-in plugin in the target application, which can be used to control the pose of a second virtual object in the target application. The target application running on the second device can call the pose control plugin to control the pose of the second virtual object according to the pose data.

[0192] For example, the target application is a game application, which includes a rendering engine. The pose control plugin is a built-in plugin within the rendering engine. It can be understood as integrating pose control functionality into the rendering engine as a plugin, and packaging it into the target application through the rendering engine, thereby extending the pose control functionality of the target application. Therefore, any application that includes this rendering engine can use this pose control plugin to implement the device control scheme executed by the second device.

[0193] For example, the pose control plugin mentioned in step 406 and the pose control plugin mentioned in step 403 can be the same pose control plugin, which has the function of detecting and controlling the pose of virtual objects. In the group control scheme, the pose control plugin can be called in the host device such as the first device to detect the pose of virtual objects, and the pose control plugin can be called in the slave device such as the second device to control the pose of virtual objects.

[0194] In this implementation, a pose control plugin is built into the target application. This plugin detects and controls the pose of virtual objects within the application, enabling fast and precise control and ensuring real-time performance. Furthermore, it expands the functionality of the target application, allowing for device synchronization control in any type of application, adapting to diverse usage scenarios and application requirements.

[0195] Figure 6This is a flowchart of another device control method provided in an embodiment of this application, such as... Figure 6 As shown, the second device begins receiving pose group control data from the first device. The second device identifies the second virtual object it controls and the object controller of the second virtual object. The second device determines whether it has received pose data synchronized from the first device. If pose data is received, it invokes the object controller of the second virtual object and controls the position and orientation of the second virtual object according to the pose data. The second device determines whether to end the pose group control. If not, it repeats the above steps until the pose group control ends.

[0196] The method provided in this application embodiment has the same scene layout for a first virtual scene displayed by a first device and a second virtual scene displayed by a second device. After the first device controls a first virtual object to move in the first virtual scene, it synchronizes the changed pose data of the first virtual object to the second device. Since the scene layout is the same, the position and orientation indicated by the same pose data in the first and second virtual scenes are consistent. Therefore, the second device controls the position and orientation of the second virtual object in the second virtual scene based on the pose data, ensuring that the second virtual object remains synchronized with the first virtual object without deviation. This achieves synchronous control of the content displayed by the second device by the first device, improving the accuracy of device control.

[0197] It should be noted that the above embodiment is only illustrated by sending a first control command to one second device. In another embodiment, the first device sends a first control command to multiple second devices, and each second device executes steps 402, 405, and 406 as described above.

[0198] It should be noted that steps 401-406 are executed instantaneously, meaning that the next step can be executed as soon as one step is completed, thus enabling real-time control of the second device by the first device. In another embodiment, steps 401-406 are not executed instantaneously. After the first device executes step 401, whenever the pose of the first virtual object changes, the changed pose data is stored, i.e., step 403 is repeated. After the display of the first virtual object ends, the first device stores multiple pose data of the gradually changing first virtual object. The first device sends control commands including multiple pose data to the second device, and the second device controls the second virtual object sequentially according to the multiple pose data, reproducing the effect of the second virtual object moving according to the first virtual object, thus enabling non-real-time control of the second device by the first device.

[0199] The above embodiments illustrate the process by which the first device controls the second device through synchronized pose data. In addition, the first device can also control the second device by triggering data through synchronized controls. For details, please refer to the following... Figure 7 Examples of implementations. Figure 7 This is a flowchart of another device control method provided in this application embodiment. This application embodiment is executed by a first device and a second device. See also... Figure 7 The method includes:

[0200] 701. The first device displays a first virtual object located in a first virtual scene, and multiple functional controls are displayed on the first virtual scene.

[0201] In this embodiment of the application, the first virtual scene displays multiple functional controls, which can be UI controls. UI controls are the basic elements that constitute the user interface, such as buttons, text boxes, sliders, etc.

[0202] In one possible implementation, the first device runs a target application, such as a game application. This target application includes a rendering engine, such as UX / UI. The multiple functional controls are provided by the target application.

[0203] For example, in the rendering engine, these multiple functional controls can be either SWidget-based controls or UMG-based UWidget controls. Slate is a C++-based UI framework for creating highly customizable game interfaces. Slate provides many predefined UI controls and layouts that developers can use to create complex interactive interfaces. SWidget controls are a fundamental component of Slate, representing a visual UI control. The SWidget class provides a set of common properties and methods for handling control layout, rendering, and input events. UWidget controls are a fundamental component of UMG, representing a visual UI control. The UWidget class inherits from the SWidget class, providing developers with UI controls for use in UMG. UserWidget is a special type of UWidget in UMG, allowing developers to add custom logic and functionality. Typically, developers create subclasses of UserWidget to add custom UI controls, event handling, and data binding features.

[0204] For example, in a rendering engine, developers can use collision detection to lay out functional controls. Collision detection detects collisions or overlaps between functional controls. By performing collision detection on functional controls, it can be ensured that the layout between functional controls is correct and that there is no overlap or occlusion.

[0205] 702. The second device displays a second virtual object located in a second virtual scene, and multiple functional controls are displayed on the second virtual scene.

[0206] In one possible implementation, the first device runs a target application, such as a game application. This target application includes a rendering engine, such as UE (User Experience). The multiple functional controls are provided by the target application. Optionally, the target application running on the second device is the same as the target application running on the first device.

[0207] 703. When the first device detects a trigger operation on the target functional control, it acquires control trigger data. The control trigger data includes the control characteristics of the target functional control and the operation characteristics of the trigger operation. The target functional control is any one of multiple functional controls.

[0208] The first device detects whether multiple functional controls on the first virtual scene are triggered. If a trigger operation on a target functional control among the multiple functional controls is detected, the device obtains the control characteristics of the target functional control and the operation characteristics of the trigger operation, and generates control trigger data based on the control characteristics and operation characteristics. The control trigger data is used to represent the trigger operation on the target functional control.

[0209] In one possible implementation, when the first device detects a trigger operation on the target functional control, it determines the operation type of the trigger operation. If the operation type of the trigger operation does not belong to a preset operation type, it obtains the control trigger data. The preset operation type refers to the operation type that does not need to be synchronized with at least one second device.

[0210] In other words, there are predefined trigger operations of certain types that do not require synchronization with the second device. If the trigger operation for the target functional control does not belong to a preset operation type, then the trigger operation needs to be synchronized with the second device, thus requiring the acquisition of control trigger data. If the trigger operation for the target functional control does not belong to a preset operation type, then the trigger operation does not need to be synchronized with the second device, and therefore, the acquisition of control trigger data is unnecessary.

[0211] Optionally, each functional control has its own preset operation type. The preset operation type of a functional control refers to the operation type of the functional control that does not need to be synchronized with at least one second device. The preset operation types of different functional controls can be the same or different.

[0212] For example, the target function control is a virtual joystick. Sliding the virtual joystick controls the movement of a virtual object, and this movement causes a change in the virtual object's pose. In the aforementioned... Figure 4In the embodiments, a scheme has been provided to control the pose of a second virtual object on a second device to remain consistent with that of a first virtual object on a first device by synchronizing pose data. If the sliding operation of the virtual joystick is also synchronized to the second device, the second device would need to control the pose of the second virtual object both through pose data and through the sliding operation of the virtual joystick, thus causing a conflict. Therefore, the sliding operation of the virtual joystick can be set as a preset operation type. If a trigger operation on the target function control is detected to be of this preset operation type, it is not necessary to synchronize the trigger operation to the second device, thereby avoiding conflicts when the second device controls the pose of the second virtual object.

[0213] In this implementation, which operation types can be asynchronous are pre-defined. Control trigger data is only retrieved to synchronize control of the second device when the operation type does not belong to the pre-defined asynchronous operation types. This avoids unnecessary transmission of asynchronous operations, thus reducing communication burden. Furthermore, which operation types do not need to be synchronized can be flexibly set according to actual conditions, improving the flexibility of device control.

[0214] Optionally, when the first device detects a trigger operation on the target functional control, it determines the operation type of the trigger operation, including: determining the start position and end position of the trigger operation; if the distance between the start position and the end position is greater than a distance threshold, determining the operation type of the trigger operation as a swipe operation; if the distance between the start position and the end position is not greater than the distance threshold, determining the operation type of the trigger operation as a click operation. The distance threshold can be a pre-set threshold.

[0215] If the distance between the start and end positions of the triggered operation is greater than a distance threshold, it indicates that the movement path of the triggered operation is relatively long, and the triggered operation can be considered a swipe operation. If the distance between the start and end positions of the triggered operation is not greater than the distance threshold, it indicates that the movement path of the triggered operation is relatively short, and the triggered operation can be considered a click operation.

[0216] In this implementation, by detecting the distance between the start and end positions of the operation, click and swipe operations can be accurately distinguished. This method of determining positional distance is simple and intuitive, helping to improve the accuracy of operation type recognition. Furthermore, different operation types for the same functional control may be used to implement different functions; by automatically distinguishing between click and swipe operations, the accuracy of device control can be improved.

[0217] In one possible implementation, the first device runs a target application, and multiple functional controls are provided by the target application. Upon detecting a trigger operation, the first device determines the coordinates of the start position of the trigger operation in the target application's interface coordinate system; it obtains the coordinates of the multiple functional controls in the target application's interface coordinate system, where the coordinates of the functional controls represent their display positions on the target application's interface; and it identifies the functional control whose coordinates match the start position of the operation as the triggered target functional control.

[0218] The target application's interface coordinate system is an attribute of the target application itself. The target application's interface coordinate system is independent of the device running the target application, and the target application's interface coordinate system is the same on any device.

[0219] Matching the coordinates of the operation's starting position with the coordinates of a functional control means that the coordinates of the operation's starting position are the same as the coordinates of the functional control, or that the coordinates of the operation's starting position are within a preset range centered on the coordinates of the functional control. This preset range refers to the area occupied by the functional control on the interface. The first device searches among multiple functional controls for coordinates that match the coordinates of the operation's starting position, and determines the functional control corresponding to the found coordinates as the target functional control to be triggered.

[0220] In this implementation, by matching the start position of the operation with the position of the functional control in the interface coordinate system, the actual functional control triggered by the user can be accurately identified. This avoids misjudgments caused by changes in control layout or inconsistent coordinates, ensuring the accuracy of detecting which functional control was triggered. Furthermore, detection based on the target application's interface coordinate system can adapt to different devices and screen resolutions. Regardless of the device or screen size on which the target application runs, as long as the control position is obtained in the interface coordinate system, the accuracy of trigger recognition can be guaranteed, thereby improving the accuracy of device control.

[0221] In one possible implementation, the first device runs a target application, and the first virtual scene is a virtual scene within the target application. The target application includes a control control plugin. When a trigger operation on a target functional control is detected, control trigger data is acquired, including: detecting multiple functional controls on the first virtual scene through the control control plugin, and acquiring control trigger data when a trigger operation on a target functional control is detected through the control control plugin.

[0222] The control plugin is a built-in plugin in the target application. It can be used to detect whether functional controls in the target application are triggered. The target application running on the first device can call the control plugin to detect multiple functional controls on the first virtual scene and obtain control trigger data.

[0223] For example, the target application is a game application, which includes a rendering engine. The control plugin is a built-in plugin within the rendering engine. It can be understood as integrating control detection functionality into the rendering engine as a plugin, and packaging it into the target application through the rendering engine, thereby extending the target application's control detection capabilities. Therefore, any application that includes this rendering engine can use this control plugin to implement the device control scheme executed by the first device.

[0224] In this implementation, a control plugin is built into the target application. This plugin enables the detection and control of functional controls within the target application, allowing for fast and accurate acquisition of control trigger data and ensuring real-time data retrieval. Furthermore, it expands the functionality of the target application, enabling device synchronization control methods to be implemented in any type of target application, adapting to different usage scenarios and application requirements.

[0225] In one possible implementation, the operation characteristics that trigger the operation include at least one of the following: operation duration, operation type, operation start position, or operation end position.

[0226] In this implementation, the first device synchronizes detailed information such as operation duration, operation type, operation start position, or operation end position to the second device. The second device can use this information to more accurately simulate the triggering operation of the target function control, thereby improving the accuracy of the first device controlling the second device.

[0227] In one possible implementation, the functional controls include a SWidget control and a UWidget control. The control characteristics of the SWidget control are shown in Table 1 below, and the control characteristics of the UWidget control are shown in Table 2 below.

[0228] Table 1

[0229]

[0230] Table 2

[0231]

[0232] In one possible implementation, the first device registers control detection events with the rendering engine via a control control plugin to detect whether the user has clicked or swiped a functional control, such as clicking a shooting button or swiping a character selection list in a game application. When the user begins to touch the screen of the first device, the first device receives a trigger start event callback from the rendering engine, determining the trigger start position as (X1, Y1) and the trigger start time as t1. The control control plugin determines the currently triggered functional control as Widget_A and obtains the control feature_A of Widget_A. When the user ends touching the screen of the first device, the first device receives a trigger end event callback from the rendering engine, determining the trigger end position as (X2, Y2) and the trigger end time as t2. The distance between the trigger end position (X2, Y2) and the trigger start position (X1, Y1) is determined. If the distance is greater than a distance threshold, the trigger operation is determined to be a swipe operation; otherwise, it is determined to be a click operation.

[0233] 704. The first device sends a second control instruction to at least one second device, the second control instruction including control trigger data.

[0234] The first device is used to synchronously control at least one second device. After acquiring control trigger data, the first device generates a second control instruction including the control trigger data and sends the second control instruction to at least one second device. The second control instruction is used to instruct the second device to simulate the triggering operation of the target function control according to the control trigger data.

[0235] Figure 8 This is a flowchart of another device control method provided in an embodiment of this application, such as... Figure 8 As shown, the first device initiates control group control, which refers to synchronously controlling at least one second device by sending control trigger data. The first device registers control detection events, determines whether a trigger operation is detected, and if so, identifies the Swidget control at the trigger start position. The first device checks if the Swidget control is wrapped by a Uwidget control; if so, it obtains the control characteristics of the Uwidget control; otherwise, it obtains the control characteristics of the Swidget control. The second device generates control trigger data based on the obtained control characteristics and the operation characteristics of the trigger operation, and synchronizes the control trigger data to at least one second device. The first device determines whether to end control group control; if not, it repeats the above steps until control group control ends.

[0236] 705. The second device receives a second control instruction sent by the first device, the second control instruction including control trigger data.

[0237] The second device receives a first control instruction including control trigger data, which represents a trigger operation detected by the first device on a target functional control. The target functional control is any one of a plurality of functional controls, and the control trigger data includes the control characteristics of the target functional control and the operation characteristics of the trigger operation.

[0238] 706. The second device responds to the second control command and simulates the triggering operation of the target function control according to the control trigger data.

[0239] The control trigger data includes control characteristics and operation characteristics. The second device determines the target functional control based on the control characteristics in the control trigger data, and determines the trigger operation based on the operation characteristics. For example, the operation characteristics can determine the trigger type, trigger start position, trigger end position, and trigger duration. The second device searches for the target functional control among multiple functional controls displayed on the second virtual scene and simulates the trigger operation on the target functional control to ensure that the operation performed by the second device is consistent with the operation performed by the first device.

[0240] In one possible implementation, the second device runs a target application, and the second virtual scene is a virtual scene within the target application. The target application includes a control plugin. Responding to a second control command, the second device, through the control plugin, simulates triggering operations on the target functional control according to the control trigger data.

[0241] For example, the target application is a game application, which includes a rendering engine. The control plugin is a built-in plugin within the rendering engine. It can be understood as integrating control functionality into the rendering engine as a plugin, and then packaging it into the target application, thereby extending the control functionality of the target application. Therefore, any application that includes this rendering engine can use this control plugin to implement the device control scheme executed by the second device.

[0242] For example, the control plugin mentioned in step 706 and the control plugin mentioned in step 703 above can be the same control plugin, which has the function of detecting and controlling functional controls. In the group control scheme, in the host device such as the first device, the control plugin can be called to detect whether the functional control is triggered, and in the slave device such as the second device, the control plugin can be called to simulate the triggering of the functional control.

[0243] In this implementation, a control plugin is built into the target application. This plugin detects and controls the functional controls within the target application, enabling fast and precise triggering of these controls and ensuring real-time synchronous triggering. Furthermore, it expands the functionality of the target application, allowing for device synchronization control in any type of application, adapting to different usage scenarios and application requirements.

[0244] Figure 9 This is a flowchart of another device control method provided in an embodiment of this application, such as... Figure 9 As shown, the second device begins receiving control group control from the first device. The second device determines whether it has received control trigger data synchronized from the first device. If it has, it checks if the control characteristics in the trigger data match those of a Uwidget control. If so, it locates the corresponding Uwidget control and its interface coordinates based on the control characteristics. Otherwise, it locates the corresponding Swidget control and its interface coordinates based on the control characteristics. The second device uses a control control plugin to simulate triggering operations on the target functional control indicated by the control characteristics. The second device then determines whether to end the pose group control. If not, it repeats the above steps until the pose group control ends.

[0245] The method provided in this application embodiment allows the first device to send control trigger data to the second device, enabling the second device to accurately simulate the user's trigger operation on the target function control on the first device. This ensures that the user's trigger operation remains consistent across multiple devices, thereby achieving synchronous control of the devices and improving the accuracy of device control.

[0246] It should be noted that the above embodiments are only illustrated by sending a second control command to one second device. In another embodiment, the first device sends a second control command to multiple second devices, and each second device performs steps 702, 705, and 706 as described above.

[0247] It should be noted that steps 701-706 are executed instantaneously, meaning that the next step can be executed as soon as one step is completed, thus enabling real-time control of the second device by the first device. In another embodiment, steps 701-706 are not executed instantaneously. After the first device executes step 701, it stores the control trigger data whenever a function control is triggered, which means repeating step 703. After the first virtual object is displayed, the first device stores multiple control trigger data that change progressively. The first device sends a control command including the multiple control trigger data to the second device, and the second device sequentially simulates the triggering operation of the function control according to the multiple control trigger data, thereby enabling non-real-time control of the second device by the first device.

[0248] Figure 10 This is an architecture diagram of a target application provided in an embodiment of this application, such as... Figure 10 As shown, the target application includes a rendering engine, which comprises the FSlate application class, virtual object class, object controller class, UI layout-related classes, and control-related classes. The FSlate application class is used for control detection event registration, simulating trigger operations, and control collision detection. The virtual object class and object controller class are used to read and control position and orientation. The UI layout-related classes are used for UUserWidget traversal, UWidget traversal, and obtaining UWidget characteristics. The control-related classes are used to obtain SWidget characteristics.

[0249] It should be noted that the above embodiments are only illustrated by the example of the first device directly sending the first control command to the second device. In another embodiment, both the first device and the second device are connected to a management device, which forwards the control command from the first device to the second device. For details, please refer to the following... Figure 11 Examples of implementations. Figure 11 This is a flowchart of another device control method provided in this application embodiment. This application embodiment is executed by a management device, a first device, and a second device. See also... Figure 11 The method includes:

[0250] 1101. Management device display and control management interface.

[0251] The first device is used to synchronously control at least one second device, and the first device and at least one second device have established communication connections with the management device respectively.

[0252] This control management interface is used to manage the group control between the first device and at least one second device. Figure 12 This is a schematic diagram of a control and management interface provided in an embodiment of this application, such as... Figure 12As shown, the control management interface allows for the configuration of group control modes, master devices, and slave devices. The group control function is used to implement these settings. Figure 7 In an embodiment, pose group control is used to achieve the above. Figure 4 In this embodiment, if sliding group control is not selected, the sliding operation will be set to a preset operation type, that is, the sliding operation will not be synchronized.

[0253] In one possible implementation, the management device runs a group control application, and the control management interface is the interface provided in the group control application.

[0254] 1102. The first device displays the first virtual object located in the first virtual scene.

[0255] 1103. The second device displays the second virtual object located in the second virtual scene.

[0256] 1104. When the first device detects that the pose of the first virtual object has changed in the first virtual scene, it sends a first control command to the management device. The first control command includes pose data, which represents the position and orientation of the first virtual object after the change in the first virtual scene.

[0257] 1105. The management device receives the first control command and forwards the first control command to the second device.

[0258] In one possible implementation, the management device runs a group control application. Through this application, the management device receives a first control command and forwards it to a second device.

[0259] 1106. The second device responds to the first control command and controls the position and orientation of the second virtual object in the second virtual scene according to the pose data.

[0260] 1107. When the first device detects a trigger operation on the target functional control, it sends a second control instruction to the management device. The second control instruction includes control trigger data, which includes the control characteristics of the target functional control and the operation characteristics of the trigger operation. The target functional control is any one of the multiple functional controls displayed on the first virtual scene.

[0261] 1108. The management device receives the second control command and forwards the second control command to the second device.

[0262] In one possible implementation, the management device runs a group control application. Through this application, the management device receives second control commands and forwards them to second devices.

[0263] 1109. The second device responds to the second control command and simulates the triggering operation of the target function control according to the control trigger data.

[0264] In this embodiment, the scene layout of the first virtual scene displayed by the first device is the same as that of the second virtual scene displayed by the second device. After the first device controls the first virtual object to move in the first virtual scene, it synchronizes the changed pose data of the first virtual object to the second device through the management device. Since the scene layout is the same, the position and orientation indicated by the same pose data in the first and second virtual scenes are consistent. Therefore, the second device controls the position and orientation of the second virtual object in the second virtual scene according to the pose data, which ensures that the second virtual object remains synchronized with the first virtual object and does not deviate. This enables the first device to synchronously control the content displayed by the second device through the management device, improving the accuracy of device control.

[0265] Furthermore, the first device synchronizes the control trigger data to the second device through the management device, enabling the second device to accurately simulate the user's trigger operation on the target function control on the first device. In this way, the user's trigger operation can be kept consistent across multiple devices, thereby achieving synchronous control of the devices and improving the accuracy of device control.

[0266] The device control method provided in this application can be applied to any scenario requiring group control of devices. For example, in a UE game scenario, the device control method of this application can be used to perform functional testing, performance testing, and compatibility testing on the UE game. If the pose data or control trigger data is saved, and after the first device finishes displaying, the saved pose data or control trigger data is sent to the second device, the display content of the first device can be reproduced on the second device, forming a recording and playback scheme that can be used for automated functional testing, automated performance testing, and automated compatibility testing of the UE game. The detailed process of the device control method when applied to a UE game scenario can be found below. Figure 13 Examples of implementations.

[0267] Figure 13 This is a flowchart of another device control method provided in an embodiment of this application, see [link to flowchart]. Figure 13 The method includes the following steps:

[0268] 1301. The first device displays a first game character located in a first virtual scene in a game application. The first virtual scene displays multiple game controls. The game application includes a rendering engine, and the rendering engine has a built-in pose control plugin and a control control plugin.

[0269] 1302. The second device displays a second game character located in a second virtual scene in the game application, and multiple game controls are displayed on the second virtual scene.

[0270] 1303. When the first device detects a change in the pose of the first game character in the first virtual scene through the pose control plugin, it sends a first control command to the management device. The first control command includes pose data, which represents the position and orientation of the first game character in the first virtual scene after the change.

[0271] 1304. The management device receives the first control command and forwards the first control command to the second device.

[0272] 1305. The second device responds to the first control command and, through the pose control plugin, controls the position and orientation of the second game character in the second virtual scene according to the pose data.

[0273] 1306. When the first device detects a trigger operation on the target game control through the control control plugin, it sends a second control instruction to the management device. The second control instruction includes control trigger data, which includes the control characteristics of the target game control and the operation characteristics of the trigger operation. The target game control is any one of the multiple game controls displayed on the first virtual scene.

[0274] 1307. The management device receives the second control command and forwards the second control command to the second device.

[0275] 1308. The second device responds to the second control command and, through the control control plugin, simulates the triggering operation of the target game control according to the control trigger data.

[0276] In this embodiment, from the perspective of game controls, the function of accurate group control of game controls in a game application is realized. From the perspective of game character poses, the function of accurate group control of game character poses in a game application is realized, thereby improving the accuracy of device group control.

[0277] Figure 14 This is a schematic diagram of the structure of a device control apparatus provided in an embodiment of this application. See also... Figure 14 The device includes:

[0278] Display module 1401 is used to display the first virtual object located in the first virtual scene;

[0279] The first acquisition module 1402 is used to acquire the pose data of the first virtual object after the pose of the first virtual object changes in the first virtual scene. The pose data represents the position and orientation of the first virtual object in the first virtual scene.

[0280] The first transmitting module 1403 is used to send a first control command to at least one second device. The first control command includes pose data. The first device is used to perform synchronous control on at least one second device.

[0281] The second device displays a second virtual scene, which includes a second virtual object. The scene layout of the second virtual scene is the same as that of the first virtual scene. The first control command is used to instruct the second device to control the position and orientation of the second virtual object in the second virtual scene according to the pose data.

[0282] The device control apparatus provided in this application embodiment has the same scene layout for a first virtual scene displayed by a first device and a second virtual scene displayed by a second device. After the first device controls a first virtual object to move in the first virtual scene, it synchronizes the changed pose data of the first virtual object to the second device. Since the scene layout is the same, the position and orientation indicated by the same pose data in the first and second virtual scenes are consistent. Therefore, the second device controls the position and orientation of the second virtual object in the second virtual scene based on the pose data, ensuring that the second virtual object remains synchronized with the first virtual object without deviation. This achieves synchronous control of the content displayed by the second device by the first device, improving the accuracy of device control.

[0283] Optionally, see Figure 15 The first acquisition module 1402 is used for:

[0284] If the change in pose data of the first virtual object between the i-th frame and the k-th frame is greater than the change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed. The k-th frame is the frame before the i-th frame, where i and k are positive integers.

[0285] The pose data of the first virtual object in the i-th frame is determined as the pose data of the first virtual object after the change.

[0286] Optionally, see Figure 15 The pose data includes position parameters and orientation parameters; the first acquisition module 1402 is used to perform any of the following:

[0287] The position change is obtained. If the position change is greater than the position change threshold, it is determined that the pose of the first virtual object in the first virtual scene has changed. The position change represents the difference between the position parameters of the i-th frame and the position parameters of the k-th frame.

[0288] The orientation change is obtained. If the orientation change is greater than the orientation change threshold, it is determined that the pose of the first virtual object in the first virtual scene has changed. The orientation change represents the difference between the orientation parameter of the i-th frame and the orientation parameter of the k-th frame.

[0289] The position change and orientation change are obtained. If the position change is greater than the position change threshold and the orientation change is greater than the orientation change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed.

[0290] Optionally, see Figure 15 The first acquisition module 1402 is configured to perform at least one of the following:

[0291] The kth frame refers to the frame preceding the i-th frame. If the change between the pose data of the i-th frame and the pose data of the k-th frame is greater than the first change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed.

[0292] The kth frame refers to the frame from which the control command was last sent. If the change in pose data between the i-th frame and the k-th frame is greater than the second change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed; wherein the second change threshold is greater than the first change threshold.

[0293] Optionally, see Figure 15 The first acquisition module 1402 is also used for:

[0294] When the pose of the first virtual object changes in the first virtual scene, the target frame number is determined. The target frame number refers to the number of frames between the kth frame and the ith frame. The kth frame refers to the frame when the control command was last sent, and the ith frame refers to the frame when the first control command was sent.

[0295] The first control instruction also includes a target number of frames. The first control instruction is used to instruct the second device to control the position and orientation of the second virtual object in the second virtual scene according to the pose data within a target number of frames.

[0296] Optionally, see Figure 15 The first and second devices run a target application, and the first and second virtual scenes are virtual scenes within the target application. The target application includes a pose control plugin; the first acquisition module 1402 is used for:

[0297] The pose control plugin is used to detect the pose of the first virtual object in the first virtual scene. When the pose control plugin detects a change in the pose of the first virtual object, the pose data of the first virtual object after the change is obtained.

[0298] Optionally, see Figure 15 The first virtual scene displays multiple functional controls; the device also includes:

[0299] The second acquisition module 1404 is used to acquire control trigger data when a trigger operation on a target functional control is detected. The control trigger data includes the control characteristics of the target functional control and the operation characteristics of the trigger operation. The target functional control is any one of multiple functional controls.

[0300] The second sending module 1405 is used to send a second control instruction to at least one second device, the second control instruction including control trigger data;

[0301] In the second virtual scene of the second device, multiple functional controls are displayed. The second control command is used to instruct the second device to simulate the triggering operation of the target functional control according to the control trigger data.

[0302] Optionally, see Figure 15 The second acquisition module 1404 is used for:

[0303] If a trigger operation on the target functional control is detected, determine the type of the trigger operation;

[0304] If the operation type that triggers the operation is not a preset operation type, obtain the control trigger data. The preset operation type refers to the operation type that does not need to be synchronized with at least one second device.

[0305] Optionally, see Figure 15 The second acquisition module 1404 is used for:

[0306] Upon detecting a trigger operation on the target functional control, determine the start and end positions of the trigger operation;

[0307] If the distance between the start and end positions of an operation is greater than a distance threshold, the operation type that triggers the operation is determined to be a sliding operation.

[0308] If the distance between the start and end positions of an operation is not greater than a distance threshold, the operation type that triggered the operation is determined to be a click operation.

[0309] Optionally, see Figure 15 The first and second devices run a target application, and multiple functional controls are provided by the target application; the second acquisition module 1404 is used for:

[0310] If a triggering operation is detected, determine the coordinates of the starting position of the triggering operation in the target application's interface coordinate system;

[0311] Get the coordinates of multiple functional controls in the target application's interface coordinate system. The coordinates of the functional controls represent their display positions on the target application's interface.

[0312] The function control whose coordinates match the starting position of the operation is identified as the target function control to be triggered.

[0313] Optionally, see Figure 15 The operation characteristics that trigger the operation include at least one of the following: operation duration, operation type, operation start position, or operation end position.

[0314] Optionally, see Figure 15 The first and second devices run a target application, and the first and second virtual scenes are virtual scenes within the target application. The target application includes a control plugin. The second acquisition module 1404 is used for:

[0315] The control plugin detects multiple functional controls on the first virtual scene. When the control plugin detects a trigger operation on the target functional control, the control trigger data is obtained.

[0316] It should be noted that the device control apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device control apparatus and the device control method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0317] Figure 16 This is a schematic diagram of the structure of a device control apparatus provided in an embodiment of this application. See also... Figure 16 The device includes:

[0318] Display module 1601 is used to display a second virtual object located in the second virtual scene;

[0319] The first receiving module 1602 is used to receive a first control command sent by the first device. The first control command includes pose data. The first device is used to perform synchronous control on the second device. The first device displays a first virtual scene. The first virtual scene includes a first virtual object. The scene layout of the second virtual scene is the same as that of the first virtual scene. The pose data represents the position and orientation of the first virtual object after changes in the first virtual scene.

[0320] The first control module 1603 is used to respond to the first control command and control the position and orientation of the second virtual object in the second virtual scene according to the pose data.

[0321] The device control apparatus provided in this application embodiment has the same scene layout for a first virtual scene displayed by a first device and a second virtual scene displayed by a second device. After the first device controls a first virtual object to move in the first virtual scene, it synchronizes the changed pose data of the first virtual object to the second device. Since the scene layout is the same, the position and orientation indicated by the same pose data in the first and second virtual scenes are consistent. Therefore, the second device controls the position and orientation of the second virtual object in the second virtual scene based on the pose data, ensuring that the second virtual object remains synchronized with the first virtual object without deviation. This achieves synchronous control of the content displayed by the second device by the first device, improving the accuracy of device control.

[0322] Optionally, see Figure 17 The first control command also includes a target frame number, which represents the number of frames between the k-th frame and the i-th frame. The k-th frame refers to the frame displayed by the first device when the control command was received last time, and the i-th frame refers to the frame displayed by the first device when the first control command was received.

[0323] The first control module 1603 is used for:

[0324] In response to the first control command, within a few screen frames of the target frame, the second virtual object is controlled to move in the second virtual scene to the position and orientation indicated by the pose data.

[0325] Optionally, see Figure 17 The first and second devices run a target application, and the first and second virtual scenes are virtual scenes in the target application. The target application includes a pose control plugin.

[0326] The first control module 1603 is used for:

[0327] In response to the first control command, the position and orientation of the second virtual object in the second virtual scene are controlled by the pose control plugin according to the pose data.

[0328] Optionally, see Figure 17 The second virtual scene displays multiple functional controls; the device also includes:

[0329] The second receiving module 1604 is used to receive a second control instruction sent by the first device. The second control instruction includes control trigger data, which represents a trigger operation detected by the first device on a target functional control. The target functional control is any one of a plurality of functional controls. The control trigger data includes the control characteristics of the target functional control and the operation characteristics of the trigger operation.

[0330] The second control module 1605 is used to respond to the second control command and simulate the triggering operation of the target function control according to the control trigger data.

[0331] Optionally, see Figure 17 The first and second devices run a target application, and the first and second virtual scenes are virtual scenes in the target application. The target application includes a control plugin.

[0332] The second control module 1605 is used for:

[0333] In response to the second control command, the control plugin simulates the triggering operation of the target function control according to the control trigger data.

[0334] It should be noted that the device control apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device control apparatus and the device control method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0335] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to implement the operations performed in the device control method of the above embodiments.

[0336] Optionally, the computer device is provided as a terminal. Figure 18 A schematic diagram of the structure of a terminal 1800 provided in an exemplary embodiment of this application is shown.

[0337] Terminal 1800 includes a processor 1801 and a memory 1802.

[0338] Processor 1801 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0339] Memory 1802 may include one or more computer-readable storage media, which may be non-transitory. Memory 1802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 1802 are used to store at least one computer program, which is used by processor 1801 to implement the device control method provided in the method embodiments of this application.

[0340] In some embodiments, the terminal 1800 may also optionally include: a peripheral device interface 1803 and at least one peripheral device. The processor 1801, memory 1802, and peripheral device interface 1803 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1803 via a bus, signal line, or circuit board. Optionally, the peripheral device includes at least one of: a radio frequency circuit 1804, a display screen 1805, a camera assembly 1806, an audio circuit 1807, and a power supply 1808.

[0341] Peripheral device interface 1803 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1801 and memory 1802. In some embodiments, processor 1801, memory 1802 and peripheral device interface 1803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1801, memory 1802 and peripheral device interface 1803 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0342] The radio frequency (RF) circuit 1804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1804 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1804 can communicate with other devices through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1804 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0343] Display screen 1805 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1805 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1801 for processing. In this case, display screen 1805 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1805, disposed on the front panel of terminal 1800; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1800 or in a folded design; in still other embodiments, display screen 1805 may be a flexible display screen, disposed on a curved or folded surface of terminal 1800. Furthermore, display screen 1805 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1805 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0344] The camera assembly 1806 is used to acquire images or videos. Optionally, the camera assembly 1806 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the terminal 1800, and the rear-facing camera is disposed on the back of the terminal 1800. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1806 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0345] The audio circuit 1807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are input to the processor 1801 for processing, or to the radio frequency circuit 1804 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1800. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1801 or the radio frequency circuit 1804 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1807 may also include a headphone jack.

[0346] The power supply 1808 is used to power the various components in the terminal 1800. The power supply 1808 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 1808 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0347] In some embodiments, the terminal 1800 further includes one or more sensors 1809. The one or more sensors 1809 include, but are not limited to: an acceleration sensor 1810, a gyroscope sensor 1811, a pressure sensor 1812, an optical sensor 1813, and a proximity sensor 1814.

[0348] Accelerometer 1810 can detect the magnitude of acceleration along the three axes of a coordinate system established by terminal 1800. For example, accelerometer 1810 can be used to detect the components of gravitational acceleration along the three axes. Processor 1801 can control display screen 1805 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1810. Accelerometer 1810 can also be used for games or for acquiring user motion data.

[0349] The gyroscope sensor 1811 can detect the orientation and rotation angle of the terminal 1800. The gyroscope sensor 1811 can work in conjunction with the accelerometer sensor 1810 to acquire the user's 3D movements on the terminal 1800. Based on the data acquired by the gyroscope sensor 1811, the processor 1801 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0350] The pressure sensor 1812 can be disposed on the side bezel of the terminal 1800 and / or on the lower layer of the display screen 1805. When the pressure sensor 1812 is disposed on the side bezel of the terminal 1800, it can detect the user's grip signal on the terminal 1800, and the processor 1801 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1812. When the pressure sensor 1812 is disposed on the lower layer of the display screen 1805, the processor 1801 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1805. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0351] An optical sensor 1813 is used to collect ambient light intensity. In one embodiment, the processor 1801 can control the display brightness of the display screen 1805 based on the ambient light intensity collected by the optical sensor 1813. Optionally, when the ambient light intensity is high, the display brightness of the display screen 1805 is increased; when the ambient light intensity is low, the display brightness of the display screen 1805 is decreased. In another embodiment, the processor 1801 can also dynamically adjust the shooting parameters of the camera assembly 1806 based on the ambient light intensity collected by the optical sensor 1813.

[0352] The proximity sensor 1814, also known as the distance sensor, is installed on the front panel of the terminal 1800. The proximity sensor 1814 is used to detect the distance between the user and the front of the terminal 1800. In one embodiment, when the proximity sensor 1814 detects that the distance between the user and the front of the terminal 1800 is gradually decreasing, the processor 1801 controls the display screen 1805 to switch from a screen-on state to a screen-off state; when the proximity sensor 1814 detects that the distance between the user and the front of the terminal 1800 is gradually increasing, the processor 1801 controls the display screen 1805 to switch from a screen-off state to a screen-on state.

[0353] Those skilled in the art will understand that Figure 18 The structure shown does not constitute a limitation on terminal 1800 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0354] Optionally, the computer device is provided as a server. Figure 19This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1900 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1901 and one or more memories 1902. The memories 1902 store at least one computer program, which is loaded and executed by the processor 1901 to implement the methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0355] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the operations performed by the device control method of the above embodiments.

[0356] This application also provides a computer program product, including a computer program that is loaded and executed by a processor to perform the operations performed by the device control method of the above embodiments.

[0357] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0358] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. A device control method, characterized in that, Performed by a first device, the method includes: Display the first virtual object located in the first virtual scene; When the pose of the first virtual object changes in the first virtual scene, the pose data of the first virtual object after the change is obtained, and the pose data represents the position and orientation of the first virtual object in the first virtual scene; Send a first control command to at least one second device, the first control command including the pose data, the first device being used to perform synchronous control of the at least one second device; The second device displays a second virtual scene, which includes a second virtual object. The scene layout of the second virtual scene is the same as that of the first virtual scene. The first control command is used to instruct the second device to control the position and orientation of the second virtual object in the second virtual scene according to the pose data.

2. The method according to claim 1, characterized in that, When the pose of the first virtual object changes in the first virtual scene, obtaining the changed pose data of the first virtual object includes: If the change in pose data of the first virtual object between the i-th frame and the k-th frame is greater than the change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed. The k-th frame is the frame before the i-th frame, where i and k are positive integers. The pose data of the first virtual object in the i-th frame is determined as the pose data of the first virtual object after the change.

3. The method according to claim 2, characterized in that, The pose data includes position parameters and orientation parameters; when the change in pose data of the first virtual object between the i-th frame and the k-th frame is greater than a change threshold, it is determined that the pose of the first virtual object in the first virtual scene has changed, including any one of the following: The position change amount is obtained. If the position change amount is greater than the position change amount threshold, it is determined that the pose of the first virtual object in the first virtual scene has changed. The position change amount represents the difference between the position parameters of the i-th frame and the position parameters of the k-th frame. If the orientation change amount is greater than the orientation change amount threshold, it is determined that the pose of the first virtual object in the first virtual scene has changed. The orientation change amount represents the difference between the orientation parameter of the i-th frame and the orientation parameter of the k-th frame. The position change and the orientation change are obtained. If the position change is greater than the position change threshold and the orientation change is greater than the orientation change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed.

4. The method according to claim 2, characterized in that, If the change in pose data of the first virtual object between the i-th frame and the k-th frame is greater than a change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed, including at least one of the following: The kth frame refers to the frame preceding the i-th frame. If the change between the pose data of the i-th frame and the pose data of the k-th frame is greater than a first change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed. The kth frame refers to the frame from which the control command was last sent. If the change between the pose data of the i-th frame and the pose data of the k-th frame is greater than a second change threshold, then it is determined that the pose of the first virtual object in the first virtual scene has changed; wherein, the second change threshold is greater than the first change threshold.

5. The method according to claim 1, characterized in that, The method further includes: When the pose of the first virtual object in the first virtual scene changes, the target frame number is determined. The target frame number refers to the number of frames from the kth frame to the ith frame. The kth frame refers to the frame when the control command was last sent, and the ith frame refers to the frame when the first control command was sent. The first control instruction further includes the target number of frames. The first control instruction is used to instruct the second device to control the position and orientation of the second virtual object in the second virtual scene according to the pose data within the target number of frames.

6. The method according to claim 1, characterized in that, The first device and the second device run a target application, the first virtual scene and the second virtual scene are virtual scenes in the target application, and the target application includes a pose control plugin; the step of acquiring the pose data of the first virtual object after the pose of the first virtual object in the first virtual scene changes includes: The pose control plugin detects the pose of the first virtual object in the first virtual scene. When the pose control plugin detects a change in the pose of the first virtual object, it acquires the pose data of the first virtual object after the change.

7. The method according to any one of claims 1 to 6, characterized in that, The first virtual scene displays multiple functional controls; the method further includes: Upon detecting a trigger operation on a target functional control, control trigger data is acquired. The control trigger data includes the control characteristics of the target functional control and the operation characteristics of the trigger operation. The target functional control is any one of the plurality of functional controls. Send a second control command to the at least one second device, the second control command including the control trigger data; The second virtual scene in the second device displays the plurality of functional controls, and the second control instruction is used to instruct the second device to simulate the triggering operation of the target functional control according to the control trigger data.

8. The method according to claim 7, characterized in that, The step of acquiring control trigger data upon detecting a trigger operation on the target functional control includes: If a trigger operation on the target functional control is detected, the operation type of the trigger operation is determined; If the operation type of the triggered operation does not belong to the preset operation type, the control trigger data is obtained. The preset operation type refers to the operation type that does not need to be synchronized with the at least one second device.

9. The method according to claim 7, characterized in that, Both the first device and the second device run a target application, and the multiple functional controls are provided by the target application. The step of acquiring control trigger data upon detecting a trigger operation on the target functional control includes: Upon detecting the trigger operation, determine the coordinates of the start position of the trigger operation in the interface coordinate system of the target application; Obtain the coordinates of the plurality of functional controls in the interface coordinate system of the target application, wherein the coordinates of the functional controls represent the display position of the functional controls on the interface of the target application; The function control that matches the coordinates of the operation start position is identified as the target function control to be triggered.

10. The method according to claim 7, characterized in that, The first device and the second device run a target application, and the first virtual scene and the second virtual scene are virtual scenes within the target application. The target application includes a control plugin. The step of acquiring control trigger data upon detecting a trigger operation on a target functional control includes: The control plugin detects the multiple functional controls on the first virtual scene, and when the control plugin detects a trigger operation on the target functional control, it acquires the control trigger data.

11. A device control method, characterized in that, Performed by a second device, the method includes: Display the second virtual object located in the second virtual scene; The device receives a first control command sent by a first device, the first control command including pose data; the first device is used to synchronously control the second device, the first device displays a first virtual scene, the first virtual scene includes a first virtual object, the scene layout of the second virtual scene is the same as the scene layout of the first virtual scene, and the pose data represents the position and orientation of the first virtual object after changes in the first virtual scene; In response to the first control command, the position and orientation of the second virtual object in the second virtual scene are controlled according to the pose data.

12. The method according to claim 11, characterized in that, The first device and the second device are running a target application, the first virtual scene and the second virtual scene are virtual scenes in the target application, and the target application includes a pose control plugin; The step of responding to the first control command and controlling the position and orientation of the second virtual object in the second virtual scene according to the pose data includes: In response to the first control command, the position and orientation of the second virtual object in the second virtual scene are controlled by the pose control plugin according to the pose data.

13. The method according to claim 11 or 12, characterized in that, The second virtual scene displays multiple functional controls; the method further includes: The system receives a second control instruction sent by the first device. The second control instruction includes control trigger data, which represents a trigger operation detected by the first device on a target functional control. The target functional control is any one of the plurality of functional controls. The control trigger data includes the control characteristics of the target functional control and the operation characteristics of the trigger operation. In response to the second control command, the triggering operation of the target functional control is simulated according to the control triggering data.

14. The method according to claim 13, characterized in that, The first device and the second device are running a target application, the first virtual scene and the second virtual scene are virtual scenes in the target application, and the target application includes a control plugin; The step of responding to the second control command and simulating the triggering operation of the target functional control according to the control triggering data includes: In response to the second control command, the control plugin simulates the triggering operation of the target function control according to the control trigger data.

15. A device control system, characterized in that, The equipment control system includes a management device, a first device, and at least one second device. The first device is used to synchronously control the at least one second device. The first device and the at least one second device are respectively connected to the management device. The first device is configured to send a first control command to the management device when it detects a change in the pose of the first virtual object in the first virtual scene. The first control command includes pose data, which represents the position and orientation of the first virtual object after the change in the first virtual scene. The management device is configured to receive the first control command and forward the first control command to the at least one second device. The at least one second device is configured to respond to the first control command and control the position and orientation of the second virtual object in the second virtual scene according to the pose data, wherein the scene layout of the second virtual scene is the same as that of the first virtual scene.

16. A device control apparatus, characterized in that, Configured in a first device, the means comprising: The display module is used to display the first virtual object located in the first virtual scene; The first acquisition module is used to acquire the changed pose data of the first virtual object when the pose of the first virtual object in the first virtual scene changes, wherein the pose data represents the position and orientation of the first virtual object in the first virtual scene; A first transmitting module is configured to transmit a first control command to at least one second device, the first control command including the pose data, and the first device is configured to perform synchronous control on the at least one second device. The second device displays a second virtual scene, which includes a second virtual object. The scene layout of the second virtual scene is the same as that of the first virtual scene. The first control command is used to instruct the second device to control the position and orientation of the second virtual object in the second virtual scene according to the pose data.

17. A device control apparatus, characterized in that, Configured in a second device, the device includes: The display module is used to display the second virtual object located in the second virtual scene; A first receiving module is used to receive a first control command sent by a first device, the first control command including pose data; the first device is used to perform synchronous control on the second device, the first device displays a first virtual scene, the first virtual scene includes a first virtual object, the scene layout of the second virtual scene is the same as the scene layout of the first virtual scene, and the pose data represents the position and orientation of the first virtual object after changes in the first virtual scene; The first control module is used to respond to the first control command and control the position and orientation of the second virtual object in the second virtual scene according to the pose data.

18. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the operations performed by the device control method as described in any one of claims 1 to 10, or to implement the operations performed by the device control method as described in any one of claims 11 to 14.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the operations performed by the device control method as described in any one of claims 1 to 10, or to implement the operations performed by the device control method as described in any one of claims 11 to 14.

20. A computer program product, comprising a computer program, characterized in that, The computer program is loaded and executed by a processor to perform the operations performed by the device control method as described in any one of claims 1 to 10, or to perform the operations performed by the device control method as described in any one of claims 11 to 14.

Images