Human-computer interaction method and apparatus

Abstract

The present application relates to the technical field of cloud computing. Provided are a human-computer interaction method and apparatus. The method is applied to an XR system, and comprises: a terminal device acquiring user action data, and converting the user action data into HID data corresponding to an application; the terminal device sending the HID data to a cloud device, wherein the HID data is used for instructing the application to execute an operation matching the HID data; and the terminal device receiving a video stream of the application that is sent by the cloud device, wherein the video stream is obtained by means of the cloud device performing cloud rendering. In this way, for any application compatible with or not compatible with XR, when there is a mapping relationship between the user action data and the HID data, the terminal device and the cloud device can cooperatively process the user action data to perform an application operation and a cloud rendering output, so that a user can operate the application on the basis of a human-computer interaction mode in an XR scenario, thereby improving the applicability of XR.

Description

Human-computer interaction methods and devices

[0001] This application claims priority to Chinese patent application No. 202410501234.8, filed with the State Intellectual Property Office of China on April 24, 2024, entitled "License Reuse Control Method Based on Cloud Technology", and Chinese patent application No. 202410799310.8, filed with the State Intellectual Property Office of China on June 19, 2024, entitled "Human-Computer Interaction Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of cloud computing technology, and in particular to a human-computer interaction method and apparatus. Background Technology

[0003] Extended reality (XR) refers to the combination of real and virtual environments and human-computer interaction created by computer technology and wearable devices. In XR scenarios, terminal devices such as TVs, tablets, mobile phones, and virtual reality (VR) all-in-one machines typically serve as access points, working in conjunction with cloud servers to process user-input human-computer interaction data and interface display data output by the terminal devices.

[0004] However, for a series of XR applications from a particular manufacturer, or even for a single XR application, each manufacturer or application typically has its own dedicated interaction device. For applications not adapted for XR, users cannot interact using the human-computer interaction methods available in XR scenarios. Therefore, when users use different XR applications or applications not adapted for XR, the dedicated XR interaction device has poor applicability across different XR applications or applications not adapted for XR. Summary of the Invention

[0005] This application provides a human-computer interaction method and apparatus, thereby improving the applicability of human-computer interaction methods in XR scenarios in different applications.

[0006] Firstly, this application provides a human-computer interaction method applied to a terminal device in an XR system, which also includes a cloud device connected to the terminal device. In the process of the human-computer interaction method, the terminal device acquires user action data and converts it into human interface device (HID) data corresponding to the application. Then, the terminal device sends the HID data to the cloud device, which instructs the application to perform an HID data matching operation. Next, the terminal device receives a video stream from the cloud device, which is obtained through cloud rendering by the cloud device.

[0007] Based on the aforementioned human-computer interaction method, in XR scenarios, terminal devices can convert user action data into HID data corresponding to different applications, and then send the HID data to the cloud device. The application in the cloud device performs HID data matching, and the terminal device then receives the application's video stream sent by the cloud device. In this way, for any application adapted to or not adapted to XR, where there is a mapping relationship between user action data and HID data, the terminal device and the cloud device can collaboratively process user action data for application operation and cloud rendering output. This allows users to operate applications based on the human-computer interaction methods in XR scenarios, improving the applicability of XR.

[0008] As one possible implementation, user action data includes posture data, which is collected by an inertial measurement unit (IMU) located at the user's first site and transmitted to the terminal device. The terminal device converts the posture data into HID data corresponding to the application based on a first mapping relationship of the application.

[0009] Optionally, the user's first body part can be any part such as the head, hand, foot, wrist, or ankle. For example, the first body part is the head.

[0010] Optionally, the first mapping relationship includes: when the attitude data represents the attitude angle change of the first part, the attitude data corresponds to the first HID data. The operation of matching the first HID data includes touch screen operation or button operation.

[0011] For example, attitude angle changes include pitch angle changes, yaw angle changes, or roll angle changes. The first HID data includes data generated by touching the screen or pressing a button to activate the first operation key. The operation corresponding to the first operation key includes moving the target forward, moving the target backward, moving the target left, moving the target right, moving the view to the left, or moving the view to the right.

[0012] Based on the above implementation, the terminal device can convert the user's posture data when wearing the IMU into the first HID data corresponding to the application, and then operate the application based on the first HID data. Since the terminal device supports data transmission with different types of IMUs, the mapping relationship between posture data and first HID data can be adjusted according to different applications. The terminal device can realize XR human-computer interaction in different applications based on any type of IMU, avoiding the need for users to use dedicated interaction devices for different applications. This improves the applicability of XR while reducing the hardware overhead of XR interaction devices.

[0013] As one possible implementation, user action data includes posture data, which is collected by an IMU located at a second part of the user's body and transmitted to the terminal device. The terminal device converts the posture data into HID data corresponding to the application based on a second mapping relationship of the application.

[0014] Optionally, the second part can be any part of the head, hand, foot, wrist, ankle, etc. For example, the second part is a hand or foot.

[0015] Optionally, the second mapping relationship includes: when the attitude data represents the acceleration change of the second part, the attitude data corresponds to the second HID data. The operation for matching the second HID data includes touch screen operation or button operation.

[0016] For example, acceleration changes include acceleration values ​​greater than or equal to a preset threshold, and the second HID data includes data generated by touching the screen or clicking the second operation key.

[0017] As one possible implementation, user motion data includes motion data captured by a camera, which is collected by a camera connected to or built into the terminal device and then transmitted to the terminal device.

[0018] Optionally, when the motion data indicates that the user's third body part performs a specified action at the first position in the camera's field of view, the motion data corresponds to third HID data. The operation matched by the third HID data includes data generated by clicking a second operation key, where the second operation key is located at a second position on the application's display interface, and the second position is associated with the first position.

[0019] In this way, the terminal device recognizes the user's action data based on the camera's captured images, converts the action data into the third-party HID data corresponding to the application, and can realize the user's action capture based on the camera's captured images, and operate the terminal device or application with non-touch screen operation logic according to the user's action data.

[0020] Optionally, the terminal device can also output an overlay of the application's display interface and the camera's captured image. The overlay image includes a first identifier, which indicates a second position in the display interface that coincides with the first position.

[0021] In this way, when users view the display interface of the terminal device, they can see their own actions and the corresponding position of the third part on the display interface, which makes it easier to perform operations such as adjusting the position of the third part and selecting icons on the display interface, thus improving the accuracy of XR interactive operations.

[0022] Secondly, this application provides a human-computer interaction method applied to a cloud device in an XR system. The XR system also includes a terminal device connected to the cloud device. In the human-computer interaction method, the cloud device receives HID data sent by the terminal device. The HID data is obtained by converting user action data, and the conversion method between the user action data and the HID data corresponds to the application. Then, the cloud device inputs the HID data into the application to instruct the application to perform an HID data matching operation. Next, the application's video stream is output to the terminal device; the video stream is obtained by cloud rendering performed by the cloud device.

[0023] Based on the above human-computer interaction method, in XR scenarios, cloud devices are used as application containers. In scenarios where the computing power of terminal devices is limited, cloud rendering is used to realize the output of application video streams after XR human-computer interaction. This enables different terminal devices to cooperate with cloud devices to realize XR human-computer interaction for different applications, improving the applicability of XR and reducing the hardware overhead for users in XR scenarios.

[0024] As one possible implementation, cloud devices determine the accuracy of user action data based on the degree of matching between the input features of multiple HID data points and preset data features. Input features include the input order or interval of multiple HID data points. This achieves the evaluation of the accuracy of user actions, introduces an evaluation mechanism, and improves user-application interactivity.

[0025] Thirdly, this application provides a human-computer interaction device, including a transceiver module and a processing module. The transceiver module is used to acquire user action data. The processing module is used to convert the user action data into HID data corresponding to an application. The transceiver module is also used to send the HID data to a cloud device; the HID data is used to instruct the application to perform an operation matching the HID data. The transceiver module is also used to receive a video stream of the application sent by the cloud device; the video stream is obtained by cloud rendering performed by the cloud device.

[0026] As one possible implementation, the user motion data includes attitude data transmitted from an inertial measurement unit (IMU) located at a first part of the user's body. The processing module is specifically used to: convert the attitude data into HID data corresponding to the application, based on a first mapping relationship of the application.

[0027] Optionally, the first mapping relationship includes: when the attitude data represents the attitude angle change of the first part, the attitude data corresponds to the first HID data; the operation matched by the first HID data includes touch screen operation or button operation.

[0028] Optionally, the attitude angle change includes pitch angle change, yaw angle change, or roll angle change, and the first HID data includes data generated by touching the screen or clicking the first operation key. The operation corresponding to the first operation key includes moving the target forward, moving the target backward, moving the target left, moving the target right, moving the field of view left, or moving the field of view right.

[0029] As one possible implementation, user motion data includes posture data transmitted from an IMU located at a second part of the user's body. The processing module is specifically used to: convert the posture data into HID data corresponding to the application, based on a second mapping relationship of the application.

[0030] Optionally, the second mapping relationship includes: when the attitude data represents the acceleration change of the second part, the attitude data corresponds to the second HID data; the operation matched by the second HID data includes touch screen operation or button operation.

[0031] Optionally, the acceleration change includes an acceleration value greater than or equal to a preset threshold, and the second HID data includes data generated by touching the screen or clicking the second operation key.

[0032] As one possible implementation, the user action data includes action data captured by the camera. When the action data indicates that the user's third body part is performing a specified action at a first position within the camera's view, the action data corresponds to third HID data; the operation matched by the third HID data includes data generated by clicking a second operation key, the second operation key being located at a second position on the application's display interface, the second position being associated with the first position.

[0033] As one possible implementation, the processing module is also configured to: output an overlay interface of the application's display interface and the camera's captured image; the overlay image includes a first identifier, which is used to indicate a second position in the display interface that coincides with the first position.

[0034] As one possible implementation, the human-computer interaction device may also include other modules that perform the operational steps of the human-computer interaction method described in the first aspect.

[0035] Fourthly, this application provides a human-computer interaction device, including a transceiver module and a processing module. The transceiver module receives HID data sent by a terminal device; the HID data is obtained by converting user action data, and the conversion method between the user action data and the HID data corresponds to the application. The processing module inputs the HID data into the application; the HID data instructs the application to perform an operation matching the HID data. The transceiver module is also used to output a video stream of the application to the terminal device; the video stream is obtained by cloud rendering performed by a cloud device.

[0036] As one possible implementation, the processing module is also used to: determine the accuracy of the action corresponding to the user action data based on the degree of matching between the input features of the multiple input HID data and the preset data features; the input features include the input order or input interval of the multiple HID data.

[0037] As one possible implementation, the human-computer interaction device may also include other modules that perform the operational steps of the human-computer interaction method described in the second aspect.

[0038] Fifthly, a computing device is provided, including a processor and a memory. The processor of the computing device is used to execute instructions stored in the memory of the computing device, so that the computing device performs the human-computer interaction method described in any possible embodiment of the first aspect above.

[0039] Sixthly, a computing device is provided, including a processor and a memory. The processor of the computing device is used to execute instructions stored in the memory of the computing device, so that the computing device performs the human-computer interaction method described in any possible embodiment of the second aspect above.

[0040] In a seventh aspect, a computer program product is provided. The computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the human-computer interaction method described in any of the possible embodiments of the first aspect.

[0041] Eighthly, a computer program product is provided. The computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the human-computer interaction method described in any of the possible embodiments of the second aspect above.

[0042] A ninth aspect provides a computer-readable storage medium. The readable storage medium includes: a computer program or instructions; when the computer program or instructions are executed on a computer, the computer causes the computer to perform the human-computer interaction method described in any of the possible embodiments of the first aspect.

[0043] A tenth aspect provides a computer-readable storage medium. The readable storage medium includes: a computer program or instructions; when the computer program or instructions are executed on a computer, the computer causes the computer to perform the human-computer interaction method described in any of the possible embodiments of the first aspect. Attached Figure Description

[0044] Figure 1 is a schematic diagram of the architecture of a human-computer interaction system provided in this application;

[0045] Figure 2 is a structural schematic diagram of a cloud device provided in this application;

[0046] Figure 3 is a hierarchical diagram of a human-computer interaction system provided in this application;

[0047] Figure 4 is a flowchart illustrating a human-computer interaction method provided in this application;

[0048] Figure 5 is a schematic diagram of the operation logic of an interactive interface provided in this application;

[0049] Figure 6 is a schematic diagram of an application operation logic provided in this application;

[0050] Figure 7 is a schematic diagram of another application operation logic provided in this application;

[0051] Figure 8 is a schematic diagram of another application operation logic provided in this application;

[0052] Figure 9 is a structural schematic diagram of a human-computer interaction device provided in this application;

[0053] Figure 10 is a structural schematic diagram of another human-computer interaction device provided in this application;

[0054] Figure 11 is a schematic diagram of the structure of a computing device provided in this application;

[0055] Figure 12 is a schematic diagram of the structure of a computing device cluster provided in this application;

[0056] Figure 13 is a schematic diagram of a network connection structure between computing devices provided in this application. Detailed Implementation

[0057] The human-computer interaction method and apparatus provided in this application can be applied to extended reality scenarios. A brief introduction to the technologies that may be involved in this application is provided below.

[0058] (1) Cloud computing

[0059] Cloud computing is a type of distributed computing that refers to breaking down massive data processing programs into countless smaller programs through a network "cloud." These smaller programs are then processed and analyzed by a system composed of multiple servers to obtain results and return them to the user.

[0060] Cloud computing is based on networking and virtualization technologies. Through virtualization, cloud providers can run multiple virtual machines on a single physical server, each running its own operating system and applications. This allows users to access resources and services on these virtual machines via the internet without the need to purchase and maintain large amounts of hardware.

[0061] (2) Cloud Extended Reality

[0062] Extended Reality (XR) is a cloud-based extended reality application that transfers the XR experience from the local device to a cloud device for processing and delivery. It streams virtual reality, augmented reality, or mixed reality content to the terminal device by placing computation and rendering tasks on cloud servers, i.e., cloud devices, thus enabling users to experience XR without needing to render, store, and process large amounts of data locally on their terminal devices.

[0063] (3) Human-machine interface equipment

[0064] Human-Machine Interface (HMI) devices (or HMI protocols) are communication protocols based on the Universal Serial Bus (USB) used for data transmission between computers and input devices. HMI protocols define standard data formats, command structures, and transmission methods, enabling operating systems to recognize and maintain compatibility with input devices from different manufacturers.

[0065] (4) Inertial Measurement Unit

[0066] An inertial measurement unit (IMU) is a device that measures an object's three-axis attitude angles (or angular rates) and acceleration. Typically, an IMU contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometers detect the object's acceleration signals along the three independent axes of the carrier coordinate system, while the gyroscopes detect the carrier's angular velocity signals relative to the navigation coordinate system. By measuring the object's angular velocity and acceleration in three-dimensional space, the object's attitude can be calculated.

[0067] This application provides a human-computer interaction method, specifically a method for "converting user action data into HID data for application operation". A terminal device acquires user action data and converts it into human interface device (HID) data corresponding to the application. Then, the terminal device sends the HID data to a cloud device, whereby the HID data instructs the application to perform an HID data matching operation. Next, the terminal device receives a video stream from the cloud device, which is obtained through cloud rendering.

[0068] Based on the aforementioned human-computer interaction method, in XR scenarios, terminal devices can convert user action data into HID data corresponding to different applications, and then send the HID data to the cloud device. The application in the cloud device performs HID data matching, and the terminal device then receives the application's video stream sent by the cloud device. In this way, for any application adapted to or not adapted to XR, where there is a mapping relationship between user action data and HID data, the terminal device and the cloud device can collaboratively process user action data for application operation and cloud rendering output. This allows users to operate applications based on the human-computer interaction methods in XR scenarios, improving the applicability of XR.

[0069] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0070] Figure 1 is a schematic diagram of the architecture of a human-computer interaction system provided in this application. As shown in Figure 1, the human-computer interaction system (or XR system) 100 includes a cloud device 110, a terminal device 120, and an interaction device 130. Among them, the cloud device 110 is communicatively connected to the terminal device 120, and the terminal device 120 is communicatively connected to the interaction device 130.

[0071] The cloud device 110 can be obtained by virtualizing the hardware resources of one or more server clusters. Figure 2 is a schematic diagram of the structure of a cloud device provided in this application. As shown in Figure 2, the cloud device 110 includes a computing server cluster 111, a storage server cluster 112, a management server cluster 113, and a network device cluster 114. The computing server cluster 111, storage server cluster 112, and management server cluster 113 communicate with the terminal device 120 through the network device cluster 114. The computing server cluster 111, storage server cluster 112, and management server cluster 113 can also communicate with the terminal device 120 through the network device cluster 114.

[0072] The computing server cluster 111 includes one or more computing servers (two computing servers are shown in Figure 2, but it is not limited to two computing servers).

[0073] Computing servers, such as servers and desktop computers, serve as computing resources within cloud device 110. They are used to generate and allocate computing resources based on virtualization technology and user needs. At the hardware level, computing servers are equipped with processors and memory (not shown in Figure 2). The computing functions of the computing server are implemented by the processor running programs in memory. The computing server can also read / write data from various storage servers in the storage server cluster 111 according to user needs.

[0074] Storage server cluster 112 includes one or more storage servers (two storage servers are shown in Figure 2, but it is not limited to two storage servers).

[0075] Storage servers, as storage resources within cloud device 110, such as servers, desktop computers, or storage array controllers and hard disk enclosures, provide logical disk storage, semi-structured data storage, and integrated backup services for cloud virtual machines within computer system 100. In terms of hardware, storage servers include network interface cards (NICs), processors, and memory. The processor in the storage server processes data from outside the storage server. The NIC controls the access process to the memory, such as controlling address signals, data signals, and various command signals, enabling the storage server to provide the memory as a storage resource to users. Memory is used to store data and may include RAM and / or hard disks. RAM refers to internal memory that directly exchanges data with the processor; RAM can quickly read and write data at any time, serving as temporary data storage for the operating system or other running programs. Unlike RAM, hard disks are slower to read and write data and are typically used for persistent data storage.

[0076] The management server cluster 113 includes one or more management servers (two management servers are shown in Figure 2, but it is not limited to two management servers).

[0077] The management server is used to manage all computing services, shared storage, and network of the entire cloud device 113, and provides users or administrators with an application program interface (API) for managing the entire node. In this application, the cloud device 110 can provide the program product of this application to users by providing an accessible application program interface.

[0078] Network device cluster 114 includes one or more network devices, as shown in Figure 2. In this embodiment, network device cluster 114 includes multiple switches, gateways, or routers. Each computing server in computing server cluster 111 is connected via a switch. Each computing server in computing server cluster 111 and each storage server in storage server cluster 112 are each connected to a switch. Each computing server in computing server cluster 111, each storage server in storage server cluster 112, and each management server in management server cluster 113 are each connected to a switch.

[0079] Optionally, the number and type of network devices included in the network device cluster 114 can be adjusted according to the needs of the cloud device 110. The network devices can be switches, routers or gateways with different functions. Taking a switch as an example, a switch can be a core switch for managing a specific network segment, an internal or external switching network segment switch, a storage network segment switch or a management network segment switch.

[0080] Terminal device 120 includes one or more terminal devices (three terminal devices are shown in Figures 1 and 2, but not limited to two terminal devices). The terminal devices contain the interfaces and applications required for accessing cloud device 110.

[0081] The terminal device is used to receive user operation commands, collaborate with cloud device 110 to process application data, and display the video stream of the application output by cloud device 110 to the user. The terminal device can also be called a terminal, terminal node, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device can be a wired terminal such as a mobile phone, tablet, computer with wireless transceiver capabilities, virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless terminal in self-driving vehicles, wireless terminal in smart grids, wireless terminal in smart cities, wireless terminal in smart homes, etc. The embodiments of this application do not limit the specific technology or device form used in the terminal device; for example, terminal device 120 includes a television.

[0082] The interactive device 130 includes one or more motion-sensing devices (five IMUs are shown in Figure 1, but it is not limited to five IMUs). The motion-sensing devices contain protocol interfaces for communicating with the terminal device 120.

[0083] The motion sensing device is used to collect user motion data and send the user motion data to the terminal device 120. The user motion data can be the posture angle (or angular rate) and acceleration of the user's body parts.

[0084] It is worth noting that Figures 1 and 2 are merely schematic diagrams and should not be construed as limiting the scope of this application. The human-computer interaction system 100 may also include other devices, which are not shown in Figures 1 and 2.

[0085] Figure 3 is a hierarchical diagram of a human-computer interaction system provided in this application. As shown in Figure 3, the human-computer interaction system 100 can be logically divided into a terminal device side, an application side, and a cloud side.

[0086] The terminal device side can be supported by terminal device 120, and its functional modules may include: camera cloud uploading, sensor (such as IMU) access, sensor operation conversion, and sensor plug-in, etc.

[0087] Camera to the cloud is used to upload video streams captured by the camera to cloud device 110, so that cloud device 110 processes the video streams captured by the camera and overlays the camera's captured images with the application's display interface.

[0088] Sensor access is used to support communication between terminal device 120 and sensors, such as sensors transmitting user action data to terminal device 120.

[0089] Sensor operation conversion is used to support terminal device 120 in converting user action data into HID data corresponding to the application.

[0090] The sensor plugin is used to support the terminal device 120 in driving the sensor.

[0091] The application side can be supported collaboratively by cloud device 110 and terminal device 120, and its functional modules can include applications, application stores, etc. Cloud device 110 processes the application data and transmits the processed application video stream to terminal device 120, which then outputs the application video stream.

[0092] Applications can be home-oriented (to home, to H) applications, such as applications developed based on the XR platform, ecosystem partner applications, and existing applications on terminal devices 110.

[0093] The cloud side can be supported by cloud device 110, whose functional modules include cloud rendering engine, operation and maintenance, and infrastructure.

[0094] The cloud rendering engine includes the cloud terminal operating system (OS) kernel, collaboration engine, scheduling engine, open source project (Android Open Source Project, AOSP) management, image management, virtualization (such as ARM virtualization), etc.

[0095] Operations and maintenance (O&M) includes user management, authentication and authorization, O&M monitoring, O&M analysis, cloud machine management, security and trust, and application management.

[0096] The basic setup can be, as shown in Figure 2, a computing server cluster 111, a storage server cluster 112, a management server cluster 113, and a network device cluster 114, etc.

[0097] As one possible implementation, the cloud-side functional modules can be based on the human-computer interaction system 100 shown in Figure 1, implementing the functions of virtualized nodes based on Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS), and providing services to users through these virtualized nodes. These virtualized nodes can be nodes obtained by virtualizing the resources of the human-computer interaction system 100.

[0098] In possible embodiments of this application, the cloud device may refer to the cloud device 110 in FIG1 or the cloud side in FIG3, and the terminal device may refer to the terminal device 120 in FIG1 or the terminal device side in FIG3. Exemplarily, the human-computer interaction method provided in the embodiments of this application will be described below with reference to the interaction between the cloud device 110 and the terminal device 120 in FIG1.

[0099] It is worth noting that Figure 3 is only a schematic diagram and should not be construed as a limitation of this application. The human-computer interaction system 100 may also include other modules, which are not shown in Figure 3.

[0100] The steps of the human-computer interaction method provided in this application can be collaboratively executed by the cloud device 110, terminal device 120, and interaction device 130 of the aforementioned human-computer interaction system 100. Figure 4 is a flowchart illustrating a human-computer interaction method provided in this application. As shown in Figure 4, the human-computer interaction method provided in this embodiment may include the following steps 401-408.

[0101] Step 401: Interactive device 130 sends user action data to terminal device 120.

[0102] After collecting the user's action data, the interactive device 130 sends the user action data to the terminal device 120.

[0103] As one possible implementation, the interactive device 130 can send user action data of the application request to the terminal device 120 when the application is in the running state.

[0104] As one possible implementation, the interaction device 130 can be any device capable of converting the user's actions into electrical signals. The interaction device 130 can be an IMU, camera, displacement sensor, lidar, etc.

[0105] For example, if the interactive device 130 is an IMU, the user motion data can be posture data, i.e., three-axis posture angles or acceleration. Alternatively, if the interactive device 130 is a camera, the user motion data can be motion data, i.e., the position of various body parts of the user in the video stream captured by the camera.

[0106] Step 402: Terminal device 120 acquires user action data.

[0107] Terminal device 120 receives user action data sent by interactive device 130.

[0108] Step 403: The terminal device 120 converts the user action data into HID data corresponding to the application.

[0109] Terminal device 120 converts user action data into HID data according to the mapping relationship of the application. The mapping relationship can be different for different applications, that is, the mapping relationship (conversion method) between user action data and HID data is different for different applications.

[0110] As one possible implementation, the user action data includes attitude data transmitted from an IMU located at the user's first site. The terminal device 120 converts the attitude data into HID data corresponding to the application according to the first mapping relationship of the application.

[0111] Optionally, the first part can be any part such as the head, hand, foot, wrist, or ankle.

[0112] For example, in an XR scenario where target movement is controlled in an application, the first part is the head, and the attitude angle changes include pitch angle changes, yaw angle changes, or roll angle changes. The first HID data includes data generated by touching the screen or pressing a button to activate a first operation key. The operation corresponding to the first operation key includes moving the target forward, moving the target backward, moving the target left, moving the target right, moving the view to the left, or moving the view to the right.

[0113] Optionally, the first mapping relationship may include: when the attitude data represents the attitude angle change of the first part, the attitude data corresponds to the first HID data. The operation of matching the first HID data includes touch screen operation or button operation.

[0114] Depending on the application, the HID data corresponding to different attitude angles of the first part can also be different. For specific examples, please refer to Figures 6 and 7 and related content, which will not be repeated here.

[0115] As one possible implementation, the user action data includes attitude data transmitted from an IMU located at a second part of the user's body. The terminal device 120 converts the attitude data into HID data corresponding to the application according to the second mapping relationship of the application.

[0116] Optionally, the first part can be any part such as the head, hand, foot, wrist, or ankle.

[0117] For example, in an XR scenario where a target in a control application performs a fitness movement, the second body part is the limbs, such as the left wrist, right wrist, left ankle, and right ankle, and the posture data includes acceleration. The operation matched by the second HID data includes touch screen operation or button operation.

[0118] Optionally, the second mapping relationship may include: when the attitude data represents the acceleration change of the second part, the attitude data corresponds to the second HID data. The operation of matching the second HID data includes touch screen operation or button operation.

[0119] Depending on the application, the HID data corresponding to different accelerations in the second part may also be different. For specific examples, please refer to Figure 8 and related content, which will not be elaborated here.

[0120] As one possible implementation, user action data includes action data transmitted from a camera set on terminal device 120. When the action data indicates that the user's third body part is performing a specified action at a first position within the camera's field of view, the action data corresponds to third HID data. The operation matching the third HID data includes data generated by clicking a second operation key, which is located at a second position on the application's display interface, and this second position is associated with the first position.

[0121] Optionally, the association between the second position and the first position may mean that when the application's display interface is superimposed on the camera's captured image, i.e., the camera's video stream, the second position and the first position coincide in the screen displayed on the terminal device 120.

[0122] Depending on the application, the HID data corresponding to the specified action performed at different locations in the third part may also be different. For specific examples, please refer to Figure 5 and related content, which will not be elaborated here.

[0123] Step 404: Terminal device 120 sends HID data to cloud device 110.

[0124] Step 405: Cloud device 110 receives HID data sent by terminal device 120.

[0125] Step 406: Cloud device 110 inputs HID data into the application.

[0126] Cloud device 110 inputs HID data into the application to instruct the application to perform an HID data matching operation.

[0127] As one possible implementation, different applications have different HID operation logic. After the cloud device 110 inputs the HID data into the application, it instructs the application to perform the operation that matches the HID data.

[0128] Optionally, the operation that matches the HID data can be a target movement operation, a target limb movement operation, an option click operation of the operating system of the terminal device 120, etc.

[0129] Step 407: Cloud device 110 sends the application's video stream to terminal device 120.

[0130] Cloud device 110 performs cloud rendering on the video stream that performs HID data matching operations on the application, and sends the application's video stream to cloud device 120.

[0131] Step 408: Terminal device 120 receives the video stream of the application sent by cloud device 110.

[0132] After receiving the video stream from the cloud device 110, the terminal device 120 can process or display the video stream.

[0133] Based on the aforementioned human-computer interaction method, in XR scenarios, terminal devices can convert user action data into HID data corresponding to different applications, and then send the HID data to the cloud device. The application in the cloud device performs HID data matching, and the terminal device then receives the application's video stream sent by the cloud device. In this way, for any application adapted to or not adapted to XR, where there is a mapping relationship between user action data and HID data, the terminal device and the cloud device can collaboratively process user action data for application operation and cloud rendering output. This allows users to operate applications based on the human-computer interaction methods in XR scenarios, improving the applicability of XR.

[0134] The human-computer interaction method provided in this application has been described in general with reference to Figure 4 above. Next, with reference to Figures 5-8, different mapping relationships between user action data and HID operations in the application of the human-computer interaction method will be illustrated by example.

[0135] On the display interface of terminal device 120, users can perform operations such as setting up terminal device 120 and launching applications through the interactive interface. The camera of terminal device 120 collects the user's motion data, and terminal device 120 converts the motion data of the user's third body part into third HID data to operate the interactive interface.

[0136] As shown in Figure 5, the interactive interface operation logic may include the following steps:

[0137] Step 501: Terminal device 120 acquires motion data captured by the camera.

[0138] The terminal device 120 acquires motion data from the built-in camera or receives motion data from an external camera connected to the terminal device 120.

[0139] Step 502: Terminal device 120 identifies the third part in the camera's captured image.

[0140] Terminal device 120 identifies a third part in the camera's captured image based on image recognition principles. The image recognition can be implemented using any image recognition algorithm, such as deep learning or template matching, which will not be elaborated upon here.

[0141] Step 503: The terminal device 120 outputs the overlay interface of the application display interface and the camera shooting screen.

[0142] Terminal device 120 overlays the application's display interface and the camera's captured image, outputting an overlaid display interface. The display interface can be the user interface of an application such as a game, or the user interface of a system application such as settings or a user interface.

[0143] As one possible implementation, the transparency and layering of the application's display interface and the camera's captured image in the overlay interface can be flexibly adjusted.

[0144] Optionally, both the application's display interface and the camera's captured image are semi-transparent, or the application's display interface has a higher transparency than the camera's captured image, or the application's display interface has a lower transparency than the camera's captured image.

[0145] As one possible implementation, a first identifier is displayed on the third part of the camera's captured image in the overlay interface, and the first identifier is used to indicate the position of the third part in the overlay interface.

[0146] Alternatively, the logo can be a highlighted border, a semi-transparent halo, etc.

[0147] Step 504: When the motion data indicates that the third part is performing a specified action at the first position of the camera's captured image, the terminal device 120 converts the motion data into third HID data.

[0148] When the motion data indicates that the user's third part is in the first position of the camera's captured image and performs a specified action, the terminal device 120 converts the motion data into third HID data of the interactive interface.

[0149] As one possible implementation, when the terminal device 120 is a non-touchscreen device, the third HID data can be data generated by a controller such as a remote control clicking the second operation key.

[0150] As one possible implementation, when the terminal device 120 is a touch screen device, the third HID data can be the data generated by the touch screen clicking the second operation key.

[0151] Optionally, the second operation key can be a clickable graphic element of the interactive interface, located in the second position of the application's display interface, with the first position and the second position being related, that is, the first position and the second position overlap in the overlay screen.

[0152] As one possible implementation, the specified action could be clenching a fist, opening a palm, etc.

[0153] Step 505: Terminal device 120 inputs the third HID data into the application.

[0154] Terminal device 120 inputs third HID data into its local application, such as an interactive interface, to perform click operations on clickable elements of the interactive interface within the application.

[0155] In the above process, users can launch the application by clicking on the clickable graphic element through human-computer interaction. After the application is launched, users can also perform in-application operations based on the human-computer interaction method of this application.

[0156] As shown in Figure 6, the application operation logic may include the following steps:

[0157] Step 601: Terminal device 120 acquires the attitude data collected by the IMU.

[0158] Terminal device 120 acquires attitude data transmitted from the IMU located at the user's first location.

[0159] As one possible implementation, the first part is the head. The IMU collects the angle and pose values ​​of the user's head in three directions, or attitude angle changes (including pitch angle changes, yaw angle changes, or roll angle changes). The IMU is set on the hat or bracket of the user's head, and the idle posture of the user's head is used as the starting state of the action.

[0160] Step 602: The terminal device 120 converts the attitude data into first HID data corresponding to the application according to the first mapping relationship of the application.

[0161] According to the steps shown in Figure 5, the terminal device 120 opens the first mapping relationship of the application and converts the attitude data into the first HID data corresponding to the application.

[0162] As one possible implementation, the user's head moves based on pitch angle. Changes in the attitude angle of the attitude data indicate the user is looking down. In the first mapping relationship, the corresponding first HID data is data generated by touching the screen or pressing a button to activate the first operation key. The operation corresponding to the first operation key in the application is moving the target forward. Conversely, changes in the attitude angle of the attitude data indicate the user is looking up. In the first mapping relationship, the corresponding first HID data is data generated by touching the screen or pressing a button to activate the first operation key. The operation corresponding to the first operation key in the application is moving the target backward.

[0163] As one possible implementation, the user's head moves based on yaw angle. Changes in the attitude angle of the attitude data indicate that the user tilts their head to the left. In the first mapping relationship, the corresponding first HID data is data generated by touching the screen or pressing a button to activate the first operation key. In the application, the corresponding operation of the first operation key is to move the target in that direction. Alternatively, changes in the attitude angle of the attitude data indicate that the user tilts their head to the right. In the first mapping relationship, the corresponding first HID data is data generated by touching the screen or pressing a button to activate the first operation key. In the application, the corresponding operation of the first operation key is to move the target to the right.

[0164] As one possible implementation, the user's head moves based on a roll angle. Changes in the posture angle of the posture data represent the user turning their head left and then returning to a neutral position. In the first mapping relationship, the corresponding first HID data is the data generated by touching the screen or pressing a button to activate the first operation key. In the application, the corresponding operation for this first operation key is moving the view to the left. Similarly, changes in the posture angle of the posture data represent the user turning their head right and then returning to a neutral position. In the first mapping relationship, the corresponding first HID data is the data generated by touching the screen or pressing a button to activate the first operation key. In the application, the corresponding operation for this first operation key is moving the view to the right.

[0165] Optionally, moving the field of view to the left or right is achieved by rotating the field of view camera horizontally to the left or right at a fixed angle. The field of view can also be returned to center after moving left or right.

[0166] The first operation key mentioned above is only an example given in the embodiments of this application. Multiple operations in the above application can correspond to different operation keys, that is, different operation keys are respectively used for moving the target forward, backward, left, and right, and for moving the view left and right.

[0167] After step 602, the terminal device 120 can work with the cloud device 110 to process and output the video stream of the application. For details, please refer to steps 404-408 in Figure 4, which will not be repeated here.

[0168] The application operation logic shown in Figure 6 above can be used in scenarios where users achieve XR control of the application through head movements. Different application operation logics can be used in different applications, such as achieving XR control of the target (e.g., vehicle) in the application through limb movements.

[0169] As shown in Figure 7, another application operation logic may include the following steps:

[0170] Step 701: Terminal device 120 acquires the attitude data collected by the IMU.

[0171] Terminal device 120 acquires attitude data transmitted from the IMU located at the second part of the user's device.

[0172] As one possible implementation, the second part is the limbs. The IMU collects the posture data, or acceleration changes, of the user's limbs, namely the left wrist, right wrist, left ankle, and right ankle. The IMU is set on the user's gloves, shoes, socks, etc., and the static posture of the user's limbs is used as the starting state of the action.

[0173] Step 702: The terminal device 120 converts the attitude data into second HID data corresponding to the application according to the second mapping relationship of the application.

[0174] According to the steps shown in Figure 5, the terminal device 120 opens the second mapping relationship of the application and converts the attitude data into the second HID data corresponding to the application.

[0175] As one possible implementation, the user's action is to jump, and the acceleration change of the posture data represents the user's jump. In the second mapping relationship, the corresponding second HID data is the data generated by touching the screen or clicking the second operation key. The operation corresponding to the second operation key in the application is target braking.

[0176] As one possible implementation, the user's action is the movement of the left foot, and the acceleration change of the posture data represents the movement of the user's left foot. In the second mapping relationship, the corresponding second HID data is the data generated by the touch screen or button click of the second operation key. The operation corresponding to the second operation key in the application is that the target turns to the left.

[0177] As one possible implementation, the user's action is the movement of the right foot, and the acceleration change of the posture data represents the movement of the user's right foot. In the second mapping relationship, the corresponding second HID data is the data generated by touching the screen or clicking the second operation key. The operation corresponding to the second operation key in the application is that the target turns to the right.

[0178] As one possible implementation, the user's action is to squat down, and the acceleration change in the posture data indicates that the user is squatting down. In the second mapping relationship, the corresponding second HID data is the data generated by touching the screen or clicking the second operation key. The operation corresponding to the second operation key in the application is target acceleration.

[0179] The second operation key mentioned above is only one example given in the embodiments of this application. Multiple operations in the above application can correspond to different operation keys, that is, braking, turning left, turning right, and accelerating for the target correspond to different operation keys.

[0180] After step 702, the terminal device 120 can work with the cloud device 110 to process and output the video stream of the application. For details, please refer to steps 404-408 in Figure 4, which will not be repeated here.

[0181] The application operation logic shown in Figures 6 and 7 above can be applied to XR target control scenarios in non-touch or touch devices. In applications such as XR fitness, users can also achieve synchronization between the target (3D character) and user actions through the application operation logic of the human-computer interaction method in this application embodiment.

[0182] As shown in Figure 8, another type of application operation logic may include the following steps:

[0183] Step 801: Terminal device 120 acquires the attitude data collected by the IMU.

[0184] Terminal device 120 acquires attitude data transmitted from the IMU located at the second part of the user's device.

[0185] As one possible implementation, the second part is the limbs. The IMU collects the posture data, or acceleration changes, of the user's limbs, namely the left wrist, right wrist, left ankle, and right ankle. The IMU is set on the user's gloves, shoes, socks, etc., and the static posture of the user's limbs is used as the starting state of the action.

[0186] Step 802: The terminal device 120 converts the attitude data into second HID data corresponding to the application according to the second mapping relationship of the application.

[0187] According to the steps shown in Figure 5, the terminal device 120 opens the second mapping relationship of the application and converts the attitude data into the second HID data corresponding to the application.

[0188] As one possible implementation, the user's left wrist movement and the acceleration change of the posture data represent the user's left wrist movement. In the second mapping relationship, the corresponding second HID data is the data generated by the touch screen or button click of the second operation key. The operation corresponding to the second operation key in the application is the target left wrist movement.

[0189] As one possible implementation, the user's right wrist movement and the acceleration change of the posture data represent the user's right wrist movement. In the second mapping relationship, the corresponding second HID data is the data generated by the touch screen or button click of the second operation key. The operation corresponding to the second operation key in the application is the target right wrist movement.

[0190] As one possible implementation, the user's left ankle moves, and the acceleration change of the posture data represents the user's left ankle movement. In the second mapping relationship, the corresponding second HID data is the data generated by the touch screen or button click of the second operation key. The operation corresponding to the second operation key in the application is the target left ankle movement.

[0191] As one possible implementation, the user's right ankle moves, and the acceleration change of the posture data represents the user's right ankle movement. In the second mapping relationship, the corresponding second HID data is the data generated by the touch screen or button click of the second operation key. The operation corresponding to the second operation key in the application is the target right ankle movement.

[0192] The second operation key mentioned above is only one example given in the embodiments of this application. Multiple operations in the above application can correspond to different operation keys, that is, different operation keys are respectively assigned to the movement of the target's limbs.

[0193] The applications in Figures 6-8 above can be the same or different applications, and the applications have a preset mapping relationship.

[0194] After step 802, the terminal device 120 can work with the cloud device 110 to process and output the video stream of the application. For details, please refer to steps 404-408 in Figure 4, which will not be repeated here.

[0195] In a possible embodiment, the cloud device 110 can also determine the accuracy of the user's action data based on the degree of matching between the input features of multiple input HID data and preset data features. Input features include the input order or input interval of multiple HID data. Thus, the cloud device 110 can score the accuracy of the user's actions in the application.

[0196] To complement the human-computer interaction method shown in Figure 4 above, this application also provides a human-computer interaction device 900, which can be used to implement the functions of the terminal device 120 in the human-computer interaction method shown in Figure 4 above. As shown in Figure 9, the human-computer interaction device 900 includes a transceiver module 910 and a processing module 920.

[0197] The transceiver module 910 is used to acquire user action data. For example, the transceiver module 910 is used to execute step 402 as shown in Figure 4.

[0198] Processing module 920 is used to convert the user action data into HID data corresponding to the application. For example, processing module 920 is used to execute step 403 as shown in FIG4.

[0199] The transceiver module 910 is also used to send the HID data to the cloud device; the HID data is used to instruct the application to perform the HID data matching operation. For example, the transceiver module 910 is used to perform step 404 as shown in FIG4.

[0200] As one possible implementation, the user motion data includes attitude data transmitted from an inertial measurement unit (IMU) located at a first part of the user's body. The processing module 920 is specifically configured to: convert the attitude data into HID data corresponding to the application, based on a first mapping relationship of the application.

[0201] Optionally, the first mapping relationship includes: when the attitude data represents the attitude angle change of the first part, the attitude data corresponds to the first HID data; the operation matched by the first HID data includes touch screen operation or button operation.

[0202] Optionally, the attitude angle change includes pitch angle change, yaw angle change, or roll angle change, and the first HID data includes data generated by touching the screen or clicking the first operation key. The operation corresponding to the first operation key includes moving the target forward, moving the target backward, moving the target left, moving the target right, moving the field of view left, or moving the field of view right.

[0203] As one possible implementation, user motion data includes posture data transmitted from an IMU located at a second part of the user's body. The processing module is specifically used to: convert the posture data into HID data corresponding to the application, based on a second mapping relationship of the application.

[0204] Optionally, the second mapping relationship includes: when the attitude data represents the acceleration change of the second part, the attitude data corresponds to the second HID data; the operation matched by the second HID data includes touch screen operation or button operation.

[0205] Optionally, the acceleration change includes an acceleration value greater than or equal to a preset threshold, and the second HID data includes data generated by touching the screen or clicking the second operation key.

[0206] As one possible implementation, the user action data includes action data captured by the camera. When the action data indicates that the user's third body part is performing a specified action at a first position within the camera's view, the action data corresponds to third HID data; the operation matched by the third HID data includes data generated by clicking a second operation key, the second operation key being located at a second position on the application's display interface, the second position being associated with the first position.

[0207] As one possible implementation, the processing module 920 is further configured to: output an overlay interface of the application's display interface and the camera's captured image; the overlay interface includes a first identifier, which is used to indicate a second position in the display interface that coincides with the first position.

[0208] Both the transceiver module 910 and the processing module 920 can be implemented in software or in hardware. For example, the implementation of the transceiver module 910 will be described below. Similarly, the implementation of the processing module 920 can be referenced from the implementation of the transceiver module 910.

[0209] As an example of a software functional unit, the transceiver module 910 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the transceiver module 910 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0210] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0211] As an example of a hardware functional unit, the transceiver module 910 may include at least one computing device, such as a server. Alternatively, the transceiver module 910 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0212] The transceiver module 910 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the transceiver module 910 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the transceiver module 910 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0213] It should be noted that, in other embodiments, either the transceiver module 910 or the processing module 920 can be used to execute any step in the human-computer interaction method. The steps implemented by the transceiver module 910 and the processing module 920 can be specified as needed. The transceiver module 910 and the processing module 920 respectively implement different steps in the human-computer interaction method to realize all the functions of the human-computer interaction device 900.

[0214] To complement the human-computer interaction method shown in Figure 4 above, this application also provides a human-computer interaction device 1000, which can be used to implement the functions of the cloud device 110 in the human-computer interaction method shown in Figure 4 above. As shown in Figure 10, the human-computer interaction device 1000 includes:

[0215] The transceiver module 1010 is used to receive HID data sent by the terminal device; the HID data is obtained by converting user action data, and the conversion method between the user action data and the HID data corresponds to the application. For example, the transceiver module 1010 is used to execute step 405 as shown in Figure 4.

[0216] Processing module 1020 is used to input HID data into the application; the HID data is used to instruct the application to perform an operation of matching the HID data. For example, processing module 1020 is used to execute step 406 as shown in FIG4.

[0217] The transceiver module 1010 is also used to output the video stream of the application to the terminal device; the video stream is obtained by cloud rendering by the cloud device. For example, the transceiver module 1010 is used to execute step 407 as shown in FIG4.

[0218] As one possible implementation, the processing module 1020 is further configured to: determine the accuracy of the action corresponding to the user action data based on the degree of matching between the input features of the multiple input HID data and the preset data features; the input features include the input order or input interval of the multiple HID data.

[0219] Both the transceiver module 1010 and the processing module 1020 can be implemented in software or in hardware. For example, the implementation of the transceiver module 1010 will be described below. Similarly, the implementation of the processing module 1020 can refer to the implementation of the transceiver module 1010.

[0220] As an example of a software functional unit, the transceiver module 1010 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the transceiver module 1010 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0221] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0222] As an example of a hardware functional unit, the transceiver module 1010 may include at least one computing device, such as a server. Alternatively, the transceiver module 1010 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0223] The transceiver module 1010 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the transceiver module 1010 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the transceiver module 1010 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0224] It should be noted that, in other embodiments, either the transceiver module 1010 or the processing module 1020 can be used to execute any step in the human-computer interaction method. The steps implemented by the transceiver module 1010 and the processing module 1020 can be specified as needed. By implementing different steps in the human-computer interaction method through the transceiver module 1010 and the processing module 1020 respectively, all functions of the human-computer interaction device 1000 can be realized.

[0225] This application also provides a computing device 1100. As shown in FIG11, the computing device 1100 includes: a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. The processor 1104, the memory 1106, and the communication interface 1108 communicate with each other via the bus 1102. The computing device 1100 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1100.

[0226] Bus 1102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 11, but this does not imply that there is only one bus or one type of bus. Bus 1102 can include pathways for transmitting information between various components of computing device 1100 (e.g., memory 1106, processor 1104, communication interface 1108).

[0227] The processor 1104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0228] The memory 1106 may include volatile memory, such as random access memory (RAM). The processor 1104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0229] The memory 1106 stores executable program code, and the processor 1104 executes the executable program code to implement the functions of the various modules included in the aforementioned human-computer interaction device 900 or human-computer interaction device 1000, thereby realizing the human-computer interaction method. That is, the memory 1106 stores instructions for executing the human-computer interaction method.

[0230] Alternatively, the memory 1106 may store executable code, which the processor 1104 executes to implement the functions of the aforementioned human-computer interaction device 900 or human-computer interaction device 1000, thereby realizing the human-computer interaction method. That is, the memory 1106 stores instructions for executing the human-computer interaction method.

[0231] The communication interface 1108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1100 and other devices or communication networks.

[0232] Considering that the human-computer interaction method provided in this application is applied in a computer system 100, the infrastructure of the computer system 100, such as the computing server cluster 111 and the storage server cluster 112, typically includes multiple computing devices. Therefore, this application also provides a computing device cluster. This computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0233] As shown in Figure 12, the computing device cluster includes at least one computing device 1100. The memory 1106 of one or more computing devices 1100 in the computing device cluster may store the same instructions for executing human-computer interaction methods.

[0234] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing human-computer interaction methods. In other words, a combination of one or more computing devices 1100 can jointly execute instructions for executing human-computer interaction methods.

[0235] It should be noted that the memory 1106 in different computing devices 1100 within the computing device cluster can store different instructions, which are used to execute parts of the functions of the human-computer interaction device 900 or the human-computer interaction device 1000, respectively. That is, the instructions stored in the memory 1106 of different computing devices 1100 can implement the functions of one or more modules included in the human-computer interaction device 900 or the human-computer interaction device 1000.

[0236] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 13 illustrates one possible implementation. As shown in Figure 13, two computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 1106 in computing device 1100A stores instructions for executing the functions of one or more modules of transceiver module 910 and processing module 920. Figure 13 uses the example of the memory 1106 in computing device 1100A storing instructions for executing the functions of transceiver module 910. Similarly, the memory 1106 in computing device 1100B stores instructions for executing the functions of one or more modules of transceiver module 910 and processing module 920. Figure 13 uses the example of the memory 1106 in computing device 1100B storing instructions for executing the functions of processing module 920.

[0237] It should be understood that the functions of computing device 1100A shown in Figure 13 can also be performed by multiple computing devices 1100. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices 1100.

[0238] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any available medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to perform the human-computer interaction method shown in FIG4, or the steps performed by the cloud device 110 or terminal device 120 in the human-computer interaction method shown in FIG4.

[0239] This application embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the steps of the human-computer interaction method shown in FIG. 4, or the steps performed by the cloud device 110 or the terminal device 120 in the human-computer interaction method shown in FIG. 4.

[0240] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0241] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0242] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0243] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0244] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0245] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0246] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0247] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0248] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0249] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A human-computer interaction method, characterized in that, A terminal device applied in an extended reality (XR) system, the XR system further including a cloud device connected to the terminal device, the method comprising: Obtain user action data; The user action data is converted into HID data of the human-machine interface device corresponding to the application; The HID data is sent to the cloud device; the HID data is used to instruct the application to perform an operation of matching the HID data. Receive the video stream of the application sent by the cloud device; the video stream is obtained by the cloud device performing cloud rendering.

2. The method according to claim 1, characterized in that, The user motion data includes attitude data transmitted from an inertial measurement unit (IMU) located at the user's first body part. Converting the user motion data into HID data corresponding to the application includes: Based on the first mapping relationship of the application, the attitude data is converted into HID data corresponding to the application.

3. The method according to claim 2, characterized in that, The first mapping relationship includes: When the attitude data represents a change in the attitude angle of the first part, the attitude data corresponds to first HID data; the operation matched by the first HID data includes touch screen operation or button operation.

4. The method according to claim 3, characterized in that, The attitude angle changes include pitch angle changes, yaw angle changes, or roll angle changes. The first HID data includes data generated by touching the screen or clicking the first operation key. The operation corresponding to the first operation key includes moving the target forward, moving the target backward, moving the target left, moving the target right, moving the field of view left, or moving the field of view right.

5. The method according to claim 1, characterized in that, User motion data includes gesture data transmitted from an IMU located at a second part of the user's body. Converting the user motion data into HID data corresponding to the application includes: Based on the second mapping relationship of the application, the attitude data is converted into HID data corresponding to the application.

6. The method according to claim 5, characterized in that, The second mapping relationship includes: When the attitude data represents the acceleration change of the second part, the attitude data corresponds to the second HID data; the operation matched by the second HID data includes touch screen operation or button operation.

7. The method according to claim 6, characterized in that, The acceleration change includes acceleration values ​​greater than or equal to a preset threshold, and the second HID data includes data generated by touching the screen or clicking the second operation key.

8. The method according to claim 1, characterized in that, The user action data includes action data captured by the camera, and the step of converting the user action data into HID data corresponding to the application includes: When the action data indicates that the user's third part is performing a specified action at a first position in the camera's captured image, the action data corresponds to third HID data; the operation matched by the third HID data includes data generated by clicking a second operation key, the second operation key being located at a second position on the application's display interface, and the second position being associated with the first position.

9. The method according to claim 8, characterized in that, The method further includes: Output an overlay interface of the application's display interface and the camera's captured image; the overlay interface includes a first identifier, which is used to indicate a second position in the display interface that coincides with the first position.

10. A human-computer interaction method, characterized in that, A cloud device used in an XR system, the XR system further including a terminal device connected to the cloud device, the method comprising: Receive HID data sent by the terminal device; the HID data is obtained by converting user action data, and the conversion method between the user action data and the HID data corresponds to the application; The HID data is input into the application; the HID data is used to instruct the application to perform an operation to match the HID data. The application's video stream is output to the terminal device; the video stream is obtained by cloud rendering performed by the cloud device.

11. The method according to claim 10, characterized in that, The method further includes: The accuracy of the action corresponding to the user action data is determined based on the degree of matching between the input features of multiple HID data and preset data features; the input features include the input order or input interval of the multiple HID data.

12. A human-computer interaction device, characterized in that, include: The send / receive module is used to acquire user action data; The processing module is used to convert the user action data into HID data corresponding to the application; The transceiver module is also used to send the HID data to the cloud device; the HID data is used to instruct the application to perform the HID data matching operation; The transceiver module is also used to receive the video stream of the application sent by the cloud device; the video stream is obtained by the cloud device through cloud rendering.

13. A human-computer interaction device, characterized in that, include: The transceiver module is used to receive HID data sent by the terminal device; The HID data is obtained by converting user action data, and the conversion method between the user action data and the HID data corresponds to the application. The processing module is used to input HID data into the application; the HID data is used to instruct the application to perform an operation of matching the HID data. The transceiver module is also used to output the video stream of the application to the terminal device; The video stream is obtained by cloud rendering on cloud devices.

14. A computing device, characterized in that, Including processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1-11.

15. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1-11.

16. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-11.

Images