Human-computer interaction method and device

Abstract

The invention provides a man-machine interaction method and device, and relates to the technical field of cloud computing. The method is applied to an XR system, and comprises the following steps: acquiring user action data by terminal equipment, and converting the user action data into HID data corresponding to an application; the terminal device sends HID data to the cloud device, wherein the HID data is used for indicating the application to execute an operation matched with the HID data; and the terminal equipment receives the video stream of the application sent by the cloud equipment, wherein the video stream is obtained by cloud rendering of the cloud equipment. Thus, when any application adapted to or not adapted to the XR has the mapping relationship between the user action data and the HID data, the terminal equipment and the cloud equipment can cooperatively process the user action data to perform application operation and cloud rendering output, so that the user can operate the application in a man-machine interaction mode based on the XR scene, and the applicability of the XR is improved.

Description

[0001] This application claims priority to the Chinese Patent Application No. 202410501234.8, filed on April 24, 2024, entitled "License Multiplexing Control Method Based on Cloud Technology", and the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of cloud computing technology, and in particular to a human-computer interaction method and device. BACKGROUND

[0003] Extended reality (XR) refers to the combination of all real and virtual environments generated by computer technology and wearable devices and human-computer interaction. In the XR scenario, a terminal device such as a television, a tablet computer, a mobile phone, a virtual reality (VR) all-in-one machine, etc. is usually used as an access side to process user input human-computer interaction data and interface display data output by the terminal device in cooperation with a cloud server.

[0004] However, for a series of XR applications of a certain manufacturer or for a certain XR application, each manufacturer or each XR application usually has its own dedicated interaction device. For applications that are not adapted to XR, users cannot use the human-computer interaction mode of the XR scenario to interact. Therefore, when using different XR applications or applications that are not adapted to XR, the dedicated XR interaction device has poor applicability in different XR applications or applications that are not adapted to XR. SUMMARY

[0005] The present application provides a human-computer interaction method and device, thereby improving the applicability of the human-computer interaction mode of the XR scenario in different applications.

[0006] In a first aspect, the present application provides a human-computer interaction method applied to a terminal device in an XR system, and the XR system further includes a cloud device connected to the terminal device. In the flow of the human-computer interaction method, the terminal device obtains user action data and converts the user action data into human interface device (HID) data corresponding to an application. Then, the terminal device sends the HID data to the cloud device, and the HID data is used to instruct the application to perform an operation matched with the HID data. Next, the terminal device receives a video stream of the application sent by the cloud device, and the video stream is obtained by cloud rendering of the cloud device.

[0007] Based on the above human-computer interaction method, in the XR scene, the terminal device can convert the user action data into HID data corresponding to the application, and then send the HID data to the cloud device. The application in the cloud device performs the operation of matching the HID data, and the terminal device receives the video stream of the application sent by the cloud device. In this way, any application that is adapted or not adapted to XR can be processed by the terminal device and the cloud device based on the mapping relationship between the user action data and the HID data to perform application operation and cloud rendering output, so that the user can operate the application based on the human-computer interaction mode of the XR scene, and the applicability of the XR is improved.

[0008] As a possible implementation manner, the user action data includes posture data, and the posture data is collected by an inertial measurement unit (IMU) arranged on a first part of the user and then transmitted to the terminal device. The terminal device converts the posture data into HID data corresponding to the application according to a first mapping relationship of the application.

[0009] Optionally, the first part of the user can be the head, hand, foot, wrist, ankle, or any other part. For example, the first part is the head.

[0010] Optionally, the first mapping relationship includes: in the case that the posture data represents a change in the posture angle of the first part, the posture data corresponds to first HID data. The operation matched by the first HID data includes a touch screen operation or a key operation.

[0011] For example, the change in the posture angle includes a change in the pitch angle, a change in the yaw angle, or a change in the roll angle. The first HID data includes data generated by clicking a first operation key position of the touch screen or the key. The operation corresponding to the first operation key position includes moving the target forward, moving the target backward, moving the target left, moving the target right, moving the field of view left, or moving the field of view right.

[0012] Based on the above implementation manner, the terminal device can convert the posture data when the user wears the IMU into first HID data corresponding to the application, and then operate the application based on the first HID data. Since the terminal device supports data transmission with different types of IMUs, the mapping relationship between the posture data and the first HID data can be adjusted according to different applications. The terminal device can realize XR human-computer interaction in different applications based on any type of IMU, avoiding the need for the user to use a matching special interaction device for different applications. In this way, the applicability of the XR is improved, and the hardware overhead of the interaction device of the XR is reduced.

[0013] As a possible implementation manner, the user action data includes posture data, the posture data being collected by an IMU arranged on the second part of the user and transmitted to the terminal device.

[0014] Optionally, the second part can be a head, a hand, a foot, a wrist, an ankle, or any part. For example, the second part is a hand or a foot.

[0015] Optionally, the second mapping relationship includes: in a case where the posture data represents an acceleration change of the second part, the posture data corresponds to second HID data. The operation matched by the second HID data includes a touch screen operation or a key operation.

[0016] For example, the acceleration change includes an acceleration value greater than or equal to a preset threshold, and the second HID data includes data generated by a touch screen or a key clicking a second operation key.

[0017] As a possible implementation manner, the user action data includes action data captured by a camera, the action data being collected by a camera connected to or built in the terminal device and transmitted to the terminal device.

[0018] Optionally, in a case where the action data represents that a third part of the user performs a specified action at a first position in a camera captured image, the action data corresponds to third HID data. The operation matched by the third HID data includes data generated by clicking a second operation key, the second operation key being at a second position of a display interface of the application, and the second position being associated with the first position.

[0019] In this way, the terminal device can recognize the action data of the user based on the camera captured image, convert the action data into the third HID data corresponding to the application, and perform an operation on the terminal device or the application with non-touch screen operation logic based on the action data of the user.

[0020] Optionally, the terminal device can further output a superimposed interface of the display interface of the application and the camera captured image. The superimposed image includes a first identifier, the first identifier being used to indicate a second position in the display interface that coincides with the first position.

[0021] In this way, when the user watches the display interface of the terminal device, the user can see the action of the user and the corresponding position of the third part on the display interface, which facilitates the user to perform an operation such as position adjustment of the third part or icon selection on the display interface, and improves the accuracy of XR interaction operation.

[0022] In a second aspect, the present application provides a human-computer interaction method, which is applied to a cloud device in an XR system, and the XR system further includes a terminal device connected with the cloud device. In the flow of the human-computer interaction method, the cloud device receives HID data sent by the terminal device, the HID data is converted from user action data, and the conversion mode of the user action data and the HID data corresponds to an application. Then, the cloud device inputs the HID data into the application to instruct the application to perform an operation matched with the HID data. Next, a video stream of the application is output to the terminal device, and the video stream is obtained by cloud rendering of the cloud device.

[0023] Based on the above human-computer interaction method, in the XR scene, the cloud device is used as an application container, and in the scene where the terminal device has limited computing power, the cloud rendering scheme is used to realize the output of the application video stream after the XR human-computer interaction, so that different terminal devices can cooperate with the cloud device to realize the XR human-computer interaction for different applications, improve the applicability of the XR, and reduce the hardware cost of the user in the XR scene.

[0024] As a possible implementation manner, the cloud device determines the accuracy of the action corresponding to the user action data according to the matching degree of the input features of the input multiple HID data and the preset data features. The input features include the input order or input interval of the multiple HID data. In this way, the accuracy of the user action is evaluated, an evaluation mechanism is introduced, and the interactivity of the user and the application is improved.

[0025] In a third aspect, the present application provides a human-computer interaction device, which includes a transceiver module and a processing module. The transceiver module is used to obtain user action data. The processing module is used to convert the user action data into HID data corresponding to an application. The transceiver module is further used to send the HID data to a cloud device; the HID data is used to instruct the application to perform an operation matched with the HID data. The transceiver module is further used to receive a video stream of the application sent by the cloud device; the video stream is obtained by cloud rendering of the cloud device.

[0026] As a possible implementation manner, the user action data includes attitude data transmitted by an inertial sensor (IMU) arranged at a first part of the user. The processing module is specifically used to: convert the attitude data into HID data corresponding to the application according to a first mapping relationship of the application.

[0027] Optionally, the first mapping relationship includes: in a case where the attitude data represents a change in the attitude angle of the first part, the attitude data corresponds to first HID data; and the operation matched with the first HID data includes a touch screen operation or a key operation.

[0028] Optionally, the attitude angle change includes a pitch angle change, a yaw angle change, or a roll angle change, the first HID data includes data generated by a touch screen or a key clicking a first operation key position, and the operation corresponding to the first operation key position includes target forward movement, target backward movement, target leftward movement, target rightward movement, field of view leftward movement, or field of view rightward movement.

[0029] As a possible implementation, the user action data includes attitude data transmitted by an IMU arranged at a second part of the user. The processing module is specifically configured to: convert the attitude data into HID data corresponding to the application according to a second mapping relationship of the application.

[0030] Optionally, the second mapping relationship includes: in a case where the attitude data indicates an acceleration change of the second part, the attitude data corresponds to second HID data; and the operation matched by the second HID data includes a touch screen operation or a key operation.

[0031] Optionally, the acceleration change includes an acceleration value greater than or equal to a preset threshold, and the second HID data includes data generated by a touch screen or a key clicking a second operation key position.

[0032] As a possible implementation, the user action data includes action data captured by a camera. In a case where the action data indicates that a third part of the user performs a specified action at a first position in a camera captured image, the action data corresponds to third HID data; and the operation matched by the third HID data includes data generated by clicking a second operation key position, the second operation key position being at a second position of a display interface of the application, the second position being associated with the first position.

[0033] As a possible implementation, the processing module is further configured to: output an overlay interface of the display interface of the application and the camera captured image; and the overlay interface includes a first identifier, the first identifier being used to indicate the second position in the display interface that coincides with the first position.

[0034] As a possible implementation, the human-computer interaction device can further include other modules that perform the operation steps of the human-computer interaction method of the first aspect.

[0035] In a fourth aspect, the present application provides a human-computer interaction device, comprising a transceiver module and a processing module. The transceiver module is configured to receive HID data sent by a terminal device, wherein the HID data is converted from user action data, and the conversion manner of the user action data and the application corresponds to each other. The processing module is configured to input the HID data into the application, wherein the HID data is used to instruct the application to perform an operation matched with the HID data. The transceiver module is further configured to output a video stream of the application to the terminal device, wherein the video stream is obtained by cloud rendering of a cloud device.

[0036] As a possible implementation manner, the processing module is further configured to determine the accuracy of the corresponding action of the user action data according to a matching degree of input features of the input HID data and preset data features, wherein the input features include input sequences or input intervals of the HID data.

[0037] As a possible implementation manner, the human-computer interaction device can further comprise other modules for performing operation steps of the human-computer interaction method of the second aspect.

[0038] In a fifth aspect, a computing device is provided, comprising a processor and a memory. The processor of the computing device is configured to execute instructions stored in the memory of the computing device, so that the computing device performs the human-computer interaction method of any possible implementation manner of the first aspect.

[0039] In a sixth aspect, a computing device is provided, comprising a processor and a memory. The processor of the computing device is configured to execute instructions stored in the memory of the computing device, so that the computing device performs the human-computer interaction method of any possible implementation manner of the second aspect.

[0040] In a seventh aspect, a computer program product is provided. The computer program product comprises computer programs or instructions, which, when executed on a computer, cause the computer to perform the human-computer interaction method of any possible implementation manner of the first aspect.

[0041] In an eighth aspect, a computer program product is provided. The computer program product comprises computer programs or instructions, which, when executed on a computer, cause the computer to perform the human-computer interaction method of any possible implementation manner of the second aspect.

[0042] In a ninth aspect, a computer readable storage medium is provided. The readable storage medium comprises computer programs or instructions, which, when executed on a computer, cause the computer to perform the human-computer interaction method of any possible implementation manner of the first aspect.

[0043] In a tenth aspect, a computer readable storage medium is provided. The readable storage medium includes a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is caused to perform the human-computer interaction method of any possible implementation of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 An architecture schematic diagram of a human-computer interaction system provided by the present application is provided.

[0045] Figure 2 A structure schematic diagram of a cloud device provided by the present application is provided.

[0046] Figure 3 A hierarchical schematic diagram of a human-computer interaction system provided by the present application is provided.

[0047] Figure 4 A flow schematic diagram of a human-computer interaction method provided by the present application is provided.

[0048] Figure 5 An interactive interface operation logic schematic diagram provided by the present application is provided.

[0049] Figure 6 An application operation logic schematic diagram provided by the present application is provided.

[0050] Figure 7 Another application operation logic schematic diagram provided by the present application is provided.

[0051] Figure 8 Still another application operation logic schematic diagram provided by the present application is provided.

[0052] Figure 9 A structure schematic diagram of a human-computer interaction device provided by the present application is provided.

[0053] Figure 10 A structure schematic diagram of another human-computer interaction device provided by the present application is provided.

[0054] Figure 11 A structure schematic diagram of a computing device provided by the present application is provided.

[0055] Figure 12 A structure schematic diagram of a computing device cluster provided by the present application is provided.

[0056] Figure 13 A structure schematic diagram of a connection between computing devices through a network provided by the present application is provided. DETAILED DESCRIPTION

[0057] The human-computer interaction method and device provided by the embodiments of the present application can be applied in the scene of extended reality, and the technologies possibly involved in the present application are briefly introduced.

[0058] (1) Cloud computing

[0059] Cloud computing is a type of distributed computing, which refers to the use of a network of computers to perform large-scale data processing tasks. Instead of using a single computer, cloud computing breaks down the task into smaller parts and distributes them across multiple servers. The results are then returned to the user after processing and analysis.

[0060] The principle of cloud computing is based on network and virtualization technology. Through virtualization technology, cloud computing providers can run multiple virtual machines on a physical server, each of which can independently run different operating systems and applications. In this way, users can access resources and services on these virtual machines through the Internet without the need to purchase and maintain a large number of hardware devices.

[0061] (2) Cloud extended reality

[0062] Cloud extended reality is a type of extended reality application based on cloud computing technology, which transfers XR experiences from local devices to cloud devices for processing and delivery. It achieves this by placing computing and rendering tasks on cloud servers, or cloud devices, and streaming virtual reality, augmented reality, or mixed reality content to terminal devices, allowing users to experience XR without the need for local rendering, storage, and processing of large amounts of data.

[0063] (3) Human-computer interface device

[0064] Human-computer interface device (or human-computer interface device protocol) is a communication protocol based on universal serial bus (USB), used for data transmission between computer and input device. Human-computer interface device protocol defines standard data format, command structure and transmission method, so that input devices produced by different manufacturers can be recognized and compatible by operating system.

[0065] (4) Inertial measurement unit

[0066] Inertial measurement unit is a device that measures the three-axis attitude angle (or angular rate) and acceleration of an object. Generally, an IMU contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometer detects the acceleration signal of the object in the independent three-axis coordinate system of the carrier, while the gyroscope detects the angular velocity signal of the carrier relative to the navigation coordinate system. The angular velocity and acceleration of the object in three-dimensional space are measured, and the attitude of the object is calculated based on these measurements.

[0067] The present application provides a human-computer interaction method, specifically a human-computer interaction method for converting user action data into HID data for application operations. A terminal device obtains user action data and converts the user action data into human interface device (HID) data corresponding to the application. The terminal device then sends the HID data to a cloud device. The HID data is used to instruct the application to perform operations that match the HID data. Next, the terminal device receives a video stream of the application sent by the cloud device. The video stream is obtained by cloud rendering by the cloud device.

[0068] Based on the above human-computer interaction method, in XR scenarios, the terminal device can convert user motion data into HID data corresponding to different applications, and then send the HID data to the cloud device. The application in the cloud device performs the HID data matching operation, and the terminal device then receives the video stream of the application sent by the cloud device. In this way, for any application adapted or not adapted for XR, if there is a mapping relationship between user motion data and HID data, the terminal device and the cloud device can collaboratively process user motion data for application operations and cloud rendering output, allowing users to operate applications based on the human-computer interaction method of XR scenarios, thereby improving the applicability of XR.

[0069] The implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0070] Figure 1 This is a schematic diagram of the architecture of a human-computer interaction system provided by this application. Figure 1 As shown, the human-computer interaction system (or XR system) 100 includes a cloud device 110, a terminal device 120, and an interactive device 130. The cloud device 110 is in communication connection with the terminal device 120, and the terminal device 120 is in communication connection with the interactive device 130.

[0071] The cloud device 110 may be obtained by virtualizing the hardware resources of one or more server clusters. Figure 2 This is a schematic diagram of the structure of a cloud device provided by this application. Figure 2 As shown, cloud device 110 includes a computing server cluster 111, a storage server cluster 112, a management server cluster 113, and a network device cluster 114. Computing server cluster 111, storage server cluster 112, and management server cluster 113 communicate with each other through network device cluster 114. Computing server cluster 111, storage server cluster 112, and management server cluster 113 can also communicate with terminal device 120 through network device cluster 114.

[0072] The computing server cluster 111 includes one or more computing servers ( Figure 2Two computing servers are shown in FIG, but are not limited to two computing servers).

[0073] The computing server is a computing resource in the cloud device 110, such as a server, desktop computer, etc., which is used to generate and allocate computing resources according to user needs based on virtualization technology. At the hardware level, the computing server is equipped with a processor and memory ( Figure 2 The computing function of the computing server is realized by the processor running the program in the memory. The computing server can also read / write data in each storage server in the storage server cluster 111 according to user needs.

[0074] The storage server cluster 112 includes one or more storage servers ( Figure 2 Two storage servers are shown in FIG, but are not limited to two storage servers).

[0075] The storage server serves as a storage resource in the cloud device 110, such as a server, desktop computer, or storage array controller, hard disk frame, etc., and is used to provide services such as logical disk storage, semi-structured data storage, and integrated backup for the cloud virtual machines in the computer system 100. In terms of hardware, the storage server is provided with a network card, a processor, and a memory. The processor in the storage server is used to process data from outside the storage server. The network card is used to control the access process of the memory, such as the control of address signals, data signals, and various command signals, so that the storage server can provide the memory as a storage resource to the user. The memory is used to store data and may include memory and / or a hard disk. Memory refers to an internal memory that directly exchanges data with the processor. The memory can read and write data quickly at any time and serves as a temporary data storage for the operating system or other running programs. Unlike memory, the hard disk reads and writes data slower than memory and is usually used to store data persistently.

[0076] The management server cluster 113 includes one or more management servers ( Figure 2 Two management servers are shown in FIG, but are not limited to two management servers).

[0077] The management server is used to manage all computing services, shared storage, and networks of the entire cloud device 113, and provides users or administrators with an application program interface (API) for managing the entire node. In this application, the cloud device 110 can provide the program product of this application to users by providing an accessible application program interface.

[0078] The network device cluster 114 includes one or more network devices, such as Figure 2As shown, the network device cluster 114 in this embodiment includes multiple switches, gateways or routers. Among them, each computing server in the computing server cluster 111 is connected through a switch. Each computing server in the computing server cluster 111 and each storage server in the storage server cluster 112 are respectively connected with a switch. Each computing server in the computing server cluster 111, each storage server in the storage server cluster 112 and each management server in the management server cluster 113 are respectively connected with a switch.

[0079] Optionally, the number and type of network devices included in the network device cluster 114 can be adjusted according to the needs of the cloud device 110. The network devices can be switches, routers or gateways with different functions. Taking a switch as an example, the switch can be a core switch for managing a specific network segment, an internal and external switching network segment switch, a storage network segment switch or a management network segment switch.

[0080] The terminal device 120 includes one or more terminal devices (not limited to two terminal devices) as shown in FIG. 1. Figure 1 And Figure 2 The terminal device includes interfaces and applications required for accessing the cloud device 110.

[0081] The terminal device is used to receive operation instructions of a user, and cooperates with the cloud device 110 to process application data, and displays the video stream of the application output by the cloud device 110 to the user. The terminal device can also be called a terminal, a terminal node, a user equipment (UE), a mobile station (MS), a mobile terminal (MT) and the like. The terminal device can be a mobile phone, a pad, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device and the like wired terminal, a wireless terminal in self driving, a wireless terminal in smart grid, a wireless terminal in smart city, a wireless terminal in smart home and the like. Embodiments of the present application do not limit the specific technology and specific device form of the terminal device, for example, the terminal device 120 includes a television.

[0082] The interaction device 130 includes one or more body sensing devices (not limited to two body sensing devices) as shown in FIG. 1. Figure 1Five IMUs are shown in the figure, but the number of IMUs is not limited to five. The somatic device includes a protocol interface for communication with the terminal device 120.

[0083] The somatic device is used to collect user action data of the user and send the user action data to the terminal device 120. The user action data can be the attitude angle (or angular rate) and acceleration of the body part of the user.

[0084] It is worth noting that, Figure 1 and Figure 2 is only a schematic diagram and should not be understood as a limitation of the present application. Other devices can also be included in the human-computer interaction system 100, which are not drawn in Figure 1 and Figure 2 .

[0085] Figure 3 A hierarchical schematic diagram of a human-computer interaction system provided by the present application. As shown in Figure 3 , the human-computer interaction system 100 can be logically divided into a terminal device side, an application side, and a cloud side.

[0086] The terminal device side can be supported by the terminal device 120, and its functional modules can include camera-to-cloud, sensor (such as IMU) access, sensor operation conversion, and sensor plug-in, etc.

[0087] Camera-to-cloud is used to upload the video stream captured by the camera to the cloud device 110, so that the cloud device 110 processes the video stream captured by the camera and superimposes the camera's captured picture on the display interface of the application.

[0088] Sensor access is used to support communication between the terminal device 120 and the sensor, such as the sensor transmitting user action data to the terminal device 120.

[0089] Sensor operation conversion is used to support the terminal device 120 to convert the user action data into HID data corresponding to the application.

[0090] The sensor plug-in is used to support the terminal device 120 to drive the sensor.

[0091] The application side can be supported by the cloud device 110 and the terminal device 120 in coordination, and its functional modules can include applications, application stores, etc. The cloud device 110 processes the data of the application and transmits the processed video stream of the application to the terminal device 120, which outputs the video stream of the application.

[0092] The application can be a to home (toH) application, such as an application developed based on an XR platform, an ecological partner application, a terminal device 110 original application, etc.

[0093] The cloud side can be supported by cloud devices 110, whose functional modules include cloud rendering engine, operation and maintenance, and infrastructure.

[0094] The cloud rendering engine includes the cloud terminal operating system (OS) kernel, collaboration engine, scheduling engine, open source project (Android open source project, AOSP) management, image management, virtualization (such as ARM virtualization), etc.

[0095] Operation and maintenance includes user management, authentication and authorization, operation and maintenance monitoring, operation and maintenance analysis, cloud machine management, security and trustworthiness, and application management.

[0096] The basic settings can be Figure 2 The computing server cluster 111, storage server cluster 112, management server cluster 113 and network device cluster 114 are shown.

[0097] As a possible implementation method, the functional modules on the cloud side can be Figure 1 Based on the human-computer interaction system 100 shown, the functions of virtualized nodes are implemented based on software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS), and services are provided to users through virtualized nodes. The virtualized nodes can be nodes obtained by virtualizing the resources of the human-computer interaction system 100.

[0098] In a possible embodiment of the present application, the cloud device may refer to Figure 1 The cloud device 110 in the embodiment may also refer to Figure 3 On the cloud side, the terminal device can refer to Figure 1 The terminal device 120 in the embodiment may also refer to Figure 3 For example, the following text uses Figure 1 The interaction between the cloud device 110 and the terminal device 120 in the embodiment of the present application is described to illustrate the human-computer interaction method provided.

[0099] It is worth noting that Figure 3 It is only a schematic diagram and should not be understood as a limitation of the present application. The human-computer interaction system 100 may also include other modules. Figure 3 Not drawn in the middle.

[0100] The steps of the human-computer interaction method provided in this application can be collaboratively executed by the cloud device 110, terminal device 120 and interactive device 130 of the above-mentioned human-computer interaction system 100. Figure 4A flowchart of a human-computer interaction method provided in the present application is shown in FIG. 1. As shown in FIG. 1, the human-computer interaction method provided in the embodiments of the present application can include the following steps 401-408. Figure 4

[0101] Step 401: The interaction device 130 sends user action data of a user to the terminal device 120.

[0102] After collecting the user action data of the user, the interaction device 130 sends the user action data to the terminal device 120.

[0103] As a possible implementation, the interaction device 130 can send the user action data requested by the application to the terminal device 120 in an application starting state.

[0104] As a possible implementation, the interaction device 130 can be any device capable of converting the action of the user into an electrical signal. The interaction device 130 can be an IMU, a camera, a displacement sensor, a laser radar, etc.

[0105] For example, the interaction device 130 is an IMU, and the user action data can be attitude data, i.e., three-axis attitude angle or acceleration. For another example, the interaction device 130 is a camera, and the user action data can be action data, i.e., the positions of various body parts of the user in the video stream collected by the camera.

[0106] Step 402: The terminal device 120 acquires the user action data.

[0107] The terminal device 120 receives the user action data sent by the interaction device 130.

[0108] Step 403: The terminal device 120 converts the user action data into HID data corresponding to the application.

[0109] The terminal device 120 converts the user action data into HID data according to the mapping relationship of the application. The mapping relationship of different applications can be different, i.e., the mapping relationship (conversion mode) of the user action data and the HID data is different for different applications.

[0110] As a possible implementation, the user action data includes attitude data transmitted by an IMU arranged at a first part of the user, and the terminal device 120 converts the attitude data into HID data corresponding to the application according to a first mapping relationship of the application.

[0111] Optionally, the first part can be any part such as the head, the hand, the foot, the wrist, the ankle, etc.

[0112] ​Exemplarily, in the XR scene where the target moves in the application, the first part is the head, and the attitude angle change includes the pitch angle change, the yaw angle change, or the roll angle change. The first HID data includes data generated by clicking the first operation key position of the touch screen or the key, and the operation corresponding to the first operation key position includes target forward movement, target backward movement, target left movement, target right movement, field of view left movement, or field of view right movement.

[0113] Optionally, the first mapping relationship can include that, in a case where the attitude data represents the attitude angle change of the first part, the attitude data corresponds to the first HID data. The operation matched by the first HID data includes a touch screen operation or a key operation.

[0114] According to different applications, the HID data corresponding to different attitude angles of the first part can also be different, and specific examples can refer to Figure 6 Figure 7 and related content, which will not be repeated here.

[0115] As a possible implementation manner, the user action data includes attitude data transmitted by the IMU arranged on the second part of the user, and the terminal device 120 converts the attitude data into HID data corresponding to the application according to the second mapping relationship of the application.

[0116] Optionally, the first part can be the head, the hand, the foot, the wrist, the ankle, or any part.

[0117] Exemplarily, in the XR scene where the target makes a fitness action in the application, the second part is the limb, for example, the left wrist, the right wrist, the left ankle, the right ankle, and the like, and the attitude data includes acceleration. The operation matched by the second HID data includes a touch screen operation or a key operation.

[0118] Optionally, the second mapping relationship can include that, in a case where the attitude data represents the acceleration change of the second part, the attitude data corresponds to the second HID data. The operation matched by the second HID data includes a touch screen operation or a key operation.

[0119] According to different applications, the HID data corresponding to different accelerations of the second part can also be different, and specific examples can refer to Figure 8 and related content, which will not be repeated here.

[0120] ​As a possible implementation manner, the user action data includes action data transmitted by a camera disposed on the terminal device 120, and the terminal device 120 determines that the action data corresponds to the third HID data in a case where the action data indicates that the third part of the user performs a specified action at the first position of a camera shooting picture. The third HID data matching operation includes data generated by clicking the second operation key, and the second operation key is located at the second position of the display interface of the application, and the second position is associated with the first position.

[0121] Optionally, the second position being associated with the first position can mean that, when the display interface of the application is superimposed and displayed with the camera shooting picture, that is, the video stream of the camera, the second position and the first position coincide in the picture displayed by the terminal device 120.

[0122] According to different applications, the HID data corresponding to the specified action performed by the third part at different positions can also be different, and specific examples can refer to Figure 5 and related content, which will not be repeated here.

[0123] Step 404, the terminal device 120 sends the HID data to the cloud device 110.

[0124] Step 405, the cloud device 110 receives the HID data sent by the terminal device 120.

[0125] Step 406, the cloud device 110 inputs the HID data into the application.

[0126] The cloud device 110 inputs the HID data into the application to instruct the application to perform the operation matched with the HID data.

[0127] As a possible implementation manner, different applications have different HID operation logics, and after the cloud device 110 inputs the HID data into the application, the cloud device 110 instructs the application to perform the operation matched with the HID data.

[0128] Optionally, the operation matched with the HID data can be a target moving operation, a target limb moving operation, an option clicking operation of an operating system of the terminal device 120, and the like.

[0129] Step 407, the cloud device 110 sends the video stream of the application to the terminal device 120.

[0130] The cloud device 110 performs cloud rendering on the video stream of the application performing the operation matched with the HID data, and sends the video stream of the application to the terminal device 120.

[0131] Step 408, the terminal device 120 receives the video stream of the application sent by the cloud device 110.

[0132] After the terminal device 120 receives the video stream of the application sent by the cloud device 110, the terminal device 120 can process the video stream or display the video stream.

[0133] Based on the above human-computer interaction method, in the XR scenario, the terminal device can convert the user action data into HID data corresponding to the application, and send the HID data to the cloud device. The application in the cloud device performs the operation matched with the HID data, and the terminal device receives the video stream of the application sent by the cloud device. In this way, any application adapted or not adapted to XR can be processed by the terminal device and the cloud device based on the mapping relationship between the user action data and the HID data, so that the user can operate the application based on the human-computer interaction mode in the XR scenario, and the applicability of XR is improved.

[0134] The above describes the human-computer interaction method provided by the present application in detail. Figure 4 The human-computer interaction method is described in detail above, and the following describes the human-computer interaction method in the application of the user action data and the HID operation. Figure 5- Figure 8 The human-computer interaction method is described in detail above, and the following describes the human-computer interaction method in the application of the user action data and the HID operation.

[0135] In the display interface of the terminal device 120, the user can set the terminal device 120 and start the application according to the interactive interface of the display interface. The camera of the terminal device 120 collects the action data of the user, and the terminal device 120 converts the action data of the third part of the user into third HID data to operate the interactive interface.

[0136] As shown in FIG. 1, the interactive interface operation logic can include the following steps: Figure 5

[0137] Step 501: The terminal device 120 acquires the action data photographed by the camera.

[0138] The terminal device 120 acquires the action data from the built-in camera or receives the action data from the external camera connected to the terminal device 120.

[0139] Step 502: The terminal device 120 identifies the third part in the camera photographed picture.

[0140] The terminal device 120 identifies the third part in the camera photographed picture based on the image recognition principle. The image recognition can be implemented based on any image recognition algorithm such as a deep learning algorithm or a template matching algorithm, which is not described herein.

[0141] Step 503: The terminal device 120 outputs the superimposed interface of the display interface of the application and the camera photographed picture.

[0142] ​The terminal device 120 superimposes the display interface of the application and the camera shooting picture, and outputs a superimposed interface. The display interface can be an operation interface of the application such as a game, or an operation interface of a system application such as a setting or a user interface.

[0143] As a possible implementation, the transparency, level, etc. of the display interface of the application and the camera shooting picture in the superimposed interface can be flexibly adjusted.

[0144] Optionally, the display interface of the application and the camera shooting picture are both semi-transparent, or the transparency of the display interface of the application is higher than that of the camera shooting picture, or the transparency of the display interface of the application is lower than that of the camera shooting picture.

[0145] As a possible implementation, the third part of the camera shooting picture in the superimposed interface displays a first identifier, and the first identifier is used to indicate the position of the third part in the superimposed interface.

[0146] Optionally, the identifier can be a highlight frame, a semi-transparent aperture, etc.

[0147] Step 504, in a case where the action data indicates that the third part performs a specified action at the first position of the camera shooting picture, the terminal device 120 converts the action data into third HID data.

[0148] In a case where the action data indicates that the third part of the user is at the first position of the camera shooting picture and performs a specified action, the terminal device 120 converts the action data into third HID data of the interactive interface.

[0149] As a possible implementation, in a case where the terminal device 120 is a non-touch screen device, the third HID data can be data generated by clicking a second operation key of a controller such as a remote controller.

[0150] As a possible implementation, in a case where the terminal device 120 is a touch screen device, the third HID data can be data generated by clicking a second operation key of the touch screen.

[0151] Optionally, the second operation key can be a clickable graph element of the interactive interface, and is at a second position of the display interface of the application. The first position and the second position are associated, i.e., the first position and the second position coincide in the superimposed picture.

[0152] As a possible implementation, the specified action can be clenching a fist, opening a palm, etc.

[0153] Step 505, the terminal device 120 inputs the third HID data into the application.

[0154] The terminal device 120 inputs the third HID data into a local application, such as an interactive interface, to perform a click operation on a clickable graphic element of the interactive interface in the application.

[0155] In the above process, the user can click the clickable graphic element through human-computer interaction to start the application. After the application is started, the user can also perform in-application operations based on the human-computer interaction method of the present application.

[0156] As shown in Figure 6 The application operation logic can include the following steps:

[0157] In step 601, the terminal device 120 acquires attitude data collected by an IMU.

[0158] The terminal device 120 acquires attitude data transmitted by an IMU arranged at a first part of the user.

[0159] As a possible implementation, the first part is the head, and the IMU collects angle attitude values or attitude angle changes (including pitch angle changes, yaw angle changes, or roll angle changes) of the head of the user. The IMU is arranged in a hat or a support on the head of the user, and the idle attitude of the head of the user is taken as the starting state of the action.

[0160] In step 602, the terminal device 120 converts the attitude data into first HID data corresponding to the application according to a first mapping relationship of the application.

[0161] The terminal device 120 converts the attitude data into first HID data corresponding to the application according to the first mapping relationship of the application opened in Figure 5

[0162] As a possible implementation, the head of the user moves based on the pitch angle, and the attitude angle change of the attitude data indicates that the user is looking down. In the first mapping relationship, the corresponding first HID data is data generated by clicking a first operation key position of a touch screen or a key, and the corresponding operation of the first operation key position in the application is to move the target forward. The attitude angle change of the attitude data indicates that the user is looking up. In the first mapping relationship, the corresponding first HID data is data generated by clicking the first operation key position of the touch screen or the key, and the corresponding operation of the first operation key position in the application is to move the target backward.

[0163] ​As a possible implementation manner, the head of the user moves based on the yaw angle, the change of the attitude angle of the attitude data indicates that the user tilts the head to the left, and the corresponding first HID data in the first mapping relationship is data generated by clicking a first operation key position of a touch screen or a key, and the corresponding operation of the first operation key position in the application is moving the target to the left. The change of the attitude angle of the attitude data indicates that the user tilts the head to the right, and the corresponding first HID data in the first mapping relationship is data generated by clicking a first operation key position of a touch screen or a key, and the corresponding operation of the first operation key position in the application is moving the target to the right.

[0164] As a possible implementation manner, the head of the user moves based on the roll angle, the change of the attitude angle of the attitude data indicates that the user turns the head to the left and then returns to normal, and the corresponding first HID data in the first mapping relationship is data generated by clicking a first operation key position of a touch screen or a key, and the corresponding operation of the first operation key position in the application is moving the field of view to the left. The change of the attitude angle of the attitude data indicates that the user turns the head to the right and then returns to normal, and the corresponding first HID data in the first mapping relationship is data generated by clicking a first operation key position of a touch screen or a key, and the corresponding operation of the first operation key position in the application is moving the field of view to the right.

[0165] Optionally, moving the field of view to the left or to the right is horizontal rotation of the field of view camera to the left or to the right at a fixed angle. The field of view can also return to normal after moving to the left or to the right.

[0166] The above first operation key position is only an example given by the embodiments of the present application, and a plurality of operations in the above application can correspond to different operation key positions, that is, the front, back, left and right movements of the target and the left and right movements of the field of view correspond to different operation key positions respectively.

[0167] After step 602, the terminal device 120 can cooperatively process with the cloud device 110 to realize video stream output of the application, and the specific mode can refer to steps 404-408 shown in Figure 4 and will not be described herein again.

[0168] The above Figure 6 The application operation logic shown in the above can be used in a scene in which the user implements XR control of the application through head movement. Different application operation logics can be used in different applications of the user, for example, the user can implement XR control of a target (for example, a vehicle) in the application through four-limb movement.

[0169] As shown in the above Figure 7 Another application operation logic can include the following steps:

[0170] In step 701, the terminal device 120 acquires the attitude data collected by the IMU.

[0171] The terminal device 120 acquires the attitude data transmitted by the IMU arranged at the second part of the user.

[0172] As a possible implementation, the second part is the limbs, the IMU collects the attitude data of the limbs of the user, i.e. the acceleration change of the left wrist, the right wrist, the left ankle and the right ankle, and the IMU is arranged in the gloves, shoes, socks and the like of the limbs of the user, and the still attitude of the limbs of the user is taken as the starting state of the action.

[0173] In step 702, the terminal device 120 converts the attitude data into the second HID data corresponding to the application according to the second mapping relationship of the application.

[0174] The terminal device 120 converts the attitude data into the second HID data corresponding to the application according to the second mapping relationship of the application opened in step 701. Figure 5 The terminal device 120 converts the attitude data into the second HID data corresponding to the application according to the second mapping relationship of the application opened in step 701.

[0175] As a possible implementation, the action of the user is jumping, the acceleration change of the attitude data indicates that the user jumps, and the corresponding second HID data in the second mapping relationship is the data generated by the second operation key position of the touch screen or the key click, and the corresponding operation of the second operation key position in the application is the target brake.

[0176] As a possible implementation, the action of the user is left foot movement, the acceleration change of the attitude data indicates that the user moves the left foot, and the corresponding second HID data in the second mapping relationship is the data generated by the second operation key position of the touch screen or the key click, and the corresponding operation of the second operation key position in the application is the target left turn.

[0177] As a possible implementation, the action of the user is right foot movement, the acceleration change of the attitude data indicates that the user moves the right foot, and the corresponding second HID data in the second mapping relationship is the data generated by the second operation key position of the touch screen or the key click, and the corresponding operation of the second operation key position in the application is the target right turn.

[0178] As a possible implementation, the action of the user is squatting, the acceleration change of the attitude data indicates that the user squats, and the corresponding second HID data in the second mapping relationship is the data generated by the second operation key position of the touch screen or the key click, and the corresponding operation of the second operation key position in the application is the target acceleration.

[0179] The above-mentioned second operation key position is only an example given by the embodiments of the present application, and a plurality of operations in the above-mentioned application can correspond to different operation key positions, i.e. the target brake, the left turn, the right turn and the acceleration correspond to different operation key positions respectively.

[0180] After step 702, the terminal device 120 can process the video stream output of the application in cooperation with the cloud device 110, and the specific mode can refer to the description of the cloud device 110. Figure 4The steps 404-408 shown are not described again here.

[0181] The above Figure 6 And Figure 7 The application operation logic shown can be applied to the scene of XR target control of non-touch or touch devices, and in applications such as XR fitness, the user can also realize the synchronization of the target (three-dimensional character) and the user action through the application operation logic of the human-computer interaction method of the embodiments of the present application.

[0182] As Figure 8 Another application operation logic can include the following steps:

[0183] Step 801, the terminal device 120 acquires the posture data collected by the IMU.

[0184] The terminal device 120 acquires the posture data transmitted by the IMU arranged at the second part of the user.

[0185] As a possible implementation manner, the second part is the limbs, and the IMU collects the posture data or acceleration change of the limbs of the user, i.e., the left wrist, the right wrist, the left ankle, and the right ankle. The IMU is arranged in the gloves, shoes, socks, etc. of the user, and the static posture of the limbs of the user is taken as the starting state of the action.

[0186] Step 802, the terminal device 120 converts the posture data into second HID data corresponding to the application according to the second mapping relationship of the application.

[0187] The terminal device 120 converts the posture data into second HID data corresponding to the application according to the second mapping relationship of the application opened by the step. Figure 5

[0188] As a possible implementation manner, the left wrist of the user moves, and the acceleration change of the posture data represents the movement of the left wrist of the user. In the second mapping relationship, the corresponding second HID data is the data generated by the second operation key position of the touch screen or the key click, and the corresponding operation of the second operation key position in the application is the target left wrist movement.

[0189] As a possible implementation manner, the right wrist of the user moves, and the acceleration change of the posture data represents the movement of the right wrist of the user. In the second mapping relationship, the corresponding second HID data is the data generated by the second operation key position of the touch screen or the key click, and the corresponding operation of the second operation key position in the application is the target right wrist movement.

[0190] ​As a possible implementation, the user moves the left ankle, the acceleration change of the posture data represents the user moving the left ankle, and the corresponding second HID data in the second mapping relationship is data generated by the user clicking a second operation key position of a touch screen or a button, and the corresponding operation of the second operation key position in the application is target left ankle movement.

[0191] As a possible implementation, the user moves the right ankle, the acceleration change of the posture data represents the user moving the right ankle, and the corresponding second HID data in the second mapping relationship is data generated by the user clicking a second operation key position of a touch screen or a button, and the corresponding operation of the second operation key position in the application is target right ankle movement.

[0192] The second operation key position described above is only an example given by the embodiments of the present application, and multiple operations in the application can correspond to different operation key positions, i.e., different operation key positions correspond to target limb movement respectively.

[0193] The applications in the above Figure 6- Figure 8 may be the same or different applications, and the applications have preset mapping relationships.

[0194] After step 802, the terminal device 120 can cooperatively process with the cloud device 110 to realize video stream output of the application, and the specific mode can refer to steps 404-408 shown in Figure 4 , which will not be described here again.

[0195] In possible embodiments, the cloud device 110 can also determine the accuracy of the user action data corresponding to the action according to the matching degree of the input features of the input multiple HID data and the preset data features. The input features include the input order or input interval of the multiple HID data. In this way, the cloud device 110 can score the accuracy of the user's action in the application.

[0196] In order to cooperate with the above-mentioned Figure 4 human-computer interaction method provided by the present application, the present application further provides a human-computer interaction device 900, which can be used to realize the function of the terminal device 120 in the above-mentioned Figure 4 human-computer interaction method. As shown in Figure 9 , the human-computer interaction device 900 includes a transceiver module 910 and a processing module 920.

[0197] The transceiver module 910 is configured to obtain user action data. For example, the transceiver module 910 is configured to perform step 402 as shown in Figure 4 .

[0198] The processing module 920 is configured to convert the user action data into HID data corresponding to an application. For example, the processing module 920 is configured to perform step 404 as shown in Figure 4The step 403 shown.

[0199] The transceiver 910 is further configured to send the HID data to the cloud device, where the HID data is used to instruct the application to perform an operation matched with the HID data. For example, the transceiver 910 is configured to perform the following operations: Figure 4 The step 404 shown.

[0200] As a possible implementation, the user action data includes attitude data transmitted by an inertial measurement unit (IMU) disposed at a first part of the user. The processing module 920 is specifically configured to convert the attitude data into HID data corresponding to the application according to a first mapping relationship of the application.

[0201] Optionally, the first mapping relationship includes that, in a case where the attitude data represents a change in attitude angle of the first part, the attitude data corresponds to first HID data, and the operation matched with the first HID data includes a touch screen operation or a key operation.

[0202] Optionally, the change in attitude angle includes a change in pitch angle, a change in yaw angle, or a change in roll angle, the first HID data includes data generated by a first operation key position of a touch screen or a key, and the operation corresponding to the first operation key position includes target forward movement, target backward movement, target leftward movement, target rightward movement, field of view leftward movement, or field of view rightward movement.

[0203] As a possible implementation, the user action data includes attitude data transmitted by an IMU disposed at a second part of the user. The processing module is specifically configured to convert the attitude data into HID data corresponding to the application according to a second mapping relationship of the application.

[0204] Optionally, the second mapping relationship includes that, in a case where the attitude data represents a change in acceleration of the second part, the attitude data corresponds to second HID data, and the operation matched with the second HID data includes a touch screen operation or a key operation.

[0205] Optionally, the change in acceleration includes that an acceleration value is greater than or equal to a preset threshold, and the second HID data includes data generated by a second operation key position of a touch screen or a key.

[0206] As a possible implementation, the user action data includes action data captured by a camera. In a case where the action data represents that a third part of the user performs a specified action at a first position in a camera-captured image, the action data corresponds to third HID data, and the operation matched with the third HID data includes data generated by a second operation key position, the second operation key position is at a second position in a display interface of the application, and the second position is associated with the first position.

[0207] As a possible implementation, the processing module 920 is further configured to: output a superimposed interface of the display interface of the application and the camera shooting picture; and the superimposed picture includes a first identifier, and the first identifier is used to indicate the second position in the display interface that coincides with the first position.

[0208] The transceiver module 910 and the processing module 920 can be implemented by software or by hardware. For example, the transceiver module 910 is taken as an example, and the implementation of the transceiver module 910 is introduced as follows. Similarly, the implementation of the processing module 920 can refer to the implementation of the transceiver module 910.

[0209] As an example of the software function unit, the transceiver module 910 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the transceiver module 910 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region, or can be distributed in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ), or can be distributed in different AZs, and each AZ includes one data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0210] Similarly, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC), or can be distributed in multiple VPCs. Generally, one VPC is set in one region, and a communication gateway needs to be set in each VPC for cross-region communication between two VPCs in the same region and between VPCs in different regions, and the interconnection between VPCs is realized through the communication gateway.

[0211] As an example of a hardware functional unit, the transceiver module 910 can include at least one computing device, such as a server or the like. Alternatively, the transceiver module 910 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the like. The PLD can be implemented by a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0212] The multiple computing devices included in the transceiver module 910 can be distributed in the same region or in different regions. The multiple computing devices included in the transceiver module 910 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the transceiver module 910 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0213] It should be noted that in other embodiments, any of the transceiver module 910 and the processing module 920 can be used to perform any step of the human-computer interaction method, and the steps implemented by the transceiver module 910 and the processing module 920 can be specified as needed. The transceiver module 910 and the processing module 920 respectively implement different steps of the human-computer interaction method to realize the entire function of the human-computer interaction device 900.

[0214] To cooperate with the above-mentioned Figure 4 The human-computer interaction method shown in the present application also provides a human-computer interaction device 1000, which can be used to realize the function of the cloud device 110 in the above-mentioned Figure 4 The human-computer interaction device 1000 shown in the present application includes: Figure 10 The human-computer interaction device 1000 shown in the present application includes:

[0215] The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4 The transceiver module 1010 is configured to receive the HID data sent by the terminal device. The HID data is converted from the user action data, and the conversion mode of the user action data and the HID data corresponds to an application. For example, the transceiver module 1010 is configured to perform step 405 as shown in the above-mentioned Figure 4

[0216] The processing module 1020 is configured to input the HID data into the application, and the HID data is used to instruct the application to perform an operation matched with the HID data. For example, the processing module 1020 is configured to perform step 406 as shown in the following. Figure 4

[0217] The transceiver module 1010 is further configured to output a video stream of the application to the terminal device, and the video stream is obtained by cloud rendering of the cloud device. For example, the transceiver module 1010 is configured to perform step 407 as shown in the following. Figure 4

[0218] As a possible implementation manner, the processing module 1020 is further configured to determine an accuracy of the action corresponding to the user action data according to a matching degree of input features of the input HID data and preset data features, and the input features include an input sequence or an input interval of the HID data.

[0219] The transceiver module 1010 and the processing module 1020 can be implemented by software or hardware. For example, the implementation manner of the transceiver module 1010 is described as follows. Similarly, the implementation manner of the processing module 1020 can refer to the implementation manner of the transceiver module 1010.

[0220] As an example of the software function unit, the transceiver module 1010 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the transceiver module 1010 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region (region), or can be distributed in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ), or can be distributed in different AZs, and each AZ includes one data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0221] ​​Similarly, the plurality of hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, usually one VPC is set in one region, and a communication gateway needs to be set in each VPC for cross-region communication between two VPCs in the same region and between VPCs in different regions, and the interconnection between VPCs is realized through the communication gateway.

[0222] As an example of a hardware functional unit, the transceiver module 1010 can include at least one computing device, such as a server, etc. Alternatively, the transceiver module 1010 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. Among them, the above-mentioned PLD can be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0223] The plurality of computing devices included in the transceiver module 1010 can be distributed in the same region or in different regions. The plurality of computing devices included in the transceiver module 1010 can be distributed in the same AZ or in different AZs. Similarly, the plurality of computing devices included in the transceiver module 1010 can be distributed in the same VPC or in multiple VPCs. Among them, the plurality of computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs, etc.

[0224] It should be noted that in other embodiments, any of the transceiver module 1010 and the processing module 1020 can be used to perform any step in the human-computer interaction method, and the steps implemented by the transceiver module 1010 and the processing module 1020 can be specified as needed. By implementing different steps in the human-computer interaction method through the transceiver module 1010 and the processing module 1020 respectively, the entire function of the human-computer interaction device 1000 can be realized.

[0225] The present application also provides a computing device 1100. As shown in Figure 11As shown, the computing device 1100 includes a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. The processor 1104, the memory 1106, and the communication interface 1108 communicate with each other through the bus 1102. The computing device 1100 can be a server or a terminal device. It should be understood that the number of processors and memories in the computing device 1100 is not limited.

[0226] The bus 1102 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 Although only one line is used to represent the bus in the figure, it does not mean that there is only one bus or only one type of bus. The bus 1102 can include a path for transmitting information between various components (e.g., the memory 1106, the processor 1104, the communication interface 1108) of the computing device 1100.

[0227] The processor 1104 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0228] The memory 1106 can include a volatile memory (e.g., a random access memory (RAM)) and a non-volatile memory (e.g., a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD)).

[0229] The memory 1106 stores executable program code, and the processor 1104 executes the executable program code to respectively implement the functions of the various modules included in the aforementioned human-computer interaction device 900 or the human-computer interaction device 1000, thereby implementing the human-computer interaction method. That is, the memory 1106 has instructions for executing the human-computer interaction method.

[0230] Alternatively, the memory 1106 stores executable code that, when executed by the processor 1104, implements the functionality of the client 610 or the server 620, respectively, to implement the human-computer interaction method. That is, the memory 1106 stores instructions for implementing the human-computer interaction method.

[0231] The communication interface 1108 uses a transceiver module such as, but not limited to, a network interface card, a transceiver, and the like to enable communication between the computing device 1100 and other devices or communication networks.

[0232] In view of the human-computer interaction method provided by the present application being applied in the computer system 100, the respective infrastructures of the computing server cluster 111 and the storage server cluster 112 of the computer system 100 usually include multiple computing devices. Therefore, the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smartphone.

[0233] As shown in Figure 12 The computing device cluster includes at least one computing device 1100. The memory 1106 of one or more computing devices 1100 in the computing device cluster can store the same instructions for implementing the human-computer interaction method.

[0234] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster can also respectively store partial instructions for implementing the human-computer interaction method. In other words, the combination of one or more computing devices 1100 can collectively execute the instructions for implementing the human-computer interaction method.

[0235] It should be noted that the memories 1106 of different computing devices 1100 in the computing device cluster can store different instructions for respectively implementing part of the functions of the human-computer interaction apparatus 900 or the human-computer interaction apparatus 1000. That is, the instructions stored in the memories 1106 of different computing devices 1100 can implement the functions of one or more modules included in the human-computer interaction apparatus 900 or the human-computer interaction apparatus 1000.

[0236] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, and the like. Figure 13 A possible implementation is shown. As Figure 13As shown, the two computing devices 1100A and 1100B are connected through a network. Specifically, the network is connected through a communication interface in each computing device. In this type of possible implementation, the memory 1106 in the computing device 1100A has instructions for performing the functions of one or more of the transceiver module 910, the processing module 920, Figure 13 For example, the memory 1106 in the computing device 1100A has instructions for performing the functions of the transceiver module 910. Meanwhile, the memory 1106 in the computing device 1100B has instructions for performing the functions of one or more of the transceiver module 910, the processing module 920, Figure 13 For example, the memory 1106 in the computing device 1100B has instructions for performing the functions of the processing module 920.

[0237] It should be understood that the functions of the computing device 1100A shown in Figure 13 The functions of the computing device 1100A shown in

[0238] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be software or a program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, it makes the at least one computing device perform the steps of the method as Figure 4 the human-computer interaction method as Figure 4 the steps performed by the cloud device 110 or the terminal device 120 in the human-computer interaction method as

[0239] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk) and the like. The computer readable storage medium includes instructions, which instruct the computing device to perform the steps of the method as Figure 4 the human-computer interaction method as Figure 4 the steps performed by the cloud device 110 or the terminal device 120 in the human-computer interaction method as

[0240] The above-described embodiments can be implemented in whole or in part by software, hardware (such as circuitry), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded on or executed by a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0241] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0242] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0243] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0244] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0245] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0246] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk, and various program code storage media.

[0247] In the present application, "at least one" means one or more, and "multiple" means two or more. The "and / or" describes the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following cases: A exists alone, A and B exist together, B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c, which can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b and c can be single or multiple.

[0248] It should be noted that in the present application, the words "exemplary" or "for example" are used to mean an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0249] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A human-machine interaction method, characterized in that, A terminal device applied to an extended reality (XR) system, the XR system further comprising a cloud device connected with the terminal device, the method comprising: obtaining user action data; converting the user action data into human interface device (HID) data corresponding to an application; sending the HID data to the cloud device; the HID data being used to instruct the application to perform an operation matched with the HID data; receiving a video stream of the application sent by the cloud device; the video stream being obtained by cloud rendering performed by the cloud device.

2. The method of claim 1, wherein, The user action data comprises attitude data transmitted by an inertial measurement unit (IMU) arranged at a first part of a user, and the converting the user action data into HID data corresponding to an application comprises: converting the attitude data into HID data corresponding to the application according to a first mapping relationship of the application.

3. The method of claim 2, wherein, The first mapping relationship comprises: in a case where the attitude data represents a change in attitude angle of the first part, the attitude data corresponds to first HID data; the operation matched with the first HID data comprises a touch screen operation or a key operation.

4. The method of claim 3, wherein, The change in attitude angle comprises a change in pitch angle, a change in yaw angle, or a change in roll angle, the first HID data comprises data generated by clicking a first operation key position of a touch screen or a key, and the operation corresponding to the first operation key position comprises moving a target forward, moving the target backward, moving the target leftward, moving the target rightward, moving a field of view leftward, or moving the field of view rightward.

5. The method of claim 1, wherein, The user action data comprises attitude data transmitted by an IMU arranged at a second part of a user, and the converting the user action data into HID data corresponding to an application comprises: converting the attitude data into HID data corresponding to the application according to a second mapping relationship of the application.

6. The method of claim 5, wherein, The second mapping relationship comprises: in a case where the attitude data represents a change in acceleration of the second part, the attitude data corresponds to second HID data; the operation matched with the second HID data comprises a touch screen operation or a key operation.

7. The method of claim 6, wherein, The change in acceleration comprises an acceleration value greater than or equal to a preset threshold, and the second HID data comprises data generated by clicking a second operation key position of a touch screen or a key.

8. The method of claim 1, wherein, The user action data comprises action data captured by a camera, and the converting the user action data into HID data corresponding to an application comprises: in a case where the action data represents a specified action performed by a third part of the user at a first position of a camera-captured image, the action data corresponds to third HID data; the third HID data matches an operation comprising data generated by clicking a second operation key position, the second operation key position being at a second position of a display interface of the application, and the second position being associated with the first position.

9. The method of claim 8, wherein, The method further comprises: outputting an overlaid interface of the display interface of the application and the camera-captured image; the overlaid interface comprising a first identifier, the first identifier being used to indicate the second position in the display interface that coincides with the first position.

10. A human-machine interaction method, characterized in that, A cloud device applied to an XR system, the XR system further comprising a terminal device connected to the cloud device, the method comprising: receiving HID data sent by the terminal device; the HID data being converted from user action data, and a conversion mode of the user action data and the HID data corresponding to an application; inputting the HID data into the application; the HID data being used to instruct the application to perform an operation matched with the HID data; outputting a video stream of the application to the terminal device; the video stream being obtained by cloud rendering of the cloud device.

11. The method of claim 10, wherein, The method further comprises: determining an accuracy of a corresponding action of the user action data according to a matching degree of input features of the input HID data and preset data features; the input features including an input order or an input interval of the HID data.

12. A human-machine interaction device, characterized in that, Comprises: a transceiving module configured to obtain user action data; a processing module configured to convert the user action data into HID data corresponding to an application; the transceiving module is further configured to send the HID data to a cloud device; the HID data being used to instruct the application to perform an operation matched with the HID data; the transceiving module is further configured to receive a video stream of the application sent by the cloud device; the video stream being obtained by cloud rendering of the cloud device.

13. A human-machine interaction device, characterized in that, Comprises: a transceiving module configured to receive HID data sent by a terminal device; the HID data being converted from user action data, and a conversion mode of the user action data and the HID data corresponding to an application; a processing module configured to input the HID data into the application; the HID data being used to instruct the application to perform an operation matched with the HID data; the transceiving module is further configured to output a video stream of the application to the terminal device; the video stream being obtained by cloud rendering of a cloud device.

14. A computing device, comprising: Comprises a processor and a memory; the processor is configured to execute instructions stored in the memory, so that the computing device performs the method of any one of claims 1-11.

15. A computer program product comprising instructions, characterized in that, When the instructions are executed by a computing device, the computing device performs the method of any one of claims 1-11.

16. A computer-readable storage medium, characterized in that, Comprises computer program instructions, when the computer program instructions are executed by a computing device, the computing device performs the method of any one of claims 1-11.

Images