Cloud gaming-based device control method, device, electronic device, and computer program

Abstract

The present application provides a device control method, an apparatus, an electronic device, and a readable medium based on a cloud application, the method including: establishing a communication connection between a cloud application client on a terminal device, the cloud application client being used for presenting a received cloud application video stream; performing scene recognition for the cloud application client and a corresponding cloud application to obtain a scene recognition result, the scene recognition result including media content associated with a recognized cloud application scene; generating vibration feedback information corresponding to the cloud application scene based on the scene recognition result; and transmitting a video stream corresponding to the cloud application scene and the vibration feedback information to the cloud application client, the method controlling a physical device associated with the cloud application client to perform a vibration feedback operation based on the vibration feedback information when the cloud application client presents the video stream of the cloud application scene.

Description

[Technical field]

[0001] The present application relates to the field of computer technology, and in particular to a device control method, device, electronic device, and readable medium based on a cloud application.

[0002] This application claims priority to a Chinese patent application filed with the China Patent Office on July 21, 2021, bearing application number 202110827404.8 and entitled “Cloud gaming-based device control method, device, electronic device, and readable medium.” [Background technology]

[0003] With the rapid development of Internet technology, people's demands for entertainment forms are becoming higher and higher. Games are an important form of entertainment, especially cloud games, which are currently popular, do not require users to have high-configuration hardware, but can provide users with a superior entertainment experience.

[0004] For this type of cloud application, the operation process of the cloud application occurs on the server, so the application client on the server needs to render the application screens and scenes into video, send them to the user terminal, and play them on the user terminal to display the interface content.

[0005] However, because cloud applications run on servers, terminal devices can only receive rendered audio-video streams, and the interaction methods they can provide are limited, making it difficult to provide a rich and immersive experience and affecting the interaction effect between cloud applications and users. Summary of the Invention [Problem to be solved by the invention]

[0006] Based on the above technical problems, the present application provides a device control method, device, electronic device, and readable medium based on a cloud application, so that in the cloud application scene, vibration feedback consistent with the application content can be triggered by the server, thereby improving the accuracy of the vibration feedback and thereby improving the user experience.

[0007] Other features and advantages of the present application will become apparent from the following detailed description, or in part may be learned by the practice of the present application. [Means for solving the problem]

[0008] According to one aspect of an embodiment of the present application, there is provided a cloud application-based appliance control method, the method being executed by an electronic device, comprising: establishing a communication connection between a cloud application client on a terminal device, the cloud application client being used for presenting the received cloud application video stream; performing scene recognition for a cloud application corresponding to the cloud application client to obtain a scene recognition result, the scene recognition result including media content associated with a recognized cloud application scene; generating vibration feedback information corresponding to the cloud application scene based on the scene recognition result; The method includes a step of transmitting a video stream corresponding to the cloud application scene and the vibration feedback information to the cloud application client, and a step of controlling a physical device associated with the cloud application client to perform a vibration feedback operation based on the vibration feedback information when the cloud application client presents the video stream of the cloud application scene.

[0009] According to another aspect of the present embodiment, there is provided a cloud application-based appliance control device, A communication establishment module used for establishing a communication connection between a cloud application client on a terminal device, the cloud application client being used for presenting a received cloud application video stream; a scene recognition module adapted to perform scene recognition on a cloud application corresponding to the cloud application client to obtain a scene recognition result, the scene recognition result including media content associated with a recognized cloud application scene; a vibration feedback information generating module, which is used to generate vibration feedback information corresponding to the cloud application scene according to the scene recognition result; and a vibration feedback information transmitting module used for transmitting a video stream corresponding to the cloud application scene and the vibration feedback information to the cloud application client, the vibration feedback information transmitting module controlling a physical device associated with the cloud application client to perform a vibration feedback operation based on the vibration feedback information when the cloud application client presents the video stream of the cloud application scene.

[0010] According to another aspect of the present embodiment, there is provided a cloud application-based appliance control method, the method being executed by an electronic appliance, the method comprising: establishing a communication connection with a cloud application server; receiving a video stream and vibration feedback information corresponding to a cloud application scene transmitted from the cloud application server, the vibration feedback information being generated by an application client on a server based on the cloud application scene; playing a video stream corresponding to the cloud application scene; and controlling a physical device associated with a cloud application client to perform a vibration feedback operation based on the vibration feedback information.

[0011] According to another aspect of the present embodiment, there is provided a cloud application-based appliance control device, A communication connection establishment module used for establishing a communication connection with a cloud application server; a vibration feedback information receiving module, which is used to receive a video stream corresponding to a cloud application scene and vibration feedback information sent from the cloud application server, wherein the vibration feedback information is generated by an application client on a server according to the cloud application scene; a playback module, which is used for playing a video stream corresponding to the cloud application scene; and a vibration feedback execution module adapted to control a physical device associated with the cloud application client to perform a vibration feedback operation based on the vibration feedback information.

[0012] According to one aspect of an embodiment of the present application, an electronic device is provided, the electronic device including a processor and a memory used for storing executable instructions of the processor, the processor being configured to execute the cloud application-based appliance control method in the above technical solution via execution of the executable instructions.

[0013] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, realizes the cloud application-based device control method in the above technical solution.

[0014] According to one aspect of the present invention, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions stored in a computer-readable storage medium, a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the device control method based on the cloud application provided in the various selectable implementation manners.

[0015] It should be understood that the above general description and the following detailed description are merely illustrative and explanatory, and cannot limit the present application.

[0016] The drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to interpret the principles of the present application. It is apparent that the drawings in the following description are merely some embodiments of the present application, and those skilled in the art can further obtain other drawings based on these drawings without requiring creative work. The drawings are as follows: [Brief description of the drawings]

[0017] [Figure 1] FIG. 2 is an exemplary network architecture diagram of a device control system based on a cloud application in an embodiment of the present application. [Diagram 2]FIG. 1 is a schematic diagram of a cloud application appliance control framework in an embodiment of the present application. [Diagram 3] FIG. 2 is a schematic diagram of the vibration special effects design process in the present application. [Figure 4] 1 is a flowchart of a device control method based on a cloud application provided by an embodiment of the present application; [Diagram 5] 1 is a process of triggering vibration feedback in an embodiment of the present application. [Figure 6] 1 illustrates a process of establishing a communication connection with a cloud application client in an embodiment of the present application. [Figure 7] 4 is a flowchart of a device control method based on a cloud application provided in another embodiment of the present application. [Figure 8A] FIG. 2 is a block diagram showing the configuration of a device control device based on a cloud application according to an embodiment of the present application; [Figure 8B] FIG. 13 is a block diagram showing the configuration of a device control device based on a cloud application in another embodiment of the present application. [Figure 9] 1 shows a structural schematic diagram of an electronic device suitable for use in implementing an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Exemplary embodiments will now be described more fully with reference to the drawings. However, the exemplary embodiments can be implemented in a variety of forms and should not be understood to be limited to the examples discussed herein, but rather, these embodiments are provided to make the present application more complete and complete, and to fully convey the idea of ​​the exemplary embodiments to those skilled in the art.

[0019] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner into one or more embodiments. In the following description, many specific details are provided to provide a thorough understanding of the embodiments of the present application. However, those skilled in the art will recognize that the technical solutions of the present application can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be employed. In other circumstances, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present application.

[0020] The block diagrams shown in the drawings are merely functional entities that do not necessarily correspond to physically separate entities, i.e. they can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0021] The flowcharts shown in the drawings are merely exemplary illustrations, and do not necessarily include all contents and operations / steps, and do not necessarily need to be performed in the order described. For example, some operations / steps can be further decomposed, while some operations / steps can be merged or partially merged, and therefore the actual execution order may change based on the actual situation.

[0022] The embodiments of this application are applied to the application scenario of cloud applications. The so-called "cloud applications" refer to the implementation of cloud computing technology in the application layer. The working principle of "cloud applications" is to change the usage mode of traditional application programs (or software) "local installation, local operation" into "use without installation" services, and connect to and operate remote server clusters through the Internet or local area networks to complete business logic or operation tasks.

[0023] The main carrier of "cloud applications" is Internet technology, and the cloud application client takes thin client or smart client as the display format, and its interface integrates technologies such as HTML5, Javascript, or Flash. Cloud applications can not only help users reduce IT costs, but also greatly improve work efficiency.

[0024] Specifically, the device control system based on the cloud application can operate in the network architecture shown in Fig. 1. As shown in Fig. 1, the device control system based on the cloud application includes a cloud-side server 101 and a variety of types of terminal devices 102-106, and the cloud-side server 101 can provide operation commands sent to the cloud-side server 101 based on the control process of the cloud application of the various terminal devices 102-106, i.e., the control operation of the user on the terminal devices 102-106 side. The cloud-side server 101 performs processes such as control of the cloud application instance, scene recognition, operation result calculation, and audio / video rendering in response to the operation command, thereby returning information corresponding to the operation command, such as a screen, music, etc., to the terminal devices 102-106.

[0025] It can be understood that Fig. 1 shows multiple types of terminal devices 102-106, and the terminal devices may be computer devices. In an actual scenario, more or fewer types of terminal devices may be involved in the control process of a cloud application, but the specific quantity and types are determined by the actual scenario and are not limited herein.

[0026] Also, while FIG. 1 shows one cloud-side server 101, in an actual scene, multiple cloud-side servers may be involved, and the specific number of cloud-side servers will be determined by the actual scene, particularly in a scene where multiple different cloud game instances are running.

[0027] The cloud-side server may be an independent physical server, a server cluster consisting of multiple physical servers, or a distributed system, and may further be a cloud-side server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, big data, and artificial intelligence platforms. The terminal may be, but is not limited to, a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and the like. The terminal and the cloud-side server may be directly or indirectly connected through a wired or wireless communication method, and the terminal and the cloud-side server may be connected to form a blockchain network, but this application is not limited thereto.

[0028] In various specific cloud applications, cloud gaming is a type of online gaming technology based on cloud computing technology, and may also be called gaming on demand. Cloud gaming technology enables light-end devices (e.g., thin clients) with relatively limited graphics processing and data computing capabilities to run high-quality games.

[0029] It should be understood that the technical solution of the present application can be applied to a cloud gaming scenario, such as a cloud gaming platform, and includes a cloud-side server and a game terminal device, where the cloud-side server includes a device system or program for controlling the cloud gaming function, and the game terminal is used to control various operations of the user during the user's game playing process.

[0030] In the cloud gaming scenario, the game does not run on the player's game terminal device, but runs on the cloud server, and the game scene is rendered into an audio-video stream by the cloud server and transmitted to the player's game terminal device through the network. The player's game terminal device does not need to have powerful graphics calculation and data processing capabilities, but only needs to have basic streaming media playback capabilities and the ability to receive input commands from the player and send them to the cloud server.

[0031] According to the embodiment of the present application, when the cloud application is a cloud game, the device control method based on the cloud game may run on a personal mobile terminal, may run on a cloud-side server, or may even run on a third-party device, thereby providing control of the cloud game. A specific device control system for the cloud game may run on the device in the form of a program, may run as a system component on the device, or may even be a type of cloud-side service program. The specific operating mode is determined according to the actual scenario, and is not limited here.

[0032] This application proposes a device control method based on cloud applications, which can be applied in a cloud application device control framework as shown in Fig. 2. Fig. 2 is a schematic diagram of a cloud application device control framework in an embodiment of this application, which includes an application client 210 on a server, a cloud application server 220, and a cloud application client 230 on a terminal device.

[0033] Here, the application client 210 issues a command to the cloud application server 220 through the terminal recognition module 211, and the command finally reaches the cloud application client 230, so that the cloud application client 230 can easily inquire whether the terminal device (e.g., a smart terminal) where the cloud application client 230 is currently located or the physical device associated with the cloud application client 230 supports vibration feedback and the format content of the supported vibration feedback.

[0034] The terminal control module 213 establishes communication with the cloud application client 230 based on the information provided by the terminal recognition module 211. The communication is relayed through the cloud application server 220, and issues vibration feedback information through the established communication.

[0035] The vibration feedback control module 212 of the application client 210 triggers corresponding vibration feedback based on different application scenes, obtains an actual file of vibration feedback information, and submits the file to the cloud application server 220 via the terminal control module 213, for relay by the cloud application server 220, and finally submitted to the cloud application client 230.

[0036] The cloud application server 220 establishes a link with the cloud application client 230 through the terminal communication module 221, obtains supporting information of vibration feedback from the cloud application client 230, and issues a control protocol of vibration feedback to the cloud application client 230. The vibration protocol packaging module 222 is used to receive vibration feedback information issued by the application client 210, package the vibration feedback information according to a defined protocol, and transmit it to the cloud application client 230 through the terminal communication module 221.

[0037] The cloud application client 230 establishes a communication connection with the cloud application server 220 according to a request from the cloud application server 220 through the communication module 231, and receives information such as a hardware inquiry command and a vibration feedback command issued by the cloud application server 220. After receiving the vibration feedback command from the communication module 231, the vibration feedback analysis module 232 performs analysis according to a defined protocol and calls the hardware control module 233. The hardware control module 233 receives the command from the vibration feedback analysis module 232, and calls the corresponding interface of the terminal device to drive the actual hardware, thereby realizing vibration feedback.

[0038] Specifically, when the cloud application is a cloud game, application client 210 refers to the game client, cloud application server 220 refers to the cloud game server, and cloud application client 230 refers to the cloud game client.

[0039] In order to combine application scenes to trigger vibration feedback, the special effect design usually needs to display a dedicated effect based on the application scene. Specifically, for ease of explanation, refer to Fig. 3, which is a schematic diagram of a vibration special effect design process in an embodiment of the present application, including interactions between an application client 310 on a server, a cloud application server 320, and a cloud application client 330 on a terminal device.

[0040] Here, effect design 311 refers to the server, or an application client 310 on the server, designing a dedicated vibration feedback effect for a particular scene.

[0041] Scene recognition 312 refers to the server or application client 310 recognizing a corresponding scene during the operation of an application (e.g., a game) and matching the relevant special effect description file. For example, in a shooting game, a user can use different guns to trigger different vibration feedbacks, and when the user's virtual character is attacked, vibrations of different intensities are triggered according to the difference in the amount of hit points lost, so that the user can directly sense the amount of hit points lost through the vibrations. Therefore, when the user encounters an enemy, a high-frequency short vibration is triggered to prompt the user that danger is approaching.

[0042] Vibration trigger 313 refers to an application client 310 on a server issuing a vibration request to request that a vibration trigger interface of a physical device be invoked. However, because the cloud application is rendered on the cloud application server 320, this request cannot be communicated directly to the physical device and therefore needs to be relayed through the cloud application server 320.

[0043] The transparent transmission terminal 321 refers to the cloud application server 320 receiving a vibration request issued by the application client 310 and then transparently transmitting it to the cloud application client 330 through a communication connection.

[0044] The system trigger 331 refers to the cloud application client 330 invoking the interface of the physical device to trigger the vibration of the hardware on the terminal device after receiving a vibration request transparently transmitted from the cloud application server 320.

[0045] It can be understood that the method provided in the present application may be written as a program, may be a processing logic in a hardware system, or may be a control device for a cloud application, and may be integrated or externally implemented to realize the processing logic.

[0046] Combining the above process architecture, the present application will now describe a method for controlling a device based on a cloud application. Referring to Fig. 4, Fig. 4 is a flowchart of a method for controlling a device based on a cloud application provided by an embodiment of the present application, which is executed by an electronic device, for example, by the application client 210 on the server in Fig. 2, and includes the following steps:

[0047] Step S301: Establish a communication connection with a cloud application client on a terminal device, the cloud application client being used for presenting a received cloud application video stream.

[0048] In this embodiment, an application client on a server establishes a communication connection with a cloud application client on a terminal device, where the application client is a specific application program running on the server, and the cloud application client is an application on the terminal device used to present the received cloud application video stream.

[0049] Specifically, when the application client is a game client, the cloud application client is a cloud game client. For example, taking a racing game as an example, the cloud game client may be specifically a browser, a player, or a dedicated client on a mobile terminal, and is mainly used to play game screens provided by a server, and the game client is the actual application program of the racing game running on the server.

[0050] The communication connection in this embodiment is mainly used to transmit information related to vibration feedback, and the communication connection may be a dedicated communication channel, and may also be used for video stream communication and control information communication between the application client and the cloud application client.

[0051] Step S302: Perform scene recognition for a cloud application client and a corresponding cloud application, and obtain a scene recognition result, where the scene recognition result includes media content associated with the recognized cloud application scene.

[0052] In this embodiment, the cloud application is actually run by an application client on a server. The application client can recognize the scene in which the cloud application is located based on the execution status of the cloud application and the user's operation. For example, a game scene usually includes contents such as game progress, game screen, game data, etc., and changes in various data during the game process may cause changes in the game scene, such as changes in the environment in which a virtual character in a game is located, changes in attributes such as hit points, the virtual character encountering various events, or interacting with the environment.

[0053] The application client can recognize a specific cloud application scene based on the data content and data changes of the cloud application, thereby obtaining a scene recognition result. The scene recognition result includes media content associated with the recognized cloud application scene and is used to indicate the presentation of the virtual character in the next time period of the cloud application. For example, in a cloud game, if a virtual character falls from a high place, the scene recognition result can indicate that in a next time period, the virtual character falls to the ground and the damage caused by the fall from the high place.

[0054] Step S303: Generate vibration feedback information corresponding to a cloud application scene based on a scene recognition result.

[0055] Specifically, the cloud application scene identified in the scene recognition result can determine the changes or events that will occur in the cloud application in the next one time period, thereby determining the vibration feedback that needs to be triggered in the cloud application scene, and further generate vibration feedback information corresponding to the recognized cloud application scene. The vibration feedback information includes the type of vibration feedback that needs to be performed and the parameters for performing the vibration feedback, such as the execution time, etc.

[0056] Depending on the types of vibration feedback supported by the physical device associated with the cloud application client, the vibration feedback information may include parameters corresponding to one or more types of vibration feedback.

[0057] According to the specific content of the cloud application scene, the vibration feedback information can include parameters required for performing multiple vibration feedbacks. For example, in the cloud game scene, if the target character has three impacts with an obstacle during the descending period, and each impact needs vibration feedback, the vibration feedback information includes vibration feedback parameters for three sets of vibrations corresponding to the three impacts.

[0058] Step S304: Send a video stream corresponding to the cloud application scene and vibration feedback information to the cloud application client, so that when the cloud application client presents the video stream of the cloud application scene, based on the vibration feedback information, a physical device associated with the cloud application client is controlled to perform a vibration feedback operation.

[0059] Specifically, for ease of description, refer to FIG. 5, which shows a process of triggering vibration feedback in an embodiment of the present application, including the following steps:

[0060] Step 501: An application client sends a video stream and vibration feedback information corresponding to a cloud application scene to a cloud application client through a cloud application server.

[0061] Here, the video stream of the cloud application scene is a video stream rendered based on the application screen to be displayed in the cloud application scene.

[0062] Step 502: After receiving the video stream, the cloud application client plays the video stream for a user.

[0063] In this step, the cloud application client displays video content to the user, for example, the results generated by the user's actions.

[0064] Step 503: When playing the video stream, the cloud application client can further control the associated physical device to perform a vibration feedback operation based on the received vibration feedback information.

[0065] Here, the physical device associated with the cloud application client may be the smart terminal where the cloud application client is located.

[0066] For example, the cloud application is a racing game, and the smart terminal where the cloud application client is located is a mobile terminal. In the racing game, when a user operates a vehicle driven on the mobile terminal to collide with an opponent, a screen of the vehicle colliding with the opponent can be displayed in the video stream of the cloud game scene transmitted by the game client on the server, and when the video stream replays the vehicle colliding with the opponent, the cloud game client can further control the mobile terminal to vibrate for one second, thereby expressing the impact force of the collision and improving the user experience and immersion.

[0067] As can be seen, through vibration feedback operation, combined with the video stream of the presented cloud application scene, a more stereoscopic gaming experience can be achieved.

[0068] Step 504: After completing the vibration feedback, the physical device returns the vibration execution result to the cloud application client.

[0069] Step 505: The cloud application client generates a vibration feedback response message including the execution result of the vibration, and sends the vibration feedback response message to the application client through the cloud application server.

[0070] In the embodiment of the present application, a scene recognition result is obtained through scene recognition, vibration feedback information corresponding to the cloud application scene is generated, and the vibration feedback information is sent to the cloud application client through a communication connection with the cloud application client, so as to control the associated physical device to perform a vibration feedback operation.

[0071] 1) In the cloud application scene, by opening the full link of the cloud application, the vibration driving signal is issued in real time, but the delay is kept consistent with the video stream of the cloud application, allowing the user to experience vibration without delay, improving the accuracy of vibration feedback and thereby improving the user experience.

[0072] 2) Based on scene recognition, appropriate vibration feedback information can be designed to better match the user's actual operation, and can assist the user in their operation and experience, allowing the user to be more immersed in the application content (e.g., game content) itself, while at the same time making the most of the hardware advantages of the physical device.

[0073] In one embodiment of the present application, the above step S301 of establishing a communication connection with a cloud application client on a terminal device may include the following steps:

[0074] A communication establishment request is sent to the cloud application client.

[0075] A communication response message returned from the cloud application client is received.

[0076] A communication connection is established with the cloud application client based on the communication response message.

[0077] Here, the communication response message includes the vibration feedback type and the vibration feedback protocol version supported by the physical device.

[0078] The step of generating vibration feedback information corresponding to the cloud application scene based on the scene recognition result includes: The step of generating the vibration feedback information based on the scene recognition result, the vibration feedback type and the vibration feedback protocol version.

[0079] Specifically, the application client on the server sends a communication establishment request to the cloud application client. The communication establishment request can be sent based on an already established audio-video communication link or control communication link between the application client and the cloud application client, and can be made at any time before the cloud application is officially launched. For example, when the cloud application client logs into an account or establishes other communication connections to the cloud application client, the communication connection used for vibration feedback is established together.

[0080] Specifically, for ease of description, refer to FIG. 6, which illustrates a process of establishing a communication connection with a cloud application client in an embodiment of the present application, including the following steps:

[0081] Step 601: An application client sends a communication establishment request to a cloud application client through a cloud application server based on a list of cloud application clients that have already logged in to the server.

[0082] Step 602: When the cloud application client receives the communication establishment request, it establishes a communication connection with the application client based on the request.

[0083] Specifically, the connection establishment may be a method similar to a three-way handshake process of a network connection query and acknowledgement, or may be performed through a method of storing a server's Internet Protocol address, etc.

[0084] Step 603: The cloud application client inquires about the vibration feedback type and the vibration feedback protocol version supported by the associated physical device, and adds the inquired information to a communication response message.

[0085] Here, the vibration feedback type mainly refers to a function or module that a physical device has and can be used to perform vibration feedback. The protocol version of the vibration feedback defines a specific mode in which the vibration feedback can be performed, for example, the vibration can include long vibration, short vibration, vibration that combines long vibration and short vibration, and vibration of different frequencies and intensities, etc., and other types of vibration feedback are similar.

[0086] Step 604: The cloud application client sends a communication response message to the application client through the cloud application server.

[0087] Step 605: The application client generates vibration feedback information based on the received vibration feedback type, the vibration feedback protocol version, and the recognized application scene.

[0088] In an embodiment of the present application, by establishing a communication connection used for transmitting vibration feedback information between an application client on a server and a cloud application client, vibration feedback can be sent from the server to a terminal device in a timely manner, providing a basis for the server to control the terminal to perform vibration feedback, and improving the feasibility of the method.

[0089] In one embodiment of the present application, the above step S302 of performing scene recognition for a cloud application client and a corresponding cloud application to obtain a scene recognition result may include the following steps:

[0090] A character attribute value at a current time of a target virtual character in the cloud application is obtained.

[0091] The cloud application scene is recognized based on the character attribute value to obtain the scene recognition result.

[0092] Specifically, depending on the type of cloud application, the scene recognition result may be determined based on an event that occurs to the target virtual character itself in the cloud application, or may be determined based on the time period in the cloud application.

[0093] The target virtual character may typically be a user-controlled character of the cloud application, or a non-user-controlled character that needs to be displayed, etc. The character attribute values ​​may include data related to the application content, such as the character's hit points, health values, etc., or data related to the application actions, such as the spatial location where the character is located in the application, the steps taken, etc.

[0094] According to the character attribute value of the target virtual character at the current time, the scene of the cloud application can be recognized to obtain a scene recognition result. Specifically, if the character attribute value is higher or lower than a predetermined value, it can be recognized as a certain scene, or if a change of a predetermined range occurs in the character attribute value, it can be recognized as a scene that needs to trigger vibration.

[0095] For example, for a character's hit points, if the hit points are lower than a set threshold, a scene in which the hit points are too low can be recognized, and in the scene, the terminal device can be controlled to vibrate or flash according to a predetermined frequency, thereby prompting the user and creating a tense atmosphere.

[0096] Alternatively, the above step S302 of performing scene recognition for a cloud application client and a corresponding cloud application to obtain a scene recognition result may include the following steps:

[0097] According to a mapping relationship between a preset time period and a scene identifier, a scene identifier of a target virtual character at a current time is obtained, and the scene identifier is used to represent an environment in which the target virtual character is located.

[0098] The scene recognition result is determined based on a mapping relationship between a preset scene identifier and a scene type, and the scene identifier at the current time.

[0099] Here, a mapping relationship between a time zone where the cloud application is located and a scene identifier is preset, where the time zone can refer to a progress where the cloud application is located, and the scene identifier is used to represent a surrounding environment where the target virtual character is located.

[0100] Specifically, a scene of a cloud application may have an association relationship with the progress (length of time) that it has made. For example, for a parkour-type game, the game progress progresses with time, and when different time periods are reached, different game scenes are reached and correspond to different scene identifiers, and the scene identifier at the current time of the target virtual character can be determined based on the time period.

[0101] Or the time period may be a time period that starts recording from one particular point in time in the cloud application, for example a time period recorded by starting timing after a virtual character enters a room in a game.

[0102] Based on the mapping relationship between the preset scene identifiers and scene types, the application client can determine the scene recognition result. Specifically, the scene types are used to identify specific classifications of different scenes, and each scene type can correspond to multiple scene identifiers and require various different vibration feedbacks.

[0103] For example, the application client can determine based on the mapping relationship between time period and scene identifier that if the user's virtual character spends too much time in a certain game scene, it will fall into a trap, and the scene identifier identifies the scene in which the user falls into the trap, and the scene type corresponding to the scene identifier is a drop type, and therefore, further determine based on the drop type that the corresponding scene recognition result will be damaged.

[0104] In addition, the target scene type is used to identify a scene that needs vibration feedback. Therefore, if the scene recognition result does not belong to the target scene type, the motion of the vibration feedback will not be subsequently controlled to perform the physical device. For example, if the scene type is a relatively calm scene such as recovery or rest, the vibration feedback will not be performed, and the vibration feedback of the physical device will not be subsequently controlled.

[0105] In the embodiments of the present application, a specific method for scene recognition is provided, which can perform targeted recognition for application scenes, and is beneficial for improving the degree of matching between the subsequent required vibration feedback and the current scene content.

[0106] In one embodiment of the present application, in the cloud application, the step of obtaining a character attribute value of a target virtual character at a current time may include the following steps:

[0107] A character control command for the target virtual character sent from the cloud application client is received.

[0108] Based on the character control command, control is performed on the target virtual character to obtain a character attribute value of the target virtual character at the current time.

[0109] The step of recognizing the cloud application scene based on the character attribute value to obtain the scene recognition result may include the following steps.

[0110] An attribute value change range is determined based on the character attribute value of the target virtual character at a previous time and the character attribute value at the current time.

[0111] The scene recognition result is determined based on the attribute value variation range.

[0112] In this embodiment, the scene recognition result can be determined based on the user's operation on the target virtual character. Specifically, taking a game client and a cloud game client as an example, the game client receives a character control command for the target virtual character sent from the cloud game client. The character control command may be a motion control such as moving, attacking, or avoiding, or may be a motion such as interaction control with other physics in the scene.

[0113] The game client controls the target virtual character in the cloud game according to the character control command, and obtains the result of the control, thereby obtaining the character attribute value of the target virtual character at the current time, for example, when the target virtual character moves, the character is attacked, causing a decrease in hit points, thereby obtaining the hit points of the target virtual character at the current time.

[0114] Next, the game client determines an attribute value change range based on the character attribute value of the target virtual character at a previous time and the character attribute value at the current time. The previous time refers to a time before the game client exercises control over the target virtual character. The attribute value change range can be determined based on a comparison of the character attribute values ​​at the two times.

[0115] Furthermore, the scene recognition result is determined based on the attribute value change range. For example, if the hit point is greatly reduced, it can be determined that the scene recognition result will receive a large amount of damage, which then triggers a physical device connected to the terminal to vibrate to prompt the user.

[0116] In an embodiment of the present application, the scene recognition result is determined through changes in the attribute values ​​of the target virtual character, so that the scene recognition result can be determined in a timely manner during the cloud game process, and the granularity of scene recognition can be improved, thereby making it possible to more accurately determine the situations in which vibration feedback is required.

[0117] In one embodiment of the present application, based on the above embodiment, the vibration feedback information includes a vibration feedback type, and the cloud application client controls the physical device to perform a vibration feedback operation indicated by the vibration feedback type.

[0118] Specifically, the vibration feedback types supported by different physical devices may be different. For example, the strength and type of vibration special effects that a mobile terminal and other game peripheral devices can support will be different. When the game client generates vibration feedback information, the game client generates the vibration feedback information based on the recorded vibration feedback types supported by the physical devices and the vibration feedback that needs to be performed in the recognized scene.

[0119] When the cloud game client needs to perform vibration feedback after receiving the vibration feedback information, the cloud game client controls the physical device to perform the vibration feedback operation indicated by the vibration feedback type. For example, if the game client determines that one long vibration and multiple short vibrations need to be performed in the current scene, the game client generates vibration feedback information including a long vibration special effect type and a short vibration special effect type, and the cloud game client controls the mobile terminal to perform long vibration and short vibration after receiving the vibration feedback information.

[0120] In the embodiment of the present application, the vibration feedback type is defined in the vibration feedback information, so that different types of vibration feedback can be performed, thereby improving the diversity of vibration feedback.

[0121] In one embodiment of the present application, based on the above embodiment, the vibration feedback information includes a time parameter of the vibration feedback, the time parameter is used to indicate a time period for performing the vibration feedback, and the cloud application client controls the physical device to perform a vibration feedback operation within the time period indicated by the time parameter.

[0122] Specifically, the time parameters of the vibration feedback can include a start time and a duration of the vibration, and the start time and the duration of the vibration can be determined according to the needs of different scenes, where the start time of the vibration is usually a relative time, which is set relative to the start time of the corresponding video stream, thereby determining when to start executing the vibration feedback operation.

[0123] For example, if the start time of the vibration special effect is set to 10 ms, the cloud gaming client starts to control the physical device to vibrate after playing the corresponding video stream for 10 ms. The duration of the vibration determines the end time of the vibration feedback. In one embodiment, the vibration duration is not included and can be replaced with a predetermined uniform duration.

[0124] It should be understood that the duration of the vibration does not necessarily indicate that the vibration feedback motion must be continuous within the duration, for example, the vibration feedback motion may be an intermittent vibration, and within the duration, the physical device vibrates intermittently until the total time length reaches a value limited by the duration.

[0125] In an embodiment of the present application, the vibration feedback information includes a time parameter of the vibration feedback, which allows the start time and end time of the vibration feedback operation to be adjusted when playing a video stream, thereby making the vibration feedback operation more flexibly triggered and improving the flexibility of the means.

[0126] In one embodiment of the present application, based on the above embodiment, the vibration feedback information includes an execution frequency of the vibration feedback, and the cloud application client controls the physical device to perform the vibration feedback operation according to the execution frequency of the vibration feedback.

[0127] Specifically, the execution frequency of the vibration feedback defines the frequency at which the motion related to the vibration feedback needs to reach by the time the vibration feedback operation is ended. The execution frequency of the vibration feedback can be determined according to the needs of different scenes, for example, different vibration execution frequencies can be set for different urgencies.

[0128] For example, the vibration execution frequency represents the vibration frequency that needs to be reached when the vibration feedback is terminated, and the possible value range can be set to a relative value of 0 to 100. In one embodiment, the vibration feedback executed at the previous time may be continuous with the vibration feedback to be executed at the current time, and if the execution frequencies are different, adjustments can be made to the frequency to change it continuously, but without the need to stop and restart. For example, if the vibration frequency at the previous time was 50 and the vibration frequency that needs to be executed at the current time is 70, the vibration frequency can be uniformly adjusted from 50 to 70 according to a preset step size.

[0129] For the first period, or for a situation where the same type of vibration feedback was not performed at the previous time, the execution frequency at the previous time is implicitly accepted as 0.

[0130] In an embodiment of the present application, the vibration feedback information includes an execution frequency of the vibration feedback, so that vibration feedback can be performed at different frequencies based on the expression of different scenes, thereby creating different atmospheres and improving the flexibility and expressiveness of special effects.

[0131] In one embodiment of the present application, based on the above embodiment, the vibration feedback information includes an execution intensity of the vibration feedback, and the cloud application client controls the physical device to perform the vibration feedback operation according to the execution intensity of the vibration feedback.

[0132] Specifically, the execution strength of the vibration feedback defines the vibration strength that the motion related to the vibration feedback needs to reach when the vibration feedback operation ends. The execution strength of the vibration feedback can be determined according to the needs of different scenes, for example, different vibration execution strengths can be set according to different degrees of seriousness.

[0133] The execution intensity can be expressed as information such as amplitude, and the range of possible values ​​can be set to a relative value of 0-100.

[0134] In one embodiment, the vibration feedback performed at a previous time may be continuous with the vibration feedback to be performed at the current time, and if the execution intensity is different, it can be changed continuously by adjusting the intensity, but does not need to be stopped and then restarted.

[0135] For example, if the vibration strength at the previous time was 50 and the vibration strength that needs to be performed at the current time is 70, the vibration strength can be uniformly adjusted from 50 to 70 according to a preset step size.

[0136] For a situation in which the same type of vibration feedback was not performed in the first period or the previous time, the execution intensity for the previous time is implicitly accepted as 0.

[0137] In an embodiment of the present application, the vibration feedback information includes an execution intensity of the vibration feedback, so that vibration feedback of different intensities can be performed based on the representation of different scenes, thereby being more consistent with the game content and improving the diversity of the vibration feedback.

[0138] In one embodiment of the present application, according to the above embodiment, the step S304 of sending the video stream corresponding to the cloud application scene and the vibration feedback information to the cloud application client includes: The method includes sending a video stream corresponding to the cloud application scene and the vibration feedback information to a cloud application server, where the cloud application server packages the vibration feedback information based on a vibration protocol, and sends the packaging result and the video stream to the cloud application client.

[0139] In this embodiment, the application client can run on a single server, while the cloud application server is another server. The application client's server and the cloud application server communicate with each other through a wired or wireless network. The application client sends a video stream corresponding to the rendered cloud application scene and the generated vibration feedback information to the cloud application server, and the cloud application server packages the vibration feedback information according to a vibration protocol, and sends the packaged result and the video stream to the cloud application client.

[0140] Specifically, the vibration protocol for the vibration special effect may be defined as shown in Table 1 below.

[0141] [Table 1]

[0142] Based on the vibration feedback information sent from the application client, the cloud application server packages the vibration special effect into a data packet consisting of a start time, duration, vibration frequency, and vibration intensity according to the above protocol, and then transmits it to the cloud application client together with the video stream, thereby controlling the vibration special effect.

[0143] In an embodiment of the present application, the application client sends the video stream and vibration feedback information to the cloud application client through the cloud application server, allowing multiple application clients to share the same cloud application server, thereby reducing the operating costs of the cloud application.

[0144] The present application provides a method for controlling a device based on a cloud application, which is applied to a cloud application client. Figure 7 shows a flowchart of a method for controlling a device based on a cloud application provided in another embodiment of the present application, which includes the following steps:

[0145] Step S701: Establish a communication connection with a cloud application server.

[0146] Step S702: Receive a video stream and vibration feedback information corresponding to a cloud application scene sent from a cloud application server, where the vibration feedback information is generated by an application client on the server according to the cloud application scene.

[0147] Step S703: Play a video stream corresponding to a cloud application scene, and according to the vibration feedback information, control a physical device associated with the cloud application client to perform a vibration feedback operation.

[0148] Specifically, a cloud application client establishes a communication connection with a cloud application server. The communication establishment process is usually initiated by an application client. The cloud application client launches a cloud application after logging in to a cloud application platform. The cloud application platform will launch a corresponding application client on the server, and the application client actively establishes communication with the cloud application client. The cloud application client receives a communication establishment request sent by the application client and locally stores it for sending subsequent control information and various response messages.

[0149] After establishing communication, the cloud application client will receive the video stream and vibration feedback information corresponding to the cloud application scene from the cloud application server. Taking the cloud game as an example, the game play in the cloud game is controlled by the cloud game client, and the game client on the server executes the game content according to the user's operation, renders the game scene of the cloud game into a video stream, and generates vibration feedback information according to the content of the game scene. The cloud game client plays the video stream of the cloud game scene to the user, thereby displaying the game screen and the operation result to the user to play the subsequent game, and controls the physical device associated with the cloud game client to perform the vibration feedback operation. For example, the mobile terminal is controlled to vibrate according to the vibration information in the vibration feedback information.

[0150] In an embodiment of the present application, a cloud application client establishes a communication connection with a cloud application server, displays application content based on the received video stream and vibration feedback information, and performs vibration feedback operations, so that in cloud application scenes, the application client on the server can directly trigger vibration of a physical device associated with the cloud application client, thereby avoiding the situation of mismatching with application content that occurs when vibration is based on audio or video content, and improving the accuracy of vibration feedback.

[0151] In one embodiment of the present application, based on the above embodiment, the step of establishing a communication connection with the cloud application server may include the following steps:

[0152] A communication establishment request sent from the cloud application server is received.

[0153] Based on the communication establishment request, the physical device inquires about a vibration feedback type and a vibration feedback protocol version supported by the physical device.

[0154] A communication response message is sent to the cloud application server, and the communication response message includes the vibration feedback type and a protocol version of the vibration feedback, so that the application client generates the vibration feedback information based on the vibration feedback type and the protocol version of the vibration feedback.

[0155] Specifically, taking cloud gaming as an example, a cloud game client receives a communication establishment request sent from a cloud game server. The communication establishment request usually includes communication parameters required for communication, and the cloud game client establishes a communication connection with the cloud game server based on the communication establishment request. The communication establishment request further includes a special effect inquiry command, and the cloud game client inquires about the vibration feedback type and the vibration feedback protocol version supported by the physical device associated with the cloud game client based on the inquiry command. The vibration feedback type is mainly used to identify the vibration type that the physical device can perform, such as the vibration length, mode, etc. The protocol version is mainly used to specify the parameters and communication format, etc. that need to be used when defining a vibration feedback operation.

[0156] The cloud game client sends a communication response message to the cloud game server, the communication response message includes a vibration feedback type and a protocol version of the vibration feedback, and the communication response message is used to allow the cloud game server to establish a communication connection with the cloud game client. Specifically, the communication response message can include communication parameters of the cloud game client, such as an Internet address, so that the cloud game server can establish a corresponding communication connection.

[0157] In the embodiments of the present application, a specific manner for establishing communication between a cloud game client and a cloud game server is specifically introduced, which improves the feasibility of the implementation.

[0158] In one embodiment of the present application, based on the above embodiment, the physical device associated with the cloud application client further includes a third-party device, and the step of controlling the physical device associated with the cloud application client to perform a vibration feedback operation based on the vibration feedback information includes: generating vibration control commands for a third-party device based on the vibration feedback information; and sending a vibration control command to the third-party device, whereby the third-party device performs a vibration feedback operation based on the vibration feedback information.

[0159] The third-party device refers to a device other than the one used for controlling the application and displaying the cloud application content. The third-party device may be a material network device, and communicates with a terminal device where the cloud application client is located through a network. The cloud application client generates a vibration control command for the third-party device based on the vibration feedback information. The vibration control command includes parameters used for performing vibration, which are generated based on the vibration feedback information. For example, the vibration start time, end time, display mode, intensity, and other numerical values ​​are included. The cloud application client sends the vibration control command to the third-party device, so that the third-party device performs a vibration feedback operation based on the vibration feedback information.

[0160] Taking cloud gaming as an example, assuming that a user is using a mobile terminal to play an exploration-type game and controls a character to enter a relatively bumpy environment, the vibration feedback information sent by the game client to the cloud game client may include information regarding seat sway control, and the cloud game client generates a vibration control command used for seat sway control based on the vibration feedback information and sends it to the smart seat in the room, and the smart seat sways at a lower adjustment based on the received vibration control command, thereby giving the user a more immersive feeling.

[0161] In an embodiment of the present application, the cloud game client further controls a third-party device to collaboratively perform vibration feedback, thereby enriching the vibration display process and improving the vibration display effect.

[0162] It should be noted that the figures depict the steps of the methods herein in a particular order, however this does not require or imply that the steps must be performed in that particular order to achieve the desired results, or that all steps shown must be performed, and as additional or alternative options, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0163] Hereinafter, an embodiment of the device of the present application that can be used to execute the device control method based on the cloud application in the above embodiment of the present application will be introduced. FIG. 8A is a schematic block diagram showing the configuration of a device control device based on the cloud application in the embodiment of the present application, which is used to execute the process related to the application client on the server in the above embodiment. As shown in FIG. 8A, the device control device 800: A communication establishment module 801 used for establishing a communication connection between a cloud application client on a terminal device, the cloud application client used for presenting a received cloud application video stream; a scene recognition module 802 adapted to perform scene recognition on a cloud application corresponding to the cloud application client to obtain a scene recognition result, the scene recognition result including media content associated with a recognized cloud application scene; a vibration feedback information generating module 803, which is used to generate vibration feedback information corresponding to the cloud application scene according to the scene recognition result; and a vibration feedback information sending module 804 used for sending a video stream corresponding to the cloud application scene and the vibration feedback information to the cloud application client, the vibration feedback information sending module 804 controlling a physical device associated with the cloud application client to perform a vibration feedback operation based on the vibration feedback information when the cloud application client presents the video stream of the cloud application scene.

[0164] In some embodiments of the present application, based on the above technical solutions, the communication establishment module 801: a communication establishment request sending unit, used for sending a communication establishment request to the cloud application client; a communication response message receiving unit, which is used to receive a communication response message returned from the cloud application client, and establish a communication connection with the cloud application client according to the communication response message; wherein the communication response message includes a vibration feedback type and a vibration feedback protocol version supported by the physical device; The scene recognition module 802 is used for generating the vibration feedback information according to the scene recognition result, and the vibration feedback type and the vibration feedback protocol version.

[0165] In some embodiments of the present application, based on the above technical solutions, the scene recognition module 802: A character attribute value acquisition unit, which is used to acquire a character attribute value of a target virtual character at a current time in the cloud application; a scene recognition unit used for recognizing the cloud application scene based on the character attribute value to obtain the scene recognition result.

[0166] In some embodiments of the present application, based on the above technical solutions, the scene recognition module 802: A scene identifier acquisition unit, which is used to acquire a scene identifier of a target virtual character at a current time according to a mapping relationship between a preset time period and a scene identifier, the scene identifier being used to represent an environment in which the target virtual character is located; a scene recognition result determining unit, which is used to determine the scene recognition result according to a mapping relationship between a preset scene identifier and a scene type and the scene identifier at a current time.

[0167] In some embodiments of the present application, based on the above technical solutions, the character attribute value obtaining unit: a character control command receiving subunit, which is used to receive a character control command for the target virtual character sent from the cloud application client; a virtual character control subunit, which is used for performing control on the target virtual character based on the character control command to obtain a character attribute value of the target virtual character at the current time; The scene recognition unit is an attribute value change range determining subunit, which is used to determine an attribute value change range according to the character attribute value of the target virtual character at a previous time and the character attribute value of the target virtual character at the current time; and a recognition result determining subunit, which is used for determining the scene recognition result based on the attribute value variation range.

[0168] In some embodiments of the present application, based on the above technical solutions, the vibration feedback information includes a vibration feedback type, and the cloud application client controls the physical device to perform a vibration feedback operation indicated by the vibration feedback type.

[0169] In some embodiments of the present application, based on the above technical solutions, the vibration feedback information includes a time parameter of vibration feedback, the time parameter is used to indicate a time period for performing vibration feedback, and the cloud application client controls the physical device to perform a vibration feedback operation within the time period indicated by the time parameter.

[0170] In some embodiments of the present application, based on the above technical solutions, the vibration feedback information includes an execution frequency of vibration feedback, and the cloud application client controls the physical device to perform the vibration feedback operation according to the execution frequency of the vibration feedback.

[0171] In some embodiments of the present application, based on the above technical solutions, the vibration feedback information includes an execution intensity of vibration feedback, and the cloud application client controls the physical device to perform the vibration feedback operation according to the execution intensity of the vibration feedback.

[0172] In some embodiments of the present application, based on the above technical solutions, the vibration feedback information transmitting module 804: The cloud application server includes a server communication unit, which is used for sending a video stream corresponding to the cloud application scene and the vibration feedback information to a cloud application server, so that the cloud application server packages the vibration feedback information according to a vibration protocol, and sends the packaging result and the video stream to the cloud application client.

[0173] According to another aspect of the embodiment of the present application, a device control device based on a cloud application is provided. FIG. 8B is a schematic block diagram of a device control device based on a cloud application according to another embodiment of the present application, which is used to execute processes related to the cloud application client according to the above embodiment. As shown in FIG. 8B, the device control device 810 includes: A communication connection establishment module 811 used for establishing a communication connection with a cloud application server; a vibration feedback information receiving module 812, which is used to receive a video stream corresponding to a cloud application scene and vibration feedback information sent from the cloud application server, where the vibration feedback information is generated by an application client on a server according to the cloud application scene; a playback module 813, which is used for playing video streams corresponding to the cloud application scenes; and a vibration feedback execution module 814, which is used to control a physical device associated with the cloud application client to perform a vibration feedback operation based on the vibration feedback information.

[0174] In some embodiments of the present application, based on the above technical solutions, the communication connection establishment module 811: a communication establishment request receiving unit, which is used to receive a communication establishment request sent from the cloud application server; a vibration feedback inquiring unit, which is used to inquire about a vibration feedback type and a vibration feedback protocol version supported by the physical device according to the communication establishment request; and a communication response message sending unit used for sending a communication response message to the cloud application server, the communication response message including the vibration feedback type and a protocol version of the vibration feedback, whereby the application client generates the vibration feedback information based on the vibration feedback type and the protocol version of the vibration feedback.

[0175] In some embodiments of the present application, based on the above technical solutions, the physical device associated with the cloud application client further includes a third-party device, and the vibration feedback execution module: a vibration control command generation module that generates a vibration control command for the third-party device based on the vibration feedback information; The device further includes a vibration control command sending module, which is used to send the vibration control command to the third-party device so that the third-party device performs a vibration feedback operation based on the vibration feedback information.

[0176] It is necessary to explain that the apparatus provided in the above embodiments and the method provided in the above embodiments belong to the same idea, and the specific manner in which each module performs operations has already been described in detail in the method embodiments, so a detailed description will be omitted here.

[0177] FIG. 9 shows a structural schematic diagram of an electronic device suitable for use in implementing the embodiments of the present application.

[0178] It should be noted that the electronic device 900 shown in FIG. 9 is merely an example and should not pose any limitation to the functionality and scope of use of the embodiments of the present application.

[0179] As shown in Fig. 9, the electronic device 900 includes a central processing unit (CPU) 901, which can execute various appropriate motions and processes based on programs stored in a read-only memory (ROM) 902 or programs loaded from a storage portion 908 into a random access memory (RAM) 903. Various programs and data required for system operation are further stored in the RAM 903. The CPU 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0180] The following members are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc., an output section 907 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc., a storage section 908 including a hard disk, etc., and a communication section 909 including, for example, a network interface card such as a LAN (Local Area Network) card and a modem. The communication section 909 executes communication processing via a network such as the Internet. A driver 910 is also connected to the I / O interface 905 as necessary. A removable medium 911, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the driver 910 as necessary, and a computer program read from the driver 910 is thereby installed in the storage section 908 as necessary.

[0181] In particular, according to the embodiment of the present application, the process described in each method flowchart can be realized as a computer software program. For example, the embodiment of the present application includes a computer program product, which includes a computer program stored in a computer-readable medium, and the computer program includes program codes used to execute the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 909 and / or installed from a removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, it performs various functions defined in the system of the present application.

[0182] It is necessary to explain that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium, or a computer-readable storage medium, or any combination of the above. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to, an electrical connection having one or more conductors, a portable computer magnetic disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact magnetic disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program, which may be used in or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagating in baseband or as part of a carrier wave, in which computer-readable program code is embodied. Such propagated data signals may take several types of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may transmit, propagate, or transmit a program for use in or in conjunction with an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium may be transmitted by any suitable medium, including, but not limited to, wireless, wired, etc., or any suitable combination of the above.

[0183] The flowcharts and block diagrams in the drawings illustrate possible system architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams may represent a module, program segment, or part of code, which includes one or more executable instructions used to realize a specified logic function. It should be further noted that in some permutation realizations, the functions marked in the blocks may occur in a different order from the order marked in the drawings. For example, two blocks shown in succession may actually be essentially executed in parallel, and they may be executed in the opposite order, depending on the related functions. It should be further noted that each block in the block diagrams or flowcharts, and combinations of blocks in the block diagrams or flowcharts may be realized in a system based on dedicated hardware that executes the specified functions or operations, or in a combination of dedicated hardware and computer instructions.

[0184] It should be noted that, although the above detailed description refers to some modules or units of the device used to perform the motion, such division is not necessary. In fact, according to the embodiment of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided to be embodied by multiple modules or units.

[0185] Through the above description of the embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described herein may be realized through software, or may be realized through a method in which software is combined with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (may be CD-ROM, U disk, mobile disk, etc.) or network, and includes some instructions, so that a computing device (may be a personal computer, a server, a touch terminal, or a network device, etc.) executes the method according to the embodiments of the present application.

[0186] Those skilled in the art will be able to readily conceive of other implementations of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses, or adaptations of the present application, which modifications, uses, or adaptations follow the general principles of the present application and include common general knowledge or commonly used technical means in the art that are not disclosed herein.

[0187] It is to be understood that the present application is not limited to the exact construction already described above and illustrated in the drawings, and various modifications and changes can be made without departing from the scope thereof, which is limited only by the appended claims. [Explanation of symbols]

[0188] 101 Cloud side server 210 Application Client 211 Terminal Recognition Module 212 Vibration Feedback Control Module 213 Terminal Control Module 220 Cloud Application Server 221 Terminal communication module 222 Vibration Protocol Package Module 230 Cloud Application Client 231 Communication Module 232 Vibration Feedback Analysis Module 233 Hardware Control Module 310 Application Client 320 Cloud Application Server 321 Transparent Transmission Terminal 330 Cloud Application Client 800 Equipment Control Device 801 Communication Establishment Module 802 Scene Recognition Module 803 Vibration Feedback Information Generation Module 804 Vibration feedback information transmission module 810 Equipment control device 811 Communication Connection Establishment Module 812 Vibration feedback information receiving module 813 Playback Module 814 Vibration Feedback Execution Module 900 Electronic equipment 901 Central Processing Unit 903 Random Access Memory 904 Bus 905 I / O Interface 906 Input section 907 Output part 908 Memory part 909 Communication section 910 Driver 911 Medium

Claims

1. A device control method based on a cloud game, which is executed by an electronic device, comprising: sending a communication establishment request including a special effect inquiry command to a cloud gaming client for playing a game based on audio-video communication or control communication established with the cloud gaming client, the special effect inquiry command being used to inquire about a vibration feedback type and a vibration feedback protocol version supported by a physical device associated with the cloud gaming client, the special effect inquiry command being used by the cloud gaming client to present the received cloud gaming video stream; receiving a communication response message in response to the communication establishment request sent by the cloud gaming client, and establishing a communication connection with the cloud gaming client based on communication parameters of the cloud gaming client included in the communication response message, the communication response message further including a vibration feedback type and a vibration feedback protocol version supported by a physical device associated with the cloud gaming client, the vibration feedback type being used to indicate a function or module for performing vibration feedback included in the physical device, and the vibration feedback protocol version being used to define a specific mode of execution of vibration feedback; performing scene recognition on a cloud game corresponding to the cloud game client to obtain a scene recognition result; Generating vibration feedback information corresponding to a recognized cloud game scene according to the received vibration feedback type, the protocol version of the vibration feedback, and the scene recognition result; A cloud game-based device control method comprising: a step of transmitting a video stream of the cloud game scene to the cloud game client; and a step of transmitting the vibration feedback information to the cloud game client via the communication connection, wherein the vibration feedback information is combined with the video stream of the cloud game scene being presented when the cloud game client presents the video stream of the cloud game scene, and controlling an associated physical device to perform a vibration feedback operation based on the vibration feedback information corresponding to the cloud game scene.

2. The step of performing scene recognition on a cloud game corresponding to the cloud game client to obtain a scene recognition result includes: obtaining a character attribute value of a target virtual character in the cloud game at a current time; The method of claim 1 , further comprising: recognizing the cloud gaming scene based on the character attribute values ​​to obtain the scene recognition result.

3. The step of performing scene recognition on a cloud game corresponding to the cloud game client to obtain a scene recognition result includes: obtaining a scene identifier of a target virtual character at a current time according to a mapping relationship between a preset time period and a scene identifier, the scene identifier being used to represent an environment in which the target virtual character is located; The method of claim 1 , further comprising: determining the scene recognition result based on a mapping relationship between a preset scene identifier and a scene type, and the scene identifier at a current time.

4. The step of obtaining a character attribute value of a target virtual character in the cloud game at a current time includes: receiving a character control command for the target virtual character sent from the cloud gaming client; and performing control on the target virtual character based on the character control command to obtain a character attribute value of the target virtual character at the current time, The step of recognizing the cloud game scene based on the character attribute value to obtain the scene recognition result includes: determining an attribute value change range based on a character attribute value of the target virtual character at a previous time and a character attribute value of the target virtual character at the current time; The method of claim 2 , further comprising: determining the scene recognition result based on the attribute value variation range.

5. The method of claim 1 , wherein the vibration feedback information includes a time parameter for vibration feedback, the time parameter being used to indicate a time period during which vibration feedback is performed, and the cloud gaming client controls the physical device to perform a vibration feedback operation within the time period indicated by the time parameter.

6. The method of claim 1 , wherein the vibration feedback information includes a frequency of execution of vibration feedback, and the cloud gaming client controls the physical device to perform the vibration feedback operation according to the frequency of execution of the vibration feedback.

7. The method of claim 1 , wherein the vibration feedback information includes an execution strength of vibration feedback, and the cloud gaming client controls the physical device to perform the vibration feedback operation according to the execution strength of the vibration feedback.

8. The step of transmitting the video stream corresponding to the cloud gaming scene and the vibration feedback information to the cloud gaming client includes:

2. The method of claim 1, comprising sending a video stream corresponding to the cloud game scene and the vibration feedback information to a cloud game server, the cloud game server packaging the vibration feedback information based on a vibration protocol, and sending the packaging result and the video stream to the cloud game client.

9. A cloud gaming-based device control method executed by an electronic device, the method comprising: receiving a communication establishment request including a special effect inquiry command sent by a cloud game server for playing a game based on audio-video communication or control communication established with the cloud game server, the special effect inquiry command being used to inquire about a vibration feedback type and a vibration feedback protocol version supported by a physical device associated with the cloud game client; Inquiring about a vibration feedback type and a vibration feedback protocol version supported by a physical device associated with the cloud gaming client based on a special effect inquiry command in the communication establishment request; Including, The communication response message to the communication establishment request sent by the cloud game server includes communication parameters of the cloud game client, a vibration feedback type supported by a physical device associated with the cloud game client, and a protocol version of vibration feedback, the communication parameters being used by the cloud game server to establish a communication connection, the vibration feedback type being used to indicate a function or module for performing vibration feedback that is included in the physical device, and the protocol version of the vibration feedback being used to define a specific mode of execution of vibration feedback; The method further comprises: receiving a video stream of a cloud game scene and vibration feedback information transmitted by the cloud game server via the communication connection, the vibration feedback information being generated based on the vibration feedback type, the vibration feedback protocol version, and the cloud game scene received by the cloud game server and corresponding to the cloud game scene; A cloud game-based device control method comprising: a step of playing a video stream of the cloud game scene; and a step of controlling a physical device associated with the cloud game client to perform a vibration feedback operation based on vibration feedback information corresponding to the cloud game scene so as to combine it with the video stream of the cloud game scene being played.

10. the physical device includes a third-party device; The step of playing the video stream of the cloud gaming scene includes: generating vibration control commands for the third-party device based on the vibration feedback information; The method of claim 9 , further comprising: transmitting the vibration control command to the third party device, whereby the third party device performs a vibration feedback operation based on the vibration feedback information.

11. A device control device based on a cloud game, A communication establishment request sending module that sends a communication establishment request including a special effect inquiry command to a cloud gaming client for playing a game based on audio video communication or control communication established between the cloud gaming client and the cloud gaming client, the special effect inquiry command being used to inquire about a vibration feedback type and a vibration feedback protocol version supported by a physical device associated with the cloud gaming client, and the cloud gaming client is used to present the received cloud gaming video stream; a communication connection establishment module that receives a communication response message in response to the communication establishment request sent by the cloud game client, and establishes a communication connection with the cloud game client based on communication parameters of the cloud game client included in the communication response message, the communication response message further including a vibration feedback type and a vibration feedback protocol version supported by a physical device associated with the cloud game client, the vibration feedback type being used to indicate a function or module for performing vibration feedback included in the physical device, and the vibration feedback protocol version being used to define a specific mode of execution of the vibration feedback; A scene recognition module performs scene recognition on a cloud game corresponding to the cloud game client to obtain a scene recognition result; a vibration feedback information generating module for generating vibration feedback information corresponding to a recognized cloud game scene according to the received vibration feedback type, the vibration feedback protocol version, and the scene recognition result; a vibration feedback information transmission module that transmits a video stream of the cloud game scene to the cloud game client and transmits the vibration feedback information to the cloud game client via the communication connection, the vibration feedback information transmission module controlling an associated physical device to perform a vibration feedback operation based on the vibration feedback information corresponding to the cloud game scene so that the vibration feedback information is combined with the video stream of the cloud game scene being presented when the cloud game client presents the video stream of the cloud game scene; A cloud gaming-based device control device comprising:

12. The device of claim 11, wherein the vibration feedback information sending module is used to send a video stream corresponding to the cloud game scene and the vibration feedback information to a cloud game server, so that the cloud game server packages the vibration feedback information based on a vibration protocol, and sends the packaging result and the video stream to the cloud game client.

13. A device control device based on a cloud game, A communication establishment request receiving module receives a communication establishment request including a special effect inquiry command sent by a cloud game server based on audio / video communication or control communication established with the cloud game server for playing a game, the special effect inquiry command being used to inquire about a vibration feedback type and a vibration feedback protocol version supported by a physical device associated with the cloud game client; A special effect inquiry module that inquires about a vibration feedback type and a vibration feedback protocol version supported by a physical device associated with the cloud gaming client according to a special effect inquiry command in the communication establishment request; Including, The communication response message to the communication establishment request sent by the cloud game server includes communication parameters of the cloud game client, a vibration feedback type supported by a physical device associated with the cloud game client, and a protocol version of vibration feedback, the communication parameters being used by the cloud game server to establish a communication connection, the vibration feedback type being used to indicate a function or module for performing vibration feedback that is included in the physical device, and the protocol version of the vibration feedback being used to define a specific mode of execution of vibration feedback; The apparatus further comprises: a vibration feedback information receiving module that receives a video stream of a cloud game scene and vibration feedback information transmitted by the cloud game server via the communication connection, the vibration feedback information being generated based on the vibration feedback type, the vibration feedback protocol version, and the cloud game scene received by the cloud game server, and corresponding to the cloud game scene; a playback module for playing the video stream of the cloud gaming scene; A cloud game-based device control device comprising: a vibration feedback execution module that controls a physical device associated with the cloud game client to perform a vibration feedback operation based on vibration feedback information corresponding to the cloud game scene so as to be combined with a video stream of the cloud game scene being played.

14. The apparatus of claim 13 , wherein the physical device includes a third-party device, and the vibration feedback execution module is used to generate a vibration control command for the third-party device based on the vibration feedback information, and to send the vibration control command to the third-party device, so that the third-party device executes a vibration feedback operation based on the vibration feedback information.

15. An electronic device, A processor; a memory adapted to store executable instructions of said processor; The electronic device, wherein the processor is configured to execute the cloud gaming-based device control method described in any one of claims 1 to 8 via execution of the executable instructions.

16. An electronic device comprising: A processor; a memory adapted to store executable instructions of said processor; The electronic device, wherein the processor is configured to execute the cloud gaming-based device control method according to claim 9 or 10 via execution of the executable instructions.

17. A computer program that, when executed by a processor, realizes the device control method based on a cloud game according to any one of claims 1 to 8.

18. A computer program which, when executed by a processor, realizes the cloud gaming-based device control method described in claim 9 or 10.

Images