Terminal interaction method and device, equipment and storage medium

Abstract

The invention provides a terminal interaction method and device, equipment and a storage medium, and the terminal interaction method comprises the steps: obtaining an application operation state and a terminal load parameter of a target interaction application in response to a remote interaction request; according to the application running state and the terminal load parameter, calling a first camera to collect an initial interaction image of a target object according to a target collection mode, wherein the first camera is an external camera; determining a target interaction image corresponding to the initial interaction image based on a terminal hardware layer and a preset transmission link; and obtaining a gesture interaction instruction corresponding to the target interaction image, and driving the target interaction application to execute the gesture interaction instruction to obtain a gesture interaction result. Therefore, the terminal can perform gesture recognition on the target interaction image without decoding, terminal resources required by terminal gesture recognition are reduced, the gesture recognition efficiency and success rate are improved, the terminal operation stability is improved, and the terminal interaction lagging probability is reduced.

Description

Technical Field

[0001] The present application relates to the field of network distribution technology, and in particular to a terminal interaction method, device, equipment and storage medium. Background Art

[0002] At present, with the rapid development of smart terminals such as smart TVs and gesture recognition technology, more and more users use various applications running in smart terminals such as smart TVs for entertainment and other activities, and can also use gestures to interact with smart terminals when using these applications. However, existing gesture recognition methods require more system resources, and smart terminals such as smart TVs may have a large terminal load when running specified applications. For example, when a smart terminal plays a high-definition video, it needs to decode the high-definition video, and the terminal load is high. At this time, decoding and recognizing the gesture image may cause the system to freeze or the application process to become unresponsive, thereby affecting the accuracy of gesture recognition and the stability of application operation. Summary of the invention

[0003] The embodiments of the present application provide a terminal interaction method, apparatus, device and storage medium, aiming to solve the technical problem in the prior art that gesture recognition occupies a large amount of system resources and causes terminal freezes.

[0004] On the one hand, an embodiment of the present application provides a terminal interaction method, the terminal interaction method comprising the following steps:

[0005] Respond to the remote interaction request and obtain the application running status and terminal load parameters of the target interactive application;

[0006] Calling a first camera to collect an initial interactive image of a target object in a target collection mode according to the application running state and the terminal load parameter, wherein the first camera is an external camera;

[0007] Determine a target interactive image corresponding to the initial interactive image based on a terminal hardware layer and a preset transmission link;

[0008] Acquire a gesture interaction instruction corresponding to the target interaction image, drive the target interaction application to execute the gesture interaction instruction, and obtain a gesture interaction result.

[0009] In a possible implementation of the present application, before calling the first camera to collect the initial interactive image of the target object in the target collection mode according to the application running state and the terminal load parameter, it also includes:

[0010] Determine a target acquisition format parameter of the first camera based on the application running state and the terminal load parameter;

[0011] The target acquisition format parameter is transmitted to the first camera based on a preset transmission link, and the register of the first camera is configured according to the target acquisition format parameter to obtain the target acquisition mode of the first camera.

[0012] In a possible implementation of the present application, determining the target acquisition format parameter of the first camera based on the application running state and the terminal load parameter includes:

[0013] If the application running state is the first running state and / or the terminal load parameter is greater than or equal to a preset terminal load threshold, setting the preset first acquisition format parameter as the target acquisition format parameter of the first camera;

[0014] If the application running state is the second running state and the terminal load parameter is less than a preset terminal load threshold, the preset second acquisition format parameter is set as the target acquisition format parameter of the first camera.

[0015] In a possible implementation of the present application, determining the target interactive image corresponding to the initial interactive image based on the terminal hardware layer and the preset transmission link includes:

[0016] Transmitting the initial interaction image based on a preset transmission link, and acquiring image format information of the initial interaction image;

[0017] The target format information in the terminal hardware layer is used to match the image format information of the initial interaction image to obtain a target interaction image corresponding to the initial interaction image.

[0018] In a possible implementation of the present application, the using the target format information in the terminal hardware layer to match the image format information of the initial interaction image to obtain the target interaction image corresponding to the initial interaction image includes:

[0019] Obtaining a target image format, a target resolution, and a target frame rate in the target format information;

[0020] If the initial image format in the image format information is the same as the target image format, the initial image resolution is the same as the target resolution, and the initial image frame rate is the same as the target frame rate, then the initial interaction image is determined as the target interaction image;

[0021] If the initial image format in the image format information is the same as the target image format, and the initial image resolution and / or the initial image frame rate are different from the target resolution and / or the target frame rate in the target format information, then the initial interactive image is adjusted based on the target resolution and / or the target frame rate to obtain the target interactive image.

[0022] In a possible implementation of the present application, the acquiring the gesture interaction instruction corresponding to the target interaction image, driving the target interaction application to execute the gesture interaction instruction, and obtaining the gesture interaction result includes:

[0023] Extracting gesture features from the target interaction image using a preset gesture recognition model to obtain target gesture features;

[0024] Performing gesture recognition processing on the target gesture feature to obtain a gesture interaction instruction corresponding to the target gesture feature;

[0025] The target interactive application is driven to execute the gesture interaction instruction to obtain a gesture interaction result.

[0026] In a possible implementation of the present application, the step of driving the target interactive application to execute the gesture interaction instruction to obtain a gesture interaction result includes:

[0027] Transmitting the gesture interaction instruction to the target interactive application, and determining an application interactive object to be adjusted for the gesture interaction instruction;

[0028] Performing an interactive operation on the application interactive object according to the gesture interaction instruction to obtain a gesture interaction result.

[0029] On the other hand, the present application provides a terminal interaction device, the terminal interaction device comprising:

[0030] An information acquisition module is configured to respond to a remote interaction request and acquire an application running state and terminal load parameters of a target interactive application;

[0031] An image acquisition module is configured to call a first camera to acquire an initial interactive image of a target object in a target acquisition mode according to the application running state and the terminal load parameter, wherein the first camera is an external camera;

[0032] An image transmission module, configured to determine a target interactive image corresponding to the initial interactive image based on a terminal hardware layer and a preset transmission link;

[0033] The terminal interaction module is configured to obtain the gesture interaction instruction corresponding to the target interaction image, drive the target interaction application to execute the gesture interaction instruction, and obtain the gesture interaction result.

[0034] On the other hand, the present application also provides a terminal interaction device, the terminal interaction device comprising:

[0035] one or more processors;

[0036] Memory; and

[0037] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the steps of the terminal interaction method.

[0038] On the other hand, the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the steps in the terminal interaction method.

[0039] In this application, by responding to a remote interaction request, the application running state and terminal load parameters of the target interactive application are obtained; according to the application running state and the terminal load parameters, the first camera is called to collect the initial interactive image of the target object according to the target acquisition mode, and the first camera is an external camera; based on the terminal hardware layer and the preset transmission link, the target interactive image corresponding to the initial interactive image is determined; the gesture interaction instruction corresponding to the target interactive image is obtained, and the target interactive application is driven to execute the gesture interaction instruction to obtain the gesture interaction result. When the terminal runs an application with a high load, a pre-configured external camera is used to collect a specific specification interactive image carrying gesture interaction information such as user gestures, and the terminal hardware layer and the preset transmission link are used to transmit the target interactive image in the initial interactive image, so that the terminal can perform gesture recognition on the target interactive image without decoding, reducing the terminal resources required for terminal gesture recognition, improving gesture recognition efficiency and success rate, thereby improving terminal operation stability and reducing the probability of terminal freeze. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A schematic diagram of a scenario of a terminal interaction method according to an embodiment of the present application;

[0042] Figure 2 A schematic diagram of a flow chart of an embodiment of a terminal interaction method in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of a flow chart of an embodiment of matching a target interaction image in a terminal interaction method provided in an embodiment of the present application;

[0044] Figure 4 A flowchart of an embodiment of gesture recognition in a terminal interaction method provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of the structure of an embodiment of a terminal interaction device provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of the structure of an embodiment of a terminal interaction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0048] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0049] In this application, the word "exemplary" is used to mean "serving as an example, illustration, or illustration." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any person skilled in the art to implement and use the invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the invention can be implemented without using these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.

[0050] At present, with the rapid development of smart terminals such as smart TVs and gesture recognition technology, more and more users use various applications running in smart terminals such as smart TVs for entertainment and other activities, and can also use gestures to interact with smart terminals when using these applications. However, existing gesture recognition methods require more system resources, and smart terminals such as smart TVs may have a large terminal load when running specified applications. For example, when a smart terminal plays a high-definition video, it needs to decode the high-definition video, and the terminal load is high. At this time, decoding and recognizing the gesture image may cause the system to freeze or the application process to become unresponsive, thereby affecting the accuracy of gesture recognition and the stability of application operation.

[0051] Based on this, the present application proposes a terminal interaction method, device, equipment and computer-readable storage medium to solve the technical problem in the prior art that gesture recognition occupies a large amount of system resources and causes terminal freezes.

[0052] The terminal interaction method in the embodiment of the present invention is applied to a terminal interaction device, and the terminal interaction device is arranged in a terminal interaction device. The terminal interaction device is provided with one or more processors, a memory, and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the terminal interaction method; wherein the terminal interaction device can be an intelligent terminal, such as a mobile phone, a tablet computer, a network device, and a smart computer; optionally, the terminal interaction device can also be a server, or a service cluster composed of multiple servers.

[0053] like Figure 1 As shown, Figure 1 The following is a schematic diagram of a scenario of a terminal interaction method according to an embodiment of the present application. The terminal interaction scenario in the embodiment of the present invention includes a terminal interaction device 100 (a terminal interaction device is integrated in the terminal interaction device 100), and a computer-readable storage medium corresponding to the terminal interaction method is running in the terminal interaction device 100 to execute the steps of the terminal interaction method. It can be understood that Figure 1 The terminal interaction devices in the terminal interaction method scenario shown, or the devices included in the terminal interaction devices, do not constitute limitations on the embodiments of the present invention, that is, the number of devices and types of terminal interaction devices included in the terminal interaction method scenario, or the number of devices and types of devices included in each device do not affect the overall implementation of the technical solution in the embodiments of the present invention, and can all be regarded as equivalent replacements or derivatives of the technical solution claimed to be protected by the embodiments of the present invention.

[0054] In the embodiment of the present invention, the terminal interaction device 100 is mainly used for: responding to a remote interaction request, obtaining the application running status and terminal load parameters of a target interactive application; calling a first camera to capture an initial interactive image of a target object in a target acquisition mode according to the application running status and the terminal load parameters, wherein the first camera is an external camera; determining a target interactive image corresponding to the initial interactive image based on a terminal hardware layer and a preset transmission link; obtaining a gesture interaction instruction corresponding to the target interactive image, driving the target interactive application to execute the gesture interaction instruction, and obtaining a gesture interaction result.

[0055] The terminal interaction device 100 in the embodiment of the present invention can be an independent terminal interaction device, such as a smart terminal such as a mobile phone, a tablet computer, a network device, a smart TV, a server, and a smart computer, or it can be a terminal interaction network or a terminal interaction cluster composed of multiple terminal interaction devices.

[0056] The embodiments of the present application provide a terminal interaction method, apparatus, device, and computer-readable storage medium, which are described in detail below.

[0057] It can be understood by those skilled in the art that Figure 1 The application environment shown in is only one of the application scenarios related to the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or fewer terminal interaction devices, or terminal interaction network connection relationships shown, such as Figure 1 Only one terminal interaction device is shown, and it can be understood that the scenario of the terminal interaction method can also include one or more terminal interaction devices, which is not specifically limited here; the terminal interaction device 100 can also include a memory for storing application data and other data.

[0058] It should be noted that Figure 1 The scenario diagram of the terminal interaction method shown is only an example. The scenario of the terminal interaction method described in the embodiment of the present invention is to more clearly illustrate the technical solution of the embodiment of the present invention, and does not constitute a limitation on the technical solution provided by the embodiment of the present invention.

[0059] Based on the scenario of the above terminal interaction method, various embodiments of the terminal interaction method disclosed in the present invention are proposed.

[0060] like Figure 2 As shown, Figure 2 This is a flow chart of an embodiment of a terminal interaction method in an embodiment of the present application. The terminal interaction method includes the following steps 201 to 204:

[0061] 201. Respond to the remote interaction request and obtain the application running status and terminal load parameters of the target interactive application;

[0062] The terminal interaction method in this embodiment is applied to a terminal interaction device, and the type and number of the terminal interaction device are not specifically limited, that is, the terminal interaction device may be one or more intelligent terminals capable of performing gesture interactions such as gesture interactions. In a specific embodiment, the terminal interaction device is a smart TV with a gesture recognition module. For example, taking a smart TV as an example, when the terminal interaction device uses a player application to play a movie, it can respond to user gesture interactions to achieve functions such as playing videos, pausing videos, fast forwarding videos, and rewinding videos. Optionally, the terminal interaction device can also implement other operations based on gesture interactions such as gestures. For example, moving the focus of a smart TV through the air based on gesture recognition, etc., is not specifically limited in this embodiment.

[0063] Specifically, when the terminal interaction device is running, it can start the specified target interaction application to perform related operations, and respond to the remote interaction request during the operation of the target interaction application. Among them, the remote interaction request is an interaction event that drives the terminal interaction device to perform a specific air-to-air interaction operation. Among them, the triggering method of the remote interaction request is not specifically limited here, that is, the remote interaction request can be actively triggered by the user. For example, the user actively triggers the remote interaction request by clicking the terminal interaction button on the terminal interaction device and / or the control device corresponding to the terminal interaction device (such as a remote control or a mobile terminal connected to the terminal interaction device). For example, in a possible embodiment, before the user uses a gesture for air-to-air interaction, it is necessary to click the terminal interaction button corresponding to the terminal interaction device to trigger the gesture interaction process, and perform a specific gesture in the gesture interaction process to perform air-to-air interaction operations. In addition, the remote interaction request can also be automatically triggered by the terminal interaction device. For example, an automatic detection process is pre-set in the terminal interaction device, and the remote interaction request is automatically triggered when a specific interaction action of the user is detected.

[0064] Specifically, since the terminal interaction device responds to the gesture interaction request as an interaction event for the currently running target interaction application, and the target interaction application needs to occupy a higher terminal resource during operation and the terminal load is high, in order to avoid subsequent freezes in the target application or terminal system due to gesture recognition, the terminal interaction device also reads the application running status and terminal load parameters of the target interaction application currently running on the terminal interaction device after detecting the gesture interaction request.

[0065] Among them, the application running state of the target interactive application is a running state that characterizes the system resource occupancy state of the target interactive application. The application running state includes a first running state and a second running state. For example, in a specific embodiment, the target interactive application being run by the terminal interactive device is a player application. When the player application plays a high-definition video, it is necessary to decode and play the high-definition video, which causes the player application to occupy higher system resources when playing the high-definition video. At this time, the application running state of the player application is the first running state that characterizes a higher application load. When the player application is in a non-playing state such as home page display, at this time, the application state of the player application is the second running state that characterizes a lower application load.

[0066] The terminal load parameter is a percentage parameter that characterizes the system resource occupancy status (CPU occupancy status and memory occupancy status, etc.) in the terminal interaction device. The higher the terminal load parameter, the higher the system resources occupied by the currently running program in the corresponding terminal interaction device. Optionally, the terminal interaction device can obtain the terminal load parameter by monitoring the task manager or other application monitoring process.

[0067] 202. Calling a first camera to capture an initial interactive image of a target object in a target capture mode according to the application running state and the terminal load parameter;

[0068] Specifically, the terminal interaction device is also connected to the first camera through a preset transmission link. After responding to the gesture interaction request and determining the application running state of the target interactive application and the terminal load parameter of the terminal interaction device, the terminal interaction device also configures the target acquisition mode of the first camera based on the application running state and the terminal load parameter, and calls the first camera according to the application running state and the terminal load parameter to collect the initial interaction image of the target object according to the target acquisition mode. Among them, the first camera is an external camera connected to the terminal interaction device with the preset transmission link to shoot the specified area. The target object is the target user object to be gesture image collected. Optionally, the first camera can be set above the terminal interaction device or in front of the terminal interaction device, that is, the first camera can be assumed to be set at the top of the terminal interaction device or independently of the terminal interaction device in front of the terminal interaction device, and connected to the terminal interaction device through a preset transmission link. Optionally, in other embodiments, the first camera can also be set at other positions to improve the image acquisition effect, which is not specifically limited in this embodiment.

[0069] Specifically, since the existing gesture recognition application runs in the background, it needs to continuously read the image data (MJPEG format) of the camera for gesture analysis, and the existing gesture recognition application needs to decode the image data before performing gesture recognition during the gesture analysis process. Therefore, when the terminal interaction device runs the target interaction application in a high-load state, in order to avoid the target interaction application running in the foreground from being stuck due to the heavy gesture recognition load, the terminal interaction device determines the target acquisition mode of the first camera according to the application running state of the target interaction application running in the foreground and the terminal load parameters, and drives the first camera to acquire the initial interaction image of the target object based on the target acquisition mode.

[0070] Specifically, the terminal interaction device determines the target acquisition format parameters of the first camera based on the application running state and the terminal load parameters, that is, the terminal interaction device determines the current terminal load state of the terminal interaction device through the application running state and the terminal load parameters, and configures the corresponding target acquisition format parameters according to the terminal load state. Among them, the target acquisition format parameters are image format parameters, image resolution parameters, and image frame rate parameters corresponding to the initial interaction image captured by the first camera.

[0071] Optionally, in a specific embodiment, if the application running state of the target interactive application in the terminal interaction device is the first running state, and / or the terminal load parameter is greater than or equal to the preset terminal load threshold, the terminal interaction device determines that the current terminal load state is a high load state, and therefore, the terminal interaction device sets the preset first acquisition format parameter as the target acquisition format parameter of the first camera. Among them, the first acquisition format parameter is an image format parameter that is a YUV format parameter, and the image resolution parameter and the image frame rate parameter meet the acquisition format parameter required by the preset transmission link and gesture recognition application. In a specific embodiment, the first acquisition format parameter is a YUV 1280*720 resolution 15 frame parameter. That is, the first acquisition format parameter is a YUV image parameter for acquiring 1280*720 resolution 15 frames.

[0072] Optionally, in a specific embodiment, if the application running state of the target interactive application in the terminal interaction device is the second running state, and the terminal load parameter is less than the preset terminal load threshold, the terminal interaction device determines that the current terminal load state is a low load state, and the terminal interaction device sets the preset second acquisition format parameter as the target acquisition format parameter of the first camera. The second acquisition format parameter is an image format parameter that is an MJPEG format parameter, and the image resolution and image frame rate parameters meet the acquisition format parameters required by the preset transmission link and gesture recognition application. In a specific embodiment, the second acquisition format parameter is an MJPEG 1280*720 resolution 30 frame parameter, that is, the second acquisition format is an MJPEG image parameter that acquires 1280*720 resolution 30 frames.

[0073] Specifically, after determining the target acquisition format parameter of the first camera, the terminal interaction device also transmits the target acquisition format parameter to the first camera through a preset transmission link, and configures the register in the first camera according to the target acquisition format to obtain the target acquisition mode of the first camera. The preset transmission link may be a USB transmission link.

[0074] Specifically, the terminal interaction device queries the target register corresponding to each register in the first camera and the target acquisition format parameter, and configures the register value of the target register based on the target acquisition format parameter and the preset register configuration interface to obtain the target acquisition mode of the first camera. That is, after obtaining the target acquisition format parameter YUV 1280*720 resolution 15 frame parameter, the terminal interaction device determines the target register corresponding to the acquisition format, switches the target register value to YUV using the preset register configuration interface, and obtains the target register corresponding to the acquisition resolution and acquisition frame rate using the register configuration interface, switches the target register value to 1280*720 and 15 frames, to obtain the target acquisition mode corresponding to the target acquisition format parameter, thereby realizing customized processing of the first camera, so that the first camera can acquire interactive images that meet both the transmission requirements of the preset transmission link and the requirements of gesture recognition.

[0075] Specifically, the terminal interaction device determines that the current terminal load state of the terminal interaction device is a high load state according to the application running state and the terminal load parameter, sets the first acquisition format as the target acquisition format parameter, configures the target register in the first camera based on the target acquisition format parameter, generates the target acquisition mode, and then calls the first camera to acquire the initial interaction image of the target object according to the target acquisition mode. The target acquisition mode is to drive the first camera to acquire the initial interaction image of 15 frames with a YUV 1280*720 resolution.

[0076] 203. Determine a target interactive image corresponding to the initial interactive image based on a terminal hardware layer and a preset transmission link;

[0077] Specifically, after the terminal interaction device uses the first camera to capture the initial interaction image according to the target acquisition mode, it also transmits the initial interaction image to the terminal interaction device through a preset transmission link, and uses the terminal hardware layer in the terminal interaction device to match the initial interaction image to obtain the target interaction image.

[0078] Specifically, the terminal interaction device pre-adjusts the terminal hardware layer so that the terminal hardware layer can support the initial interaction image corresponding to the target acquisition format parameters, and uses the preset transmission link to transmit the initial interaction image to the terminal hardware layer, driving the terminal hardware layer to match the initial interaction image to obtain the target interaction image that satisfies subsequent gesture recognition.

[0079] Specifically, after receiving the initial interaction image transmitted by the first camera based on the preset transmission link, the terminal interaction device parses the initial interaction image, obtains the image format information of the initial interaction image, and uses the target format information in the terminal hardware layer and the image format information of the initial interaction image to match to obtain a target interaction image corresponding to the initial interaction image, so that in the subsequent gesture recognition process, the target interaction image can be directly used for gesture recognition without decoding the target interaction image, thereby reducing the gesture recognition resource occupancy.

[0080] 204. Acquire a gesture interaction instruction corresponding to the target interaction image, drive the target interaction application to execute the gesture interaction instruction, and obtain a gesture interaction result.

[0081] Specifically, after acquiring the target interaction image, the terminal interaction device inputs the target interaction image into the gesture recognition application, and directly performs gesture recognition operation on the target interaction image through the gesture recognition application, thereby obtaining the gesture interaction instruction corresponding to the target interaction image, and driving the target interaction application to execute the gesture interaction instruction to obtain the gesture interaction result.

[0082] Specifically, when the terminal load state is a high load state, the terminal interaction device drives the first camera to acquire the interactive image according to the first acquisition format parameter, so the target interactive image is a YUV image that can be directly gesture recognized without decoding. The terminal interaction device inputs the target interactive image into the gesture recognition application, and drives the gesture recognition application to perform gesture recognition processing on the target interactive image using a preset gesture recognition model to obtain a gesture interaction instruction corresponding to the target image.

[0083] Specifically, after obtaining the gesture interaction instruction corresponding to the target interactive image, the terminal interaction device distributes the gesture interaction instruction to the target interactive application, and determines the application interaction object with adjustment corresponding to the gesture interaction instruction. Among them, the application interaction object is the application object to be adjusted pointed to by the gesture interaction instruction. For example, in a specific embodiment, if the gesture interaction instruction is a play video instruction or a pause video instruction, the application interaction object corresponding to the gesture interaction instruction is a video play / pause control. In another specific embodiment, if the gesture interaction instruction is a fast forward or rewind instruction, the application interaction object corresponding to the gesture interaction instruction is a progress control control.

[0084] Specifically, after determining the target interaction object corresponding to the gesture interaction instruction, the terminal interaction device interacts with the application interaction object according to the gesture interaction instruction, drives the application interaction action to perform the corresponding interaction operation, and obtains the gesture interaction result. For example, if the gesture interaction instruction is a play video instruction, the terminal interaction device drives the video playback control to play the video data pointed to by the gesture interaction instruction. For another example, if the gesture interaction instruction is a pause video instruction, the terminal interaction device drives the video playback control to pause the video data pointed to by the gesture interaction instruction. For another example, if the gesture interaction instruction is a video fast-forward instruction, the terminal interaction device drives the progress control control to adjust the playback speed of the video data pointed to by the gesture interaction instruction.

[0085] In this embodiment, the terminal interaction device obtains the application running state and terminal load parameters of the target interactive application by responding to the remote interaction request; calls the first camera to collect the initial interactive image of the target object according to the target acquisition mode according to the application running state and the terminal load parameters, and the first camera is an external camera; determines the target interactive image corresponding to the initial interactive image based on the terminal hardware layer and the preset transmission link; obtains the gesture interaction instruction corresponding to the target interactive image, drives the target interactive application to execute the gesture interaction instruction, and obtains the gesture interaction result. When the terminal runs an application with a high load, a pre-configured external camera is used to collect a specific specification interactive image carrying gesture interaction information such as user gestures, and the terminal hardware layer and the preset transmission link are used to transmit the target interactive image in the initial interactive image, so that the terminal can perform gesture recognition on the target interactive image without decoding, thereby reducing the terminal resources required for terminal gesture recognition, improving gesture recognition efficiency and success rate, and thus improving terminal operation stability and reducing the probability of terminal freeze.

[0086] like Figure 3 As shown, Figure 3 This is a flow chart of an embodiment of matching a target interaction image in a terminal interaction method provided in an embodiment of the present application. Specifically, in this embodiment, the terminal interaction method further includes steps 301 to 303:

[0087] 301. Obtain a target image format, a target resolution, and a target frame rate in the target format information;

[0088] 302. If the initial image format in the image format information is the same as the target image format, the initial image resolution is the same as the target resolution, and the initial image frame rate is the same as the target frame rate, then determine the initial interaction image as the target interaction image;

[0089] 303. If the initial image format in the image format information is the same as the target image format, and the initial image resolution and / or the initial image frame rate are different from the target resolution and / or the target frame rate in the target format information, then the initial interactive image is adjusted based on the target resolution and / or the target frame rate to obtain a target interactive image.

[0090] Based on the above embodiment, in this embodiment, after receiving the initial interaction image transmitted by the first camera based on the preset transmission link, the terminal interaction device uses the terminal hardware layer in the terminal interaction device to match the initial interaction image to obtain a target interaction graph. Among them, the target interaction image is a YUV image containing gesture information of the target object that can directly perform gesture recognition without decoding. In a specific embodiment, the target interaction image is an interaction image with a YUV 1280*720 resolution and 15 frames. Among them, the terminal hardware layer is the Camera Ha l layer of the terminal interaction device.

[0091] Specifically, after obtaining the target format information of the terminal hardware layer, the terminal interaction device parses the target format information to obtain the target image format, target resolution and target frame rate in the target format information, and matches the target image format, target resolution and target frame rate with the image format information in the initial interaction image, thereby determining the target interaction image corresponding to the initial interaction image.

[0092] Optionally, if the initial image format in the image format information of the initial interaction image is the same as the target image format, and the initial image resolution is the same as the target resolution, and the initial image frame rate is the same as the target image frame rate, the terminal interaction device determines that the initial interaction image conforms to the target format information preset at the terminal hardware layer, and then determines the initial interaction image as the target interaction image to be gesture recognized.

[0093] Optionally, if the initial image format in the image format information of the initial interaction image is the same as the target image format, but the initial resolution and / or initial image frame are different from the target resolution and / or target frame rate in the target format information, that is, the terminal interaction device determines that the image format of the initial interaction image matches, but the resolution or frame rate does not match the gesture recognition application requirements, then the terminal interaction device determines that the resolution and / or frame rate of the initial interaction image needs to be adjusted to obtain the target interaction image.

[0094] Optionally, if the initial interaction image has the same format as the target image, but the target resolution in the initial resolution target format information is different, the terminal interaction device uses a preset transmission link to transmit the initial interaction image back to the first camera, drives the first camera to adjust the initial resolution of the initial interaction image according to the target resolution, and obtains the target interaction image.

[0095] If the initial interaction image has the same format as the target image, but the target frame rate in the initial frame rate target format information is different, the terminal interaction device transmits the initial interaction image back to the first camera using a preset transmission link, drives the first camera to adjust the initial frame rate of the initial interaction image according to the target frame rate, and obtains the target interaction image.

[0096] Optionally, in order to improve gesture recognition efficiency, when the terminal interaction device determines that the image format of the initial interaction image matches, but the resolution or frame rate does not match the gesture recognition application requirements, the initial interaction image can be adjusted in the terminal interaction device to obtain a target interaction image that conforms to the target format information.

[0097] Specifically, after acquiring the target interaction image, the terminal interaction device inputs the target interaction image into the gesture recognition application, and directly performs gesture recognition operation on the target interaction image through the gesture recognition application, thereby obtaining the gesture interaction instruction corresponding to the target interaction image, and driving the target interaction application to execute the gesture interaction instruction to obtain the gesture interaction result.

[0098] In this embodiment, the terminal interaction device obtains the target image format, target resolution and target frame rate in the target format information; if the initial image format in the image format information is the same as the target image format, the initial image resolution is the same as the target resolution, and the initial image frame rate is the same as the target frame rate, then the initial interaction image is determined as the target interaction image; if the initial image format in the image format information is the same as the target image format, and the initial image resolution and / or the initial image frame rate are different from the target resolution and / or the target frame rate in the target format information, then the initial interaction image is adjusted based on the target resolution and / or the target frame rate to obtain the target interaction image. The initial interaction image is matched using the terminal hardware abstraction layer to ensure that the interaction image meets the subsequent gesture recognition requirements, thereby improving the accuracy of gesture recognition.

[0099] like Figure 4 As shown, Figure 4 This is a flow chart of an embodiment of gesture recognition in a terminal interaction method provided in an embodiment of the present application. Specifically, in this embodiment, the terminal interaction method further includes steps 401 to 403:

[0100] 401. Extracting gesture features from the target interaction image using a preset gesture recognition model to obtain target gesture features;

[0101] 402. Perform gesture recognition processing on the target gesture feature to obtain a gesture interaction instruction corresponding to the target gesture feature;

[0102] 403. Drive the target interactive application to execute the gesture interaction instruction to obtain a gesture interaction result.

[0103] Based on the above embodiments, in this embodiment, after acquiring the target interaction image, the terminal interaction device inputs the target interaction image into a gesture recognition application, and directly performs a gesture recognition operation on the target interaction image through the gesture recognition application, thereby acquiring the gesture interaction instruction corresponding to the target interaction image, and driving the target interaction application to execute the gesture interaction instruction to obtain a gesture interaction result.

[0104] Specifically, after inputting the target interaction image into the gesture recognition application, the terminal interaction device calls the gesture recognition model in the gesture recognition application to extract gesture features from the target interaction image to obtain the target gesture features. That is, the terminal interaction device uses the gesture recognition module to perform hand edge detection on the target interaction image, and after detecting the hand edge features of the target object, extracts the target gesture features of the target interaction image based on the hand edge features.

[0105] Specifically, after acquiring the target gesture feature, the terminal interaction device performs gesture recognition processing on the target gesture feature to obtain the gesture interaction instruction corresponding to the gesture feature. That is, the terminal interaction device matches the acquired target gesture feature with the candidate gesture feature pre-stored in the gesture recognition database to obtain the gesture similarity, and sets the interaction instruction corresponding to the candidate gesture feature whose gesture similarity is greater than the preset similarity threshold as the gesture interaction instruction corresponding to the target gesture feature.

[0106] Specifically, after acquiring the gesture interaction instruction, the terminal interaction device distributes the gesture interaction instruction to the target interaction application, drives the target interaction application to execute the gesture interaction instruction, and obtains the gesture interaction result.

[0107] In this embodiment, the terminal interaction device extracts gesture features from the target interaction image by using a preset gesture recognition model to obtain target gesture features; performs gesture recognition processing on the target gesture features to obtain gesture interaction instructions corresponding to the target gesture features; and drives the target interaction application to execute the gesture interaction instructions to obtain gesture interaction results. The gesture recognition model is used to perform gesture recognition on the target interaction image in the gesture recognition application to improve gesture recognition efficiency and reduce gesture recognition load occupancy.

[0108] In order to better implement the terminal interaction method in the embodiment of the present application, based on the terminal interaction method, the embodiment of the present application also provides a terminal interaction device, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of a terminal interaction device provided in an embodiment of the present application. Specifically, the terminal interaction device 500 includes:

[0109] The information acquisition module 501 is configured to respond to the remote interaction request and acquire the application running state and terminal load parameters of the target interactive application;

[0110] The image acquisition module 502 is configured to call a first camera to acquire an initial interaction image of a target object in a target acquisition mode according to the application running state and the terminal load parameter, wherein the first camera is an external camera;

[0111] An image transmission module 503 is configured to determine a target interaction image corresponding to the initial interaction image based on a terminal hardware layer and a preset transmission link;

[0112] The terminal interaction module 504 is configured to obtain the gesture interaction instruction corresponding to the target interaction image, drive the target interaction application to execute the gesture interaction instruction, and obtain the gesture interaction result.

[0113] In a possible implementation of this embodiment, before the terminal interaction device calls the first camera to capture the initial interaction image of the target object in the target acquisition mode according to the application running state and the terminal load parameter, it also includes:

[0114] Determine a target acquisition format parameter of the first camera based on the application running state and the terminal load parameter;

[0115] The target acquisition format parameter is transmitted to the first camera based on a preset transmission link, and the register of the first camera is configured according to the target acquisition format parameter to obtain the target acquisition mode of the first camera.

[0116] In a possible implementation of this embodiment, the terminal interaction device determines the target acquisition format parameter of the first camera based on the application running state and the terminal load parameter, including:

[0117] If the application running state is the first running state and / or the terminal load parameter is greater than or equal to a preset terminal load threshold, setting the preset first acquisition format parameter as the target acquisition format parameter of the first camera;

[0118] If the application running state is the second running state and the terminal load parameter is less than a preset terminal load threshold, the preset second acquisition format parameter is set as the target acquisition format parameter of the first camera.

[0119] In a possible implementation of this embodiment, the terminal interaction device determines the target interaction image corresponding to the initial interaction image based on the terminal hardware layer and the preset transmission link, including:

[0120] Acquire image format information of the initial interactive image;

[0121] Using the target format information in the terminal hardware layer to match the image format of the initial interactive image, a target interactive image to be transmitted is obtained;

[0122] The target interaction image is transmitted to the gesture recognition model based on a preset transmission link.

[0123] In a possible implementation of this embodiment, the terminal interaction device uses the target format information in the terminal hardware layer to match the image format of the initial interaction image to obtain the target interaction image to be transmitted, including:

[0124] Obtaining a target image format, a target resolution, and a target frame rate in the target format information;

[0125] If the initial image format in the image format information is the same as the target image format, the initial image resolution is the same as the target resolution, and the initial image frame rate is the same as the target frame rate, then the initial interactive image is determined as the target interactive image;

[0126] If the initial image format in the image format information is the same as the target image format, and the initial image resolution and / or the initial image frame rate are different from the target resolution and / or the target frame rate in the target format information, then the initial interactive image is adjusted based on the target resolution and / or the target frame rate to obtain the target interactive image to be transmitted.

[0127] In a possible implementation of this embodiment, the terminal interaction device obtains the gesture interaction instruction corresponding to the target interaction image, drives the target interaction application to execute the gesture interaction instruction, and obtains the gesture interaction result, including:

[0128] Extracting gesture features from the target interaction image using a preset gesture recognition model to obtain target gesture features;

[0129] Performing gesture recognition processing on the target gesture feature to obtain a gesture interaction instruction corresponding to the target gesture feature;

[0130] The target interactive application is driven to execute the gesture interaction instruction to obtain a gesture interaction result.

[0131] In a possible implementation of this embodiment, the terminal interaction device drives the target interaction application to execute the gesture interaction instruction to obtain a gesture interaction result, including:

[0132] Transmitting the gesture interaction instruction to the target interactive application, and determining an application interactive object to be adjusted for the gesture interaction instruction;

[0133] Performing an interactive operation on the application interactive object according to the gesture interaction instruction to obtain a gesture interaction result.

[0134] In this embodiment, the terminal interaction device obtains the application running state and terminal load parameters of the target interactive application by responding to the remote interaction request; calls the first camera to collect the initial interactive image of the target object according to the target acquisition mode according to the application running state and the terminal load parameters, and the first camera is an external camera; determines the target interactive image corresponding to the initial interactive image based on the terminal hardware layer and the preset transmission link; obtains the gesture interaction instruction corresponding to the target interactive image, drives the target interactive application to execute the gesture interaction instruction, and obtains the gesture interaction result. When the terminal runs an application with a high load, a pre-configured external camera is used to collect a specific specification interactive image carrying gesture interaction information such as user gestures, and the terminal hardware layer and the preset transmission link are used to transmit the target interactive image in the initial interactive image, so that the terminal can perform gesture recognition on the target interactive image without decoding, thereby reducing the terminal resources required for terminal gesture recognition, improving gesture recognition efficiency and success rate, and thus improving terminal operation stability and reducing the probability of terminal freeze.

[0135] The embodiment of the present invention also provides a terminal interaction device, such as Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of an embodiment of a terminal interaction device provided in an embodiment of the present application.

[0136] The terminal interaction device integrates any one of the terminal interaction devices provided in the embodiments of the present invention, and the terminal interaction device includes:

[0137] one or more processors;

[0138] Memory; and

[0139] One or more applications, wherein the one or more applications are stored in the memory and are configured so that the processor executes the steps of the terminal interaction method described in any of the above terminal interaction method embodiments.

[0140] Specifically, the terminal interaction device may include one or more processing core processors 601, one or more computer-readable storage media memories 602, a power supply 603, an input unit 604 and other components. Those skilled in the art will understand that Figure 6 The terminal interaction device structure shown in the figure does not constitute a limitation on the terminal interaction device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0141] The processor 601 is the control center of the terminal interaction device. It uses various interfaces and lines to connect various parts of the entire terminal interaction device. By running or executing software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, it executes various functions of the terminal interaction device and processes data, thereby monitoring the terminal interaction device as a whole. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601.

[0142] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal interaction device, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0143] The terminal interaction device also includes a power supply 603 for supplying power to each component. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 603 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0144] The terminal interaction device may further include an input unit 604, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0145] Although not shown, the terminal interaction device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 601 in the terminal interaction device will load the executable files corresponding to the processes of one or more application programs into the memory 602 according to the following instructions, and the processor 601 will run the application programs stored in the memory 602, thereby realizing various functions, as follows:

[0146] Respond to the remote interaction request and obtain the application running status and terminal load parameters of the target interactive application;

[0147] Calling a first camera to collect an initial interactive image of a target object in a target collection mode according to the application running state and the terminal load parameter, wherein the first camera is an external camera;

[0148] Determine a target interactive image corresponding to the initial interactive image based on a terminal hardware layer and a preset transmission link;

[0149] Acquire a gesture interaction instruction corresponding to the target interaction image, drive the target interaction application to execute the gesture interaction instruction, and obtain a gesture interaction result.

[0150] To this end, an embodiment of the present invention provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any terminal interaction method provided in an embodiment of the present invention. For example, the computer program can be loaded by a processor to execute the following steps:

[0151] Respond to the remote interaction request and obtain the application running status and terminal load parameters of the target interactive application;

[0152] Calling a first camera to collect an initial interactive image of a target object in a target collection mode according to the application running state and the terminal load parameter, wherein the first camera is an external camera;

[0153] Determine a target interactive image corresponding to the initial interactive image based on a terminal hardware layer and a preset transmission link;

[0154] Acquire a gesture interaction instruction corresponding to the target interaction image, drive the target interaction application to execute the gesture interaction instruction, and obtain a gesture interaction result.

[0155] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0156] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0157] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0158] The above is a detailed introduction to a terminal interaction method provided in an embodiment of the present application. Specific embodiments are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A terminal interaction method, characterized in that: The terminal interaction method comprises: Respond to the remote interaction request and obtain the application running status and terminal load parameters of the target interactive application; Calling a first camera to collect an initial interactive image of a target object in a target collection mode according to the application running state and the terminal load parameter, wherein the first camera is an external camera; Determine a target interactive image corresponding to the initial interactive image based on a terminal hardware layer and a preset transmission link; Acquire a gesture interaction instruction corresponding to the target interaction image, drive the target interaction application to execute the gesture interaction instruction, and obtain a gesture interaction result.

2. The terminal interaction method according to claim 1, characterized in that: Before calling the first camera to collect the initial interactive image of the target object in the target collection mode according to the application running state and the terminal load parameter, the method further includes: Determine a target acquisition format parameter of the first camera based on the application running state and the terminal load parameter; The target acquisition format parameter is transmitted to the first camera based on a preset transmission link, and the register of the first camera is configured according to the target acquisition format parameter to obtain the target acquisition mode of the first camera.

3. The terminal interaction method according to claim 2, characterized in that: The determining the target acquisition format parameter of the first camera based on the application running state and the terminal load parameter includes: If the application running state is the first running state and / or the terminal load parameter is greater than or equal to a preset terminal load threshold, setting the preset first acquisition format parameter as the target acquisition format parameter of the first camera; If the application running state is the second running state and the terminal load parameter is less than a preset terminal load threshold, the preset second acquisition format parameter is set as the target acquisition format parameter of the first camera.

4. The terminal interaction method according to claim 1, characterized in that: The determining the target interactive image corresponding to the initial interactive image based on the terminal hardware layer and the preset transmission link includes: Transmitting the initial interaction image based on a preset transmission link, and acquiring image format information of the initial interaction image; The target format information in the terminal hardware layer is used to match the image format information of the initial interaction image to obtain a target interaction image corresponding to the initial interaction image.

5. The terminal interaction method according to claim 4, characterized in that: The using the target format information in the terminal hardware layer to match the image format information of the initial interactive image to obtain the target interactive image corresponding to the initial interactive image includes: Obtaining a target image format, a target resolution, and a target frame rate in the target format information; If the initial image format in the image format information is the same as the target image format, the initial image resolution is the same as the target resolution, and the initial image frame rate is the same as the target frame rate, then the initial interaction image is determined as the target interaction image; If the initial image format in the image format information is the same as the target image format, and the initial image resolution and / or the initial image frame rate are different from the target resolution and / or the target frame rate in the target format information, then the initial interactive image is adjusted based on the target resolution and / or the target frame rate to obtain the target interactive image.

6. The terminal interaction method according to any one of claims 1 to 5, characterized in that: The acquiring the gesture interaction instruction corresponding to the target interaction image, driving the target interaction application to execute the gesture interaction instruction, and obtaining the gesture interaction result includes: Extracting gesture features from the target interaction image using a preset gesture recognition model to obtain target gesture features; Performing gesture recognition processing on the target gesture feature to obtain a gesture interaction instruction corresponding to the target gesture feature; The target interactive application is driven to execute the gesture interaction instruction to obtain a gesture interaction result.

7. The terminal interaction method according to claim 6, characterized in that: The step of driving the target interactive application to execute the gesture interaction instruction and obtaining the gesture interaction result includes: Transmitting the gesture interaction instruction to the target interactive application, and determining an application interactive object to be adjusted for the gesture interaction instruction; Performing an interactive operation on the application interactive object according to the gesture interaction instruction to obtain a gesture interaction result.

8. A terminal interaction device, characterized in that: The terminal interaction device comprises: An information acquisition module is configured to respond to a remote interaction request and acquire an application running state and terminal load parameters of a target interactive application; An image acquisition module is configured to call a first camera to acquire an initial interactive image of a target object in a target acquisition mode according to the application running state and the terminal load parameter, wherein the first camera is an external camera; An image transmission module, configured to determine a target interactive image corresponding to the initial interactive image based on a terminal hardware layer and a preset transmission link; The terminal interaction module is configured to obtain the gesture interaction instruction corresponding to the target interaction image, drive the target interaction application to execute the gesture interaction instruction, and obtain the gesture interaction result.

9. A terminal interaction device, characterized in that: The terminal interaction device comprises: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the processor to implement the steps of the terminal interaction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps of the terminal interaction method according to any one of claims 1 to 7.

Images