Control method based on multidirectional input device, multidirectional input device, control handle and information processing system

Abstract

The invention relates to a control method based on a multidirectional input device, the multidirectional input device, a control handle and an information processing system. The control method comprises the following steps: receiving a control instruction, wherein the control instruction comprises at least one of an operation instruction obtained by responding to mechanical operation aiming at the multi-direction input device and a voice instruction acquired by the multi-direction input device; and generating a control signal according to the control instruction and sending the control signal to the terminal device, wherein the control signal is used for controlling the terminal device to execute the target operation. Wherein the target operation comprises at least one of the following operations: virtual operation in a virtual scene displayed through a graphical user interface provided by the terminal equipment. And performing device control operation on the terminal equipment. And the application function operation is realized through the terminal equipment. Obviously, the input of various complex operation instructions can be conveniently realized, the input efficiency of the user is effectively improved, and the use experience of the user is guaranteed.

Description

Technical Field

[0001] This application relates to the technical field of control of multi-direction input devices, and in particular, to a control method based on a multi-direction input device, a multi-direction input device, a control handle, and an information processing system. Background Art

[0002] A multi-direction input device is a human-computer interaction device that can recognize multi-dimensional operation inputs of users (such as pushing, pulling, rotating, pressing, etc.), and is widely used in fields such as industrial machinery control, virtual reality interaction, and intelligent home appliance control. However, in related technologies, it is often necessary for users to manually operate the multi-direction input device, and one action input can only correspond to one action signal output; when users are unable to perform manual operations or need to output multiple complex operations simultaneously, there is a situation where users cannot simultaneously implement multiple operations, which cannot meet the usage needs of users. At the same time, there are situations where users' manual operations are not timely and / or not in place, making it difficult to guarantee the user experience. Summary of the Invention

[0003] In view of this, this application provides a control method based on a multi-direction input device, a multi-direction input device, a control handle, and an information processing system, which can effectively improve the input efficiency of users and guarantee the user experience.

[0004] In a first aspect, an embodiment of this application provides a control method based on a multi-direction input device. The multi-direction input device is communicatively connected to a terminal device. The control method includes: receiving a control instruction, where the control instruction includes at least one of an operation instruction obtained in response to a mechanical operation on the multi-direction input device and a voice instruction collected by the multi-direction input device. Generating a control signal according to the control instruction and sending the control signal to the terminal device, where the control signal is used to control the terminal device to perform a target operation. The target operation includes at least one of the following operations: virtual operations in a virtual scene displayed through a graphical user interface provided by the terminal device. Device control operations on the terminal device. Application function operations implemented through the terminal device.

[0005] In a possible design, when the control instruction includes an operation instruction and a voice instruction, the multi-direction input device generates a control signal according to the control instruction, including: parsing the voice instruction to obtain a corresponding instruction text. When the operation instruction conforms to a first operation instruction combination and the instruction text includes a first text, generating a first control signal, where the first control signal is used to control the terminal device to perform a target operation. When the control instruction meets a preset condition and there is a conflict between the voice instruction and the operation instruction, according to a preset conflict resolution strategy, preferentially execute one of the voice instruction or the operation instruction, and the preset condition includes: the voice instruction does not conform to the first operation instruction combination, and / or the instruction text does not include the first text.

[0006] In a possible design, when the control instruction includes a voice instruction, the multi-direction input device generates a control signal according to the control instruction, including: parsing the voice instruction to obtain the corresponding instruction text. When the instruction text includes at least one of a second text or a reservation time, a second control signal is generated according to the instruction text, and the second control signal is used to control the terminal device to perform a target operation after the reservation time.

[0007] In a possible design, when the control instruction includes a voice instruction, the multi-direction input device generates a control signal according to the control instruction, including: parsing the voice instruction to obtain the corresponding instruction text. When receiving status feedback information fed back by the terminal device and the instruction text includes a third text, a third control signal is generated according to the instruction text, and the third control signal is used to control the terminal device to perform a target operation, where the status feedback information is used to indicate that the terminal device is in a preset state.

[0008] In a possible design, the method further includes: obtaining scenario feedback information of the terminal device. Outputting first voice information according to the scenario feedback information. Wherein, the scenario feedback information includes: user status information collected by the terminal device; or the frame rate of the video content displayed by the terminal device; or scenario information collected by the terminal device.

[0009] In a possible design, the method further includes: obtaining historical operation trajectory data and real-time operation trajectory data, the historical operation trajectory data includes control instructions and corresponding target operations that occurred within a first preset duration, the real-time operation trajectory data includes control instructions and corresponding target operations that occurred within a second preset duration, the first preset duration occurred before the current moment, and the second preset duration includes the current moment. Comparing the historical operation trajectory data with the real-time operation trajectory data to obtain a comparison result. Outputting second voice information according to the comparison result.

[0010] In a possible design, the method further includes: when the multi-direction input device is in a custom state, obtaining and saving custom voice instructions, the custom voice instructions are associated with the preset operation instructions, and the custom voice instructions are used to perform a target operation corresponding to the preset operation instructions.

[0011] In a second aspect, an embodiment of the present application provides a multi-direction input device, including an operating member and a voice module, the operating member is used to generate an operation instruction in response to a user's mechanical operation, and the voice module is used to collect a voice instruction and execute the control method based on the multi-direction input device in any one of the above embodiments.

[0012] In a third aspect, an embodiment of the present application provides a control handle, and the control handle includes the multi-direction input device in the above embodiment.

[0013] Fourthly, an embodiment of the present application provides an information processing system, which includes a terminal device and the multi-direction input device in the above embodiment, and the multi-direction input device is communicatively connected to the terminal device.

[0014] The technical solution of the present application has at least the following technical effects or advantages: A control method based on a multi-direction input device provided by the present application generates a control signal according to a control instruction received, which includes at least one of an operation instruction obtained in response to a mechanical operation on the multi-direction input device and a collected voice instruction, and sends the control signal to a terminal device. The terminal device executes a target operation according to the received control signal. In this way, on the one hand, the control method based on the multi-direction input device provided by the present application can not only receive the operation instruction input by the user through mechanical operation, but also use voice for input, which is convenient for the user to perform non-contact input, improves the convenience of user input, and improves the input efficiency of the user. On the other hand, the control method based on the multi-direction input device provided by the present application can satisfy the simultaneous input of instructions for various complex operations of the user, improving the user experience. Description of the Drawings

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0016] Figure 1 is a schematic structural diagram of a multi-direction input device provided by an embodiment of the present application.

[0017] Figure 2 is a schematic X-axis interface diagram of a multi-direction input device provided by an embodiment of the present application.

[0018] Figure 3 is a schematic Y-axis interface diagram of a multi-direction input device provided by an embodiment of the present application.

[0019] Figure 4 is a schematic module diagram of an information processing system provided by an embodiment of the present application.

[0020] Figure 5 is a schematic step diagram of a control method provided by an embodiment of the present application. Detailed Embodiments

[0021] The following will describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments.

[0022] The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular forms "a", "an", "the above", "the", and "this" are also intended to include expressions such as "one or more", unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of the present application, "at least one" and "one or more" mean one, two, or more than two. The term "and / or" is used to describe the association relationship of associated objects and indicates that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0023] Reference to "one embodiment" or "some embodiments" etc. described in this specification means that a specific feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments" etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0024] A multi-directional input device is a human-computer interaction device that can recognize a user's multi-dimensional operation input (such as pushing, pulling, rotating, pressing, etc.), and is widely used in industrial machinery control, virtual reality interaction, intelligent home appliance control and other fields. However, in related technologies, it is often necessary for the user to manually operate the multi-directional input device, and one action input can only correspond to one action signal output; when the user is unable to perform manual operations or needs to output multiple complex operations simultaneously, there is a situation where the user cannot perform multiple operations simultaneously, which cannot meet the user's usage requirements. At the same time, there are situations where the user's manual operation is untimely and / or not in place, making it difficult to guarantee the user's usage experience.

[0025] Based on this, the present application provides a control method, a multi-directional input device, a control handle, and an information processing system based on the multi-directional input device, which can effectively improve the user's input efficiency and guarantee the user's usage experience.

[0026] Next, the control method, the multi-directional input device, the control handle, and the information processing system based on the multi-directional input device provided by the embodiments of the present application will be further introduced.

[0027] First, please refer to Figure 1 , Figure 1 , which is a schematic diagram of a multi-directional input device 10 provided in an embodiment of the present application. Exemplarily, as Figure 1 shown, the multi-directional input device 10 includes a housing 101 and an operating member 102. Among them, a cavity 103 is provided inside the housing 101, and a first opening 104 communicating with the cavity 103 is further formed on the housing 101. The operating member 102 includes a button portion 1021 and a main body portion 1022. The main body portion 1022 is at least partially rotatably disposed in the cavity 103 through the first opening 104, and the button portion 1021 is disposed at one end of the main body portion 1022 away from the cavity 103. The user operates by contacting the button portion 1021, and the button portion 1021 drives the main body portion 1022 to realize the operation control in the X-axis, Y-axis, and Z-axis directions.

[0028] In an embodiment of the present application, as Figure 2 and Figure 3 shown, the multi-directional input device 10 further includes a circuit board 105. The circuit board 105 is disposed in the cavity 103, and a plurality of sensors are disposed on the circuit board 105, such as an X-axis sensor 1051, a Y-axis sensor 1052, etc. The present application does not limit the specific number of sensors disposed on the circuit board 105, and those skilled in the art can select according to the actual situation of the multi-directional input device 10. In some embodiments, a processor 1001 is further disposed on the circuit board 105 for analyzing and processing the collected data. The processor 1001 is integrated with a voice module, and the voice module has functions such as voice collection, natural language processing, and dialogue management.

[0029] The multi-directional input device 10 of the present application can be rocked and pressed through a joystick to realize flexible operations in any horizontal direction and the Z-axis direction, such as a game console handle, a drone control handle, or a mobile robot control handle, etc. The joystick is usually single-axis, double-axis, three-axis, and four-axis, etc. Among them, a single-axis joystick means that it can move in a linear direction (i.e., horizontal or vertical) between two points; a double-axis joystick means that it can move in two linear directions (i.e., horizontal and vertical) at four points; a three-axis joystick means that it can move in a direction perpendicular to the horizontal and vertical planes outside the double-axis, which are represented by the X, Y, and Z axes respectively; a four-axis joystick means that a rotation axis is added on the basis of the three-axis. The present application does not limit the specific style and specific manipulation object of the multi-directional input device.

[0030] Please continue to refer to Figure 4, in some other embodiments, the voice module 106 can also be separately and independently provided from the processor 1001, and the present application does not limit this. Understandably, in some embodiments, the multi-direction input device 10 further includes a memory 1002, and the memory 1002 is used to store computer programs and data. The multi-direction input device 10 also establishes a communication connection with the terminal device 20. Among them, the communication connection method can be a wired communication method, such as wired communication based on the RS485 bus; it can also be a wireless communication method, such as wireless communication methods such as Bluetooth, Wi-Fi, and ZigBee. The present application does not limit the specific method of establishing the communication connection. In this way, by operating the multi-direction input device 10, the user can drive the terminal device 20 communicatively connected to the multi-direction input device 10 to execute according to the user's operation instructions, thereby improving the operation efficiency of the terminal device 20.

[0031] Understandably, in some embodiments, the multi-direction input device 10 and the terminal device 20 can be two separately and independently provided electronic devices. In some other embodiments, the multi-direction input device 10 can also be directly integrated on the terminal device 20. For example, the terminal device 20 can be a handheld game console. In this case, the processor 1001 of the multi-direction input device 10 can be integrated with the processor of the terminal device 20 ( Figure 4 not shown) on the same processor.

[0032] Please continue to refer to Figure 5 , Figure 5 shows a schematic diagram of the steps of a control method based on a multi-direction input device provided by an embodiment of the present application. Understandably, the control method provided by the present application can be executed by the processor of the multi-direction input device, such as Figure 3 the processor 1001 in Figure 5 . As shown in

[0033] , the method includes the following steps S1 - S2.

[0034] Exemplarily, the control instruction is used to represent the relevant instruction signals received by the multi-direction input device 10 during operation. The control instruction can be collected by a plurality of set sensors. For example, as shown in Figure 2 and Figure 3 , the control instruction can include an operation instruction received by the X-axis sensor 1051 to move along the X-axis direction (i.e., horizontally), an operation instruction received by the Y-axis sensor 1052 to move along the Y-axis direction (i.e., vertically), and an operation instruction for the operating member 102 along the Z-axis direction, etc. Among them, the operation instruction can be an electrical signal based on a sensor or a button.

[0035] For another example, the control instruction may further include a voice instruction received through the voice module on the multi-direction input device 10. For example, the voice module may continuously monitor the sound in the environment where the multi-direction input device is located to receive voice instructions. Among them, the voice instruction may be a sound signal collected by the voice module.

[0036] The control instruction can also be collected in other ways, and the present application does not limit the specific collection method of the control instruction.

[0037] Exemplarily, after receiving the control instruction, it is necessary to parse the received control instruction to obtain the target information. It can be understood that during the operation of the multi-direction input device, when receiving various complex instructions, there may be a situation where interference information is received. For example, when receiving a voice instruction, environmental sounds may cause errors. Thus, by parsing the control instruction to obtain the corresponding target information, the true and effective instruction can be obtained more accurately, enabling the terminal device to perform accurate display and reducing the probability of executing incorrect instructions.

[0038] Step S2: Generate a control signal according to the control instruction and send it to the terminal device, where the control signal is used to control the terminal device to perform a target operation.

[0039] That is to say, the multi-direction input device can generate a control signal according to the control instruction and send it to the terminal device, so that the terminal device performs a target operation according to the received control signal.

[0040] Among them, the target operation in step S2 includes at least one of the following operations: Virtual operations in a virtual scene displayed through a graphical user interface provided by the terminal device; Device control operations on the terminal device; Application function operations implemented using the terminal device.

[0041] For example, when the terminal device is a virtual reality device, a game device, or other devices that can display a virtual scene, the virtual operation may be, for example, controlling a character in the virtual scene to perform actions, controlling the change of the virtual scene's screen, etc.

[0042] The device control operation may include setting operations on the terminal device, such as setting the sound or screen brightness, etc. The setting operation may also include setting the virtual scene displayed on the terminal device, such as game settings, etc.

[0043] The application function operations may include controlling the start, shutdown, pause, sending text messages, making phone calls, playing music, etc. of the terminal device. The function operations may also include using the terminal device to control other devices, such as starting, pausing or shutting down the device, or performing other operations such as sending text messages, making phone calls, playing music, etc. The present application does not limit the specific content of the target operation.

[0044] In summary, the control method based on the multi-direction input device provided by the present application generates a control signal by receiving at least one of an operation instruction and a voice instruction to control the terminal device, realizes non-contact input of the user, and can satisfy simultaneous input of instructions for various complex operations of the user, improves the convenience of user input, improves the input efficiency of the user, and improves the user experience.

[0045] In an embodiment of the present application, when the control instruction includes an operation instruction and a voice instruction, generating a control signal according to the control instruction in step S2 may include the following steps S101 and S102.

[0046] Step S101: Parse the voice instruction to obtain the corresponding instruction text.

[0047] In some embodiments, a voice parsing model may be obtained, and the voice instruction is parsed based on the voice parsing model to obtain the corresponding instruction text. It can be understood that the voice parsing model is a technology that combines speech recognition, natural language processing (NLP) and data analysis. Its core function is to convert voice signals into structured information and extract valuable content from it. In an embodiment of the present application, the voice parsing model may be integrated into the voice module, or a separate processor loaded with the voice parsing model may be set to process the voice instruction. It can be understood that in related scenarios, such as scenarios where it is necessary to simultaneously satisfy the output of various complex operations or the user does not touch the multi-direction input device, etc., after the instruction text is extracted through the voice instruction, the user intention can be extracted.

[0048] For example, the speech parsing model can be a statistical speech model or a neural network speech model, etc. After the speech instruction is input into the speech parsing model, the speech instruction is converted into an instruction text. In some embodiments, the speech parsing model can also perform context tracking, identify the user's language pattern and emotion, etc. based on historical data, so as to perform speech interaction with the user based on the speech instruction. The present application does not limit the specific implementation functions of the speech parsing model. Understandably, the processor in the multi-direction input device can also save the converted instruction text to obtain reference context data. The reference context data includes, but is not limited to, the user intention and historical conversation context included in the speech instruction. In this way, the multi-direction input device can better understand the user's semantics and generate a more user-friendly dialogue strategy, so as to provide the user with a coherent and natural dialogue experience. At the same time, during the interactive dialogue process, the interaction data is recorded and saved, expanding the effective range of speech interaction and ensuring the real-time nature of the dialogue of the multi-direction input device.

[0049] Step S102, when the operation instruction conforms to the first operation instruction combination and the instruction text includes the first text, generate a first control signal, and the first control signal is used to control the terminal device to execute the target operation.

[0050] Among them, the first operation instruction combination represents a preset operation instruction combination. For example, the first operation instruction combination can be pressing the multi-direction input device twice continuously or the pressing duration of the multi-direction input device reaching the time threshold, etc.; the first text represents a preset text keyword. Understandably, when the user mechanically operates the multi-direction input device, there are situations where multiple complex operation outputs need to be completed simultaneously to control the terminal to execute the target operation. When the number of operations that need to be completed simultaneously exceeds the number of mechanical operations that the sensors in the multi-direction input device can detect, it is necessary to assist in realizing complex operations through voice input. In this embodiment, by confirming whether the operation instruction conforms to the preset first operation instruction combination and whether the instruction text includes the first text, it is determined whether the current situation conforms to the preset complex operation scenario. Among them, when the first operation instruction conforms to the first operation instruction combination and the instruction text includes the first text, the preset complex operation scenario is satisfied. At this time, the operation instruction and the speech instruction can be combined to assist the user in completing complex operations, effectively meeting the user's usage requirements.

[0051] For example, when the terminal device is a gaming device, by executing steps S101 to S102, complex operations of the user can be achieved within a short period of time. For example, in a virtual scenario, a game character needs to perform multiple complex actions simultaneously or open a virtual equipment panel to replace virtual equipment during movement. Due to different hand operation speeds of users, there may be situations where users cannot complete actions quickly or select the wrong virtual equipment to replace, resulting in mistakes in the game. By executing steps S101 to S102, the user can replace virtual equipment in the game through voice while performing manual operations, reducing the probability of the user opening the virtual device panel and reducing the probability of operation errors of the user.

[0052] In an embodiment of the present application, when the control instruction meets the preset conditions and there is a conflict between the voice instruction and the operation instruction, one of the voice instruction or the operation instruction is preferentially executed according to the preset conflict resolution strategy, and the preset conditions include that the voice instruction does not conform to the first operation instruction combination, and / or the instruction text does not include the first text. Among them, the conflict resolution strategy can be a preset priority algorithm, etc. It can be understood that there are situations where the user makes mistakes in expressing the voice instruction or makes mistakes in hand operations, resulting in incorrect operation instructions during the operation process. For example, when the terminal device is a gaming device, the user issues a voice instruction to move the game character to the left, but the user pushes the multi-direction input device to the right. At this time, the conflict resolution strategy is called to preferentially execute the voice instruction or the mechanical operation instruction for the current real-time scenario. In this way, it can effectively reduce the operation errors caused by the user's expression mistakes or accidental touches, and improve the user experience.

[0053] In an embodiment of the present application, before determining whether the operation instruction conforms to the first operation instruction combination, the operation instruction collected after the user mechanically operates the multi-direction input device can also be filtered and analyzed to obtain an accurate operation instruction. It can be understood that after obtaining the operation instruction, the mechanical analysis model can be used to filter and analyze the operation instruction to filter out some interference signals. Among them, the mechanical analysis model is used to perform simulation analysis on the structure of the multi-direction input device, the movement of the operating member, and the force condition of the operating member. For example, according to the operation habits of a specified user, the force applied by the user to the operating member can be recorded in real time and the mechanical analysis model can be trained according to the recorded data. In this way, when the operation instruction is input into the mechanical analysis model, an accurate operation instruction can be obtained. In this way, it can effectively reduce the generation of interference instructions caused by the user's accidental touches and ensure the stable operation of the multi-direction input device.

[0054] In an embodiment of the present application, when the control instruction includes a voice instruction, the multi-direction input device in step S2 generating a control signal according to the control instruction may include: Parsing the voice instruction to obtain the corresponding instruction text; When the instruction text includes at least one of the second text or the appointment time, a second control signal is generated according to the instruction text, and the second control signal is used to control the terminal device to perform a target operation after the appointment time.

[0055] Among them, the process of parsing the voice instruction is substantially the same as the process in step S101, and will not be elaborated here.

[0056] The second text is used to represent the user intention text for performing an operation in a preset scenario. Among them, the preset scenario can be represented by the terminal device state, position coordinates, context-related data, etc., and the present application does not limit the specific content of the second text. Understandably, there are the following situations where the user makes the terminal device perform a target operation after the appointment time through the input voice instruction: (a) the voice instruction only includes the second text, for example, move horizontally when the virtual character displayed on the graphical user interface reaches position A; (b) the voice instruction only includes the appointment time, for example, move horizontally after 5s; (c) the voice instruction includes both the second text and the appointment time, for example, move horizontally after 5s and move vertically when reaching position A. In this way, the voice instruction can facilitate the user's advance planning, reduce the probability of mistakes caused by distracted operations, reduce the complexity of operations, improve the user's operation efficiency, and improve the user's usage experience.

[0057] In an embodiment of the present application, when the control instruction includes a voice instruction, the multi-direction input device in step S2 generating a control signal according to the control instruction may include: Parsing the voice instruction to obtain the corresponding instruction text; When receiving the status feedback information fed back by the terminal device and the instruction text includes the third text, a third control signal is generated according to the instruction text, and the third control signal is used to control the terminal device to perform a target operation. Among them, the status feedback information is used to indicate that the terminal device is in a configuration state.

[0058] Understandably, the process of parsing the voice instruction is substantially the same as the process in step S101, and will not be elaborated here.

[0059] In some embodiments, after the multi-direction input device establishes a communication connection with the terminal device, the terminal device may also periodically feedback status information to the multi-direction input device. Among them, the preset state may include a configuration state or a preset scenario state. The configuration state can be used to indicate that the terminal device is in a state where it can be set. For example, in the initial stage after the terminal device is started, it can enter the configuration state for personalized settings (such as sound settings, screen display settings, game settings, etc.) to meet the personalized needs of users. The preset scenario state is used to indicate that the terminal device is currently in a preset scenario.

[0060] Among them, the third text is used to represent the target operation that needs to be repeatedly executed and / or the operations that need to be continuously performed within a short period of time.

[0061] For example, after the terminal device is started, when it is confirmed through the status feedback information that the terminal device is in the configuration state and the instruction text includes the third text, the setup operation in the initial stage of starting the terminal device can be completed through voice instructions. In this way, compared with completing the setup through mechanical operations, the operation efficiency can be effectively improved and the user experience can be enhanced.

[0062] Another example is when the same operation needs to be performed at multiple locations and it takes a long time for the user to perform the mechanical operation. At this time, through voice instructions, the corresponding target operation is executed to replace the repetitive and cumbersome mechanical operation. In this way, through voice instructions, the manual repetitive operation can be transformed into an automated process, reducing the probability of errors in the manual repetitive operation, reducing the user's time cost, and reducing user fatigue.

[0063] In an embodiment of the present application, the control method based on the multi-direction input device further includes: Obtain the scene feedback information of the terminal device and output the first voice information according to the scene feedback information.

[0064] Among them, after the terminal device establishes a communication connection with the multi-direction input device, it can also periodically send the scene feedback information to the multi-direction input device. The scene feedback information includes: the user status information collected by the terminal device; or the frame rate of the video displayed by the terminal device; or the scene information collected by the terminal device.

[0065] Exemplarily, the user status information is used to represent the parameters of the user's own status when operating the multi-direction input device. It can be understood that when the user operates the multi-direction input device, the user's own physical status will also affect the operation status. For example, the user's mental state is highly concentrated due to long-term operation. In this way, the user's current operation status can be effectively reflected through the user status information.

[0066] Understandably, the terminal device may include a biometric collection module for collecting biometric data of the user. For example, the eye movement trajectory characteristics, the change frequency of facial expressions, and the duration of limb postures of the user are collected through a camera. The present application does not limit the specific collection method and content of the biometric data. According to the collected biometric data, the user state is evaluated through a preset evaluation model. In one embodiment, a load evaluation model is obtained. According to the load evaluation model and the biometric data, an evaluation parameter is obtained. When the evaluation parameter is greater than or equal to the first threshold, corresponding user state information is generated and sent to the multi-direction input device. After receiving the user state information, the multi-direction input device can output corresponding voice information through the voice module. For example, when the evaluation parameter is the duration of the user's attention to the screen, if the evaluation parameter is greater than or equal to the preset first threshold, it means that the user has been paying attention to the screen for a long time and may have eye fatigue. At this time, the multi-direction input device can remind the user to rest through the voice information.

[0067] During the operation of the terminal device, the frame rate of the displayed video content is obtained. When the frame rate is less than or equal to the first preset frame rate, the terminal device may have situations such as insufficient memory, high central processing unit temperature, or slow network speed. At this time, when the user issues a corresponding operation instruction through mechanical operation, the probability of inaccurate operation increases. Thus, when the multi-direction input device determines that the frame rate of the video of the received terminal device is less than or equal to the first preset frame rate, the multi-direction input device can output voice information through the voice module to remind the user or assist the user in operation, so as to effectively improve the accuracy of the user's operation. Another example is that when the frame rate is greater than or equal to the second preset frame rate, it means that the current displayed picture content of the terminal device is relatively complex. Thus, when the multi-direction input device determines that the frame rate of the video of the received terminal device is greater than or equal to the second preset frame rate, the multi-direction input device can output voice information through the voice module to remind the user or assist the user in operation, or give an evaluation of the user's operation result, so as to improve the user experience. In other embodiments, the feedback information may also include other working parameters of the terminal device, such as the update frequency of dynamic elements of the graphical user interface, the interface image quality, and the packet loss rate, etc. The present application does not limit the specific content of the feedback information collected by the terminal device during operation.

[0068] The terminal device collects scene information during operation, and scene changes have a certain impact on the user's operations. For example, in a virtual scene, when the brightness of the scene exceeds the preset brightness range, it is difficult for the user to observe the details and changes on the screen in a timely manner. At this time, the user has a reaction delay. By interacting with the user in advance or in a timely manner through voice, such as reminding or guiding the user's operations, it is convenient for the user to react quickly and improve the user experience. Another example is that when controlling a drone through a multi-direction input device, affected by factors such as the flying height of the drone and the external environment, if the user cannot pay attention to the flight situation of the drone in time during mechanical operation, it may cause the drone to fall and cause losses to the user. Thus, by outputting voice information, the relevant status of the terminal device can be reminded to the user in real time, ensuring the stable operation of the terminal device.

[0069] In an embodiment of the present application, the control method based on the multi-direction input device further includes the following steps S201 to S203.

[0070] Step S201: Obtain historical operation trajectory data and real-time operation trajectory data, and output voice information.

[0071] Among them, the historical operation trajectory data includes control instructions and corresponding target operations that occurred within a first preset time period, and the real-time operation trajectory data includes control instructions and corresponding target operations that occurred within a second preset time period. The first preset time period occurred before the current moment, and the second preset time period includes the current moment.

[0072] Step S202: Compare the historical operation trajectory data with the real-time operation trajectory data to obtain a comparison result.

[0073] Step S203: Output second voice information according to the comparison result.

[0074] Understandably, when the user operates the multi-direction input device, the operation may not be in place. By analyzing the historical operation trajectory data and the real-time operation trajectory data, the deficiencies in the user's operation can be effectively supplemented, and complex operations can also be better realized, improving the user experience. In one embodiment, a behavior pattern analysis model is obtained. The behavior pattern analysis model is used to mine the user's behavior data, identify the user's operation rules, and predict the user's operation results. According to the behavior pattern analysis model, control instructions, operation function sets, and the user's historical operation trajectory data, historical operation data is obtained. The historical operation data is compared with the user's real-time operation trajectory data to determine the user's normal operation. When the real-time operation trajectory data is abnormal, a voice message is output. Understandably, based on the user's operation data in the historical data, the result of the user's real-time operation is predicted. When there is a large deviation between the predicted result and the actual result, it indicates that there are deficiencies in the user's operation process. At this time, the multi-direction input device reminds or guides the user through voice. The multi-direction input device can also automatically supplement the parts of the user's operation that are insufficient according to the user's operation habits. By personalized judging the user's operation trajectory data, the effect of performing the target operation can be effectively analyzed, reducing the probability of deviation or error caused by small errors.

[0075] In an embodiment of the present application, the control method based on the multi-direction input device further includes: When the multi-direction input device is in the custom state, custom voice instructions are obtained and saved, and the custom voice instructions are associated with the preset operation instructions.

[0076] Among them, the custom voice instructions are used to execute the target operation corresponding to the preset operation instructions. Understandably, the user can customize the voice instructions according to their own habits, so as to facilitate the user to save time costs when multiple different control instructions need to be output simultaneously. For example, at a certain moment, the user needs to voice input 5 common voice instructions, and these 5 voice instructions respectively correspond to multiple operation instructions. If the user fully expresses all the instructions, even if the voice is pre-input, it will take a lot of time and there may be misstatements. If the user, when in the custom state, pre-sets the associated words for the common voice instructions in advance, for example, the user pre-sets the custom voice instruction as voice input of the number 1, and this custom voice instruction is associated with multiple operation instructions, then when the user is in the operation process, only need to directly say the number 1, and the operation instructions that the user expects to perform can be directly retrieved and implemented, which can effectively shorten the voice input time and reduce the probability of inputting incorrect instructions.

[0077] Please refer to again Figure 4, the present application further provides a multi-direction input device 10, which includes an operating member 102 and a voice module 106. The operating member 102 is configured to generate an operation instruction in response to a mechanical operation of a user, and the voice module 106 is configured to collect a voice instruction and execute the control method based on the multi-direction input device according to any one of the above embodiments. The voice module 106 may be integrally provided with the processor 1001. The processor 1001 is configured to analyze and process the collected data, and the voice module 106 has functions such as voice collection, natural language processing, and dialogue management. The voice module 106 may also be separately and independently provided from the processor 1001, and the present application does not limit this. Understandably, in some embodiments, the multi-direction input device 10 further includes a memory 1002, and the memory 1002 is configured to store computer programs and data. The memory 1002 stores instructions executable by the processor 1001, and the instructions are executed by the processor 1001 to execute the control method based on the multi-direction input device according to any one of the above embodiments.

[0078] Wherein, the memory 1002 and the processor 1001 are connected in a bus manner. The bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors 1001 and the memory 1002 together. The bus may also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits together, which are well known in the art, and thus will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver may be an element or multiple elements, such as multiple receivers and transmitters, and provides a unit for communicating with various other devices on a transmission medium. The data processed by the processor 1001 is transmitted on a wireless medium through an antenna. Further, the antenna also receives data and transmits the data to the memory 1002. The processor 1001 is responsible for managing the bus and general processing, and may also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory 1002 may be used to store data used by the processor 1001 when performing operations.

[0079] Please continue to refer to Figure 4 , the present application further provides a control handle 1, and the control handle 1 includes the multi-direction input device 10 described in the above embodiments. The control handle 1 allows a user to precisely control a target object in a three-dimensional space or multiple degrees of freedom through the multi-direction input device 10. By inputting a control instruction including at least one of an operation instruction obtained by responding to a mechanical operation on the multi-direction input device 10 and a voice instruction collected by the multi-direction input device 10, signal collection is performed from multiple dimensions, which can effectively meet the user's need to output multiple complex operations simultaneously, adapt to diverse scenarios, and provide a personalized service experience for the user.

[0080] Please continue to refer to Figure 4 , the present application further provides an information processing system 30, which includes a terminal device 20 and the multi-direction input device 10 described in the above embodiments. The multi-direction input device 10 is communicatively connected to the terminal device 20. For example, the multi-direction input device 10 transmits an encrypted control signal to the terminal device 20 through a low-latency communication protocol. Among them, the encrypted control signal can encrypt the signal through algorithms such as the Data Encryption Standard (DES), the Elliptic Curve Cryptography (ECC), or other encryption algorithms, or encrypt the signal in other ways. The present application does not limit the specific implementation manner of the encrypted control signal.

[0081] It should be understood that the various embodiments of the present application can be combined arbitrarily. For example, they can be used alone or in combination with each other to achieve different technical effects, and this is not limited herein.

[0082] The above content is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A control method based on a multi-directional input device, wherein the multi-directional input device establishes a communication connection with a terminal device, characterized in that: The control method comprises: receiving a control instruction, wherein the control instruction includes at least one of an operation instruction obtained in response to a mechanical operation on the multi-directional input device and a voice instruction collected by the multi-directional input device; A control signal is generated according to the control instruction and sent to the terminal device, wherein the control signal is used to control the terminal device to perform a target operation; wherein the target operation includes at least one of the following operations: Virtual operations in a virtual scene displayed by a graphical user interface provided by the terminal device; Device control operation on the terminal device; The application function operation is implemented through the terminal device.

2. The control method according to claim 1, characterized in that: When the control instruction includes the operation instruction and the voice instruction, the multi-directional input device generates a control signal according to the control instruction, including: Parsing the voice command to obtain a corresponding command text; When the operation instruction conforms to the first operation instruction combination and the instruction text includes the first text, a first control signal is generated, where the first control signal is used to control the terminal device to perform the target operation; When the control instruction meets a preset condition and there is a conflict between the voice instruction and the operation instruction, the voice instruction or the operation instruction is preferentially executed according to a preset conflict resolution strategy, and the preset condition includes: the voice instruction does not meet the first operation instruction combination, and / or the instruction text does not include the first text.

3. The control method according to claim 1, characterized in that: When the control instruction includes the voice instruction, the multi-directional input device generates a control signal according to the control instruction, including: Parsing the voice command to obtain a corresponding command text; When the instruction text includes at least one of a second text or a scheduled time, a second control signal is generated according to the instruction text, and the second control signal is used to control the terminal device to perform the target operation after the scheduled time.

4. The control method according to claim 1, characterized in that: When the control instruction includes the voice instruction, the multi-directional input device generates a control signal according to the control instruction, including: Parsing the voice command to obtain a corresponding command text; When status feedback information fed back by the terminal device is received and the instruction text includes a third text, a third control signal is generated according to the instruction text, and the third control signal is used to control the terminal device to perform the target operation, wherein the status feedback information is used to indicate that the terminal device is in a preset state.

5. The control method according to claim 1, characterized in that: The method further comprises: Obtaining scene feedback information of the terminal device; Outputting first voice information according to the scene feedback information; The scene feedback information includes: User status information collected by the terminal device; or The frame rate of the video content displayed by the terminal device; or The scene information collected by the terminal device.

6. The control method according to claim 1, characterized in that: The method further comprises: Acquire historical operation trajectory data and real-time operation trajectory data, wherein the historical operation trajectory data includes the control instruction and the corresponding target operation occurring within a first preset time period, and the real-time operation trajectory data includes the control instruction and the corresponding target operation occurring within a second preset time period, wherein the first preset time period occurs before a current moment, and the second preset time period includes the current moment; Comparing the historical operation trajectory data with the real-time operation trajectory data to obtain a comparison result; The second voice information is output according to the comparison result.

7. The control method according to claim 1, characterized in that: The method further comprises: When the multi-directional input device is in a customized state, a customized voice instruction is acquired and saved, the customized voice instruction is associated with the preset operation instruction, and the customized voice instruction is used to execute the target operation corresponding to the preset operation instruction.

8. A multi-directional input device, characterized in that: It comprises an operating element and a voice module, wherein the operating element is used to respond to the user's mechanical operation to generate an operating instruction, and the voice module is used to collect voice instructions and execute the control method based on the multi-directional input device as described in any one of claims 1 to 7.

9. A control handle, characterized in that: The control handle comprises the multi-directional input device as claimed in claim 8.

10. An information processing system, characterized in that: The information processing system comprises a terminal device and the multi-directional input device as claimed in claim 8, wherein the multi-directional input device establishes a communication connection with the terminal device.

Images