Human-computer interaction method and related device
By assisting the communication and interaction between the auxiliary interactive device and the target interactive device, the problem of complex operation of old devices in complex business scenarios is solved, and the convenience of human-computer interaction is improved.
Patent Information
- Application Number
- CN202510189171.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-20
- Publication Date
- 2025-06-06
AI Technical Summary
In the prior art, terminal devices lacking screens or small human-computer interaction interfaces have complex operations when responding to complex business scenarios, which affects the convenience of interaction.
Through the auxiliary interactive device, the auxiliary interactive interface is displayed according to the device interaction type of the target interactive device, the operation characteristic information in the interaction operation is extracted, and the target interactive device is allowed to perform corresponding device behavior.
It improves the convenience of human-computer interaction with old devices, allowing users to efficient human-computer interaction with target interaction devices through auxiliary interaction devices that adapt to complex business scenarios.
Smart Images
Figure CN120104007A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of human-computer interaction technology, and in particular to a human-computer interaction method and related devices. Background Art
[0002] In the related art, for terminal devices that lack screens or have relatively small human-computer interaction interfaces, such as old mobile phones, smart home devices, and display devices, human-computer interaction relies on the computer's keyboard and mouse input interaction mode or the touch interaction mode of smart terminals. However, when responding to complex business scenarios, the above interaction modes all have the defect of complex operation, which greatly affects the convenience of interaction. Summary of the invention
[0003] The purpose of this application is to provide a human-computer interaction method and related devices, so that users can perform human-computer interaction with target interactive devices through auxiliary interactive devices that adapt to complex business scenarios, thereby improving the convenience of human-computer interaction with old devices.
[0004] The present application embodiment provides a human-computer interaction method, which is applied to an auxiliary interaction device, including: According to the device interaction type of the target interactive device, an auxiliary interaction interface is displayed; the auxiliary interaction interface is used to display the device state of the target interactive device, the target interactive device is used to execute device behavior, and the device behavior is used to change the device state of the target interactive device; In response to an interactive operation on the auxiliary interactive interface, extracting operation feature information in the interactive operation; In response to the operation characteristic information, interact with the target interaction device so that the target interaction device executes the device behavior specified by the interaction operation.
[0005] In some embodiments, displaying the auxiliary interaction interface according to the device interaction type of the target interaction device includes: When the target interactive device has an operation interface, displaying the auxiliary interactive interface including the operation interface of the target interactive device and a first interactive control for responding to the interactive operation; the first interactive control is generated according to the interface element of the operation interface; When the target interactive device does not have an operation interface, the auxiliary interactive interface including a device status screen and a second interactive control for responding to the interactive operation is displayed; the device status screen is used to simulate the device status of the target interactive device.
[0006] In some embodiments, when the target interactive device has an operating interface, displaying the auxiliary interactive interface including the operating interface of the target interactive device includes at least one of the following: When the operation interface is a command line interface, displaying the auxiliary interaction interface including the operation interface and the first interaction control for responding to the voice interaction operation; When the operation interface is a graphical user interface, displaying the auxiliary interaction interface including the operation interface and the first interaction control for responding to the somatosensory interaction operation; When the operation interface is a multimedia user interface, the auxiliary interaction interface including the operation interface and the first interaction control for responding to voice interaction operations and somatosensory interaction operations is displayed.
[0007] In some embodiments, in response to the interactive operation on the auxiliary interactive interface, extracting the operation feature information in the interactive operation includes: In response to the somatosensory interaction operation collected by the motion collection device, the somatosensory action feature information in the somatosensory interaction operation is extracted; and / or in response to the voice interaction operation collected by the voice collection device, the voice feature information in the voice interaction operation is extracted.
[0008] In some embodiments, the method further comprises: Identifying interface elements in the auxiliary interaction interface; Display visual prompt content explaining the interface elements in the auxiliary interaction interface, or broadcast voice prompt content explaining the interface elements.
[0009] In some embodiments, in response to the operation characteristic information, interacting with the target interaction device so that the target interaction device performs the device behavior specified by the interaction operation includes: generating corresponding interactive control instructions according to the operation characteristic information; According to a preset mapping relationship, the interactive control instruction is mapped to the target interactive device, so that the target interactive device executes the device behavior specified by the interactive operation; After the target interactive device executes the device behavior, the auxiliary interactive interface is updated.
[0010] In some embodiments, the method further comprises: Acquire device information of the target interactive device; the device information includes the device interaction type, device status, and executable device behavior of the target interactive device; According to the device information, a first mapping relationship and a second mapping relationship are constructed; the first mapping relationship describes the mapping relationship between the operation feature information and the interactive control instruction, and the second mapping relationship describes the mapping relationship between the device behavior and the interactive control instruction.
[0011] The present application also provides a human-computer interaction device, including: The first module is used to display an auxiliary interaction interface according to the device interaction type of the target interaction device; the auxiliary interaction interface is used to display the device state of the target interaction device, the target interaction device is used to execute a device behavior, and the device behavior is used to change the device state of the target interaction device; A second module is used to extract operation feature information in response to an interaction operation on the auxiliary interaction interface; The third module is used to interact with the target interactive device in response to the operation characteristic information, so that the target interactive device executes the device behavior specified by the interactive operation.
[0012] An embodiment of the present application further provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned human-computer interaction method when executing the computer program.
[0013] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned human-computer interaction method is implemented.
[0014] The beneficial effects of the present application are as follows: the auxiliary interactive device is used to communicate and interact with the target interactive device. The auxiliary interactive device can display an auxiliary interactive interface for displaying the device status of the target interactive device according to the device interaction type of the target interactive device. When the user performs an interactive operation on the auxiliary interactive interface, the auxiliary interactive device can respond to the interactive operation on the auxiliary interactive interface, extract the operation feature information in the interactive operation, and respond to the operation feature information to interact with the target interactive device, so that the target interactive device performs the device behavior specified by the interactive operation. Since an auxiliary interactive device that can adapt to complex business scenarios is used to provide a human-computer interaction scenario, the auxiliary interactive device generates an auxiliary interactive interface according to the device status of the target interactive device for the user to perform interactive operations, so that the user can perform human-computer interaction with the target interactive device through the auxiliary interactive device that can adapt to complex business scenarios, thereby improving the convenience of human-computer interaction with old devices. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 This is a diagram of an implementation environment of a human-computer interaction method provided by an embodiment.
[0016] Figure 2 A flowchart of a human-computer interaction method provided by an embodiment.
[0017] Figure 3 A schematic diagram of a method for displaying an auxiliary interactive interface provided by an embodiment.
[0018] Figure 4 It is a flowchart of a specific method of step S203 provided by an embodiment.
[0019] Figure 5 It is a structural schematic diagram of a human-computer interaction device provided by an embodiment.
[0020] Figure 6 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment. DETAILED DESCRIPTION
[0021] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0022] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown can be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second" and the like in the specification, claims and drawings are used to distinguish similar objects, and are not used to describe a specific order or sequence.
[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0024] The method provided in this application can be applied to Figure 1 The implementation environment shown includes an auxiliary interactive device and a target interactive device, and the auxiliary interactive device and the target interactive device are connected via a communication network 130. The auxiliary interactive device and the target interactive device can communicate with each other via the communication network 130. The communication network 130 uses standard communication technology and / or protocols, usually the Internet, but can also be any network, including but not limited to Bluetooth, local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, dedicated network or any combination of virtual private networks). In some embodiments, customized or dedicated data communication technology can be used to replace or supplement the above data communication technology.
[0025] The auxiliary interaction devices involved in this application include but are not limited to intelligent voice interaction devices, intelligent somatosensory interaction devices, and virtual reality devices, etc. The target interaction devices include but are not limited to mobile phones, computers, smart home appliances, vehicle terminals and aircraft, etc. The human-computer interaction method involved in this application runs on the auxiliary interaction device, and can run on the auxiliary interaction device in the form of a browser, or in the form of an independent APP, etc.
[0026] In combination with the above-mentioned implementation environment, users can generate interactive operations on the auxiliary interactive interface provided by the auxiliary interactive device. The auxiliary interactive device can respond to the interactive operations on the auxiliary interactive interface to extract the operating feature information in the interactive operations. Based on the extracted operating feature information, the auxiliary interactive device can interact with the target interactive device through the communication network 130 and send corresponding instructions to the target interactive device to enable the target interactive device to perform the device behavior specified by the interactive operation.
[0027] In one embodiment, Figure 2 As shown, a human-computer interaction method is provided, which is applied to Figure 1 The auxiliary interactive device in the example is used to illustrate, including the following steps: Step S201: displaying an auxiliary interaction interface according to the device interaction type of the target interaction device.
[0028] Among them, the auxiliary interaction interface is used to display the device status of the target interaction device, the target interaction device is used to execute device behavior, and the device behavior is used to change the device status of the target interaction device.
[0029] The device interaction type refers to the type of interaction between the user and the device. For example, the device interaction type may include command line interaction, menu-driven interface, graphical user interface, touch interaction, gesture interaction, voice interaction, and multimodal interaction.
[0030] Device behavior refers to the operation performed by the device after receiving the interactive operation generated by the user during human-computer interaction between the user and the device. For example, when the user clicks the icon of the corresponding application on the touch screen of the device, the device performs the device behavior of opening the application; when the user types the corresponding character information through the keyboard, the device performs the device behavior of searching for the corresponding character; when the user types the remote control information of increasing the temperature through the remote control, the device performs the device behavior of increasing the temperature, etc.
[0031] The device state refers to the visual feature information that the device feeds back to the user after executing the corresponding device behavior. For example, after the device executes the device behavior of opening the application, the device state of the device changes to displaying the interface of the application; after the device executes the device behavior of searching for the corresponding characters, the device state of the device changes to displaying the searched characters; after the device executes the device behavior of increasing the temperature, the device state of the device changes to outputting a higher temperature.
[0032] In a specific implementation, the auxiliary interaction device communicates with the target interaction device through a communication network, obtains the device hardware information of the target interaction device, determines the device type and device interaction type of the target interaction device according to the device hardware information of the target interaction device, and then displays an auxiliary interaction interface that can be adapted to the target interaction device, and displays the device status of the target interaction device through the auxiliary interaction interface. For example, when the target interaction device is a smart phone, after the auxiliary interaction device obtains the device hardware information of the target interaction device, it generates an auxiliary interaction interface that can display the operation interface of the target interaction device; when the target interaction device is an air conditioner, after the auxiliary interaction device obtains the device hardware information of the target interaction device, it generates an auxiliary interaction interface that can display the working condition information of the target interaction device.
[0033] Step S202: In response to an interactive operation on the auxiliary interactive interface, extracting operation feature information in the interactive operation.
[0034] Interaction operations on the auxiliary interaction interface refer to the interaction operations performed by the user directly or indirectly on the auxiliary interaction interface. For example, the interaction operations on the auxiliary interaction interface can be performed by controlling the mouse or by somatosensory movements. When the user performs an interaction operation on the auxiliary interaction interface, the auxiliary interaction device extracts the operation feature information in the interaction operation.
[0035] In a specific implementation, the auxiliary interaction device provides the user with a scenario for implementing interactive operations on the auxiliary interaction interface, and detects the user's interactive operations on the auxiliary interaction interface in real time. When the interactive operations on the auxiliary interaction interface are detected, the auxiliary interaction device extracts the operation feature information in the interactive operations. For example, the auxiliary interaction device provides the user with a scenario for implementing interactive operations based on somatosensory movements. The user can implement somatosensory movement interactive operations on the auxiliary interaction interface in the somatosensory movement detection area. The auxiliary interaction device detects the somatosensory movement interactive operations implemented by the user in real time. When the somatosensory movement interactive operations are detected, the auxiliary interaction device extracts the somatosensory movement feature information in the somatosensory movement interactive operations.
[0036] Furthermore, in order to facilitate fixing some specific operation actions of the target interactive device, somatosensory action data and corresponding mapping instructions are constructed in the auxiliary interactive device.
[0037] First, a somatosensory interactive operation instruction set is constructed in the auxiliary interactive device, and the interactive operation instruction set supports the application and operating system of the target interactive device. The interactive operation instruction set can be used for touch-screen-like operation of traditional computers plus projection or smart TV screens. The existing operating system basically supports touch-screen operation and traditional keyboard and mouse operation, and a mapping relationship can be easily established for somatosensory operation.
[0038] Furthermore, to detect whether the target interactive device is a local device or a remote device, it can be determined whether it is a traditional keyboard and mouse application or a remote application by whether the system of the target interactive device monitors the port of the common remote desktop protocol or obtains the system API of the application. If it is detected that the current application has not created relevant gestures and somatosensory controls, it can be reminded that only the most basic somatosensory or gesture control can be performed.
[0039] The steps to create motion control include: For these fixed interface navigation operations, which require operating the cursor, some somatosensory operation data can be preset to give a unique ID and somatosensory operation information for binding. Custom modification is supported, and each ID corresponds to a named action function, which can be referenced again through a text description or a voice command function corresponding to the text. A requirement description, function description, and parameter description can be edited afterwards, and it can be invoked through a command entry or a voice assistant to make up for the shortcomings of a single command channel.
[0040] The first category is for action functions that support automatic setting of coordinate parameters in a system operation in the function description, that is, automatic setting of parameter type somatosensory keyword functions, which are expressed as Mmove (position_A, position_B) in the editing stage, where position_A automatically corresponds to the movement of the position of a human body origin (such as a hand) from A to B. The second category is for parameters that need to lock a specific target or the content of a UI element, which need to describe the parameter content of the target corresponding to the action, and replace the automatically filled position_B in the first category of statements with the position of capturing a recognizable icon element; a non-basic somatosensory operation function needs to include a position parameter description acquisition process, which does not need to be reflected in the action, but in the description of the created action, and finally the operation function becomes: Function doTouchButton() { / / Get the right button and click it position_B = GetPosition(ButtonID); Mmove(position_A,position_B); Touch(); / / Set a click action here } Bindaction("action name", ID_in_Database, doTouchButton); The above pseudo code represents a binding process of creating somatosensory data and operation instructions. After an "action name" is successfully registered, it can continue to be used by other voice commands. The function references a series of actions represented by the doTouchButton function to facilitate continuous execution when non-somatosensory actions are triggered. For example, it is fixed to match basic touch screen operations, such as single click, slide, press and drag, and requires large-screen devices (such as computers and smart TVs) to support 3D shooting of multiple cameras to obtain positions; model the instruction data from somatosensory data to the operating device, generate a basic function set, and use parameterless action functions to correspond to the corresponding system operation instructions.
[0041] The final virtual space is mapped to the trajectory and click effect of the operation interface of the auxiliary interactive device, which is determined by the mouse cursor mapping step and the coordinate determination step of the marked body part, which is determined by any convenient body part such as the hand or the head. The coordinate system is created with the position and gesture of the hand as the core, and modeling is performed through 3D photography to establish a physical virtual space and project it to the corresponding position of the operation interface of the current smart device, assuming it is (x, y, z). If it is a full-simulation touch screen operation, the position of the hand is used as the contact point or a touch is formed when the hand is clenched to avoid accidental touch. The projection position of the hand on the screen can have an aperture visually, so that a touch screen operation such as a mobile phone can be converted into a content that is operated by somatosensory at a distance; these are compatible with the traditional mapping of buttons and touch screen actions to the operation interface.
[0042] For some applications that need to be operated in virtual space, there can be a set of somatosensory action databases that require the body as the origin of the coordinate system of the virtual space. There is no need to rely on somatosensory devices. The entire posture of the human body is obtained with pure vision to correspond to the virtual space of the game screen. It is modeled for separate game applications. If the hardware supports infrared depth detection, the infrared probe can be used to establish a matrix point for the object for position correction.
[0043] For gesture and body recognition, we can base on the common gesture recognition model. The basic process mainly includes image preprocessing step, key point detection step, gesture estimation step, gesture recognition step and mapping step. The specific principles of each step are as follows: 1. Image preprocessing step: Perform preprocessing operations such as denoising, contrast enhancement, and resizing on the captured images or video frames to improve the accuracy of subsequent processing; 2. Key point detection step: Detect key points of the human body in the image through deep learning algorithms (such as convolutional neural network CNN). These key points usually include characteristic parts of the human body such as joints, head, hands, etc., and perform deep measurement of gestures and postures; 3. Posture estimation step: Based on the detected key points, the posture of the human body is estimated using a mathematical model or algorithm. This includes two-dimensional posture estimation and three-dimensional posture estimation, the latter of which can more accurately reflect the spatial position and direction of the human body; 4. Posture recognition step: Match the estimated posture with the predefined posture template to identify the specific posture or action of the human body, extract key gestures or directly transmit posture data for comparison with template posture data or somatosensory operation instruction data set that can map operation instructions; 5. Mapping step: According to the result of the gesture recognition step and the mapping position of the hand in the virtual space, the simulated somatosensory operation is automatically converted into keyboard and mouse operation based on the coordinate system of the aforementioned virtual space.
[0044] In some embodiments, for applications that support keyboard and mouse operations, each "touch" action or gesture can be defined, which means that the cursor moves to the aperture position where the hand is finally projected. The mapping can be operated without drastic changes in the hand, and then the actions of single-click, right-click, and single-click and long-press can be defined.
[0045] Step S203: interact with the target interactive device in response to the operation characteristic information, so that the target interactive device executes the device behavior specified by the interactive operation.
[0046] In a specific implementation, the auxiliary interactive device generates corresponding interactive control instructions according to the operation characteristic information, communicates with the target interactive device through a communication network, and the auxiliary interactive device interacts with the target interactive device, maps the interactive control instructions to the target interactive device, and the mapped interactive control instructions are control instructions that the target interactive device can recognize, so that the target interactive device receives the mapped interactive control instructions and executes the corresponding device behavior. After the target interactive device executes the device behavior, it changes to the corresponding device state. For example, the device interaction type of the target interactive instruction is that human-computer interaction can be performed through mouse operation. The auxiliary interactive device extracts the somatosensory action characteristic information in the somatosensory action interaction operation, converts the somatosensory action characteristic information into an interactive control instruction, and then when interacting with the target interactive device, maps the interactive control instruction to the interactive control instruction generated by the mouse operation. When the target interactive device receives the mapped interactive control instruction, it executes the device behavior corresponding to the mouse operation.
[0047] In some embodiments, the above step S201 specifically includes: when the target interactive device has an operation interface, displaying an auxiliary interactive interface including the operation interface of the target interactive device and a first interactive control for responding to interactive operations; the first interactive control is generated according to the interface elements of the operation interface; when the target interactive device does not have an operation interface, displaying an auxiliary interactive interface including a device status screen and a second interactive control for responding to interactive operations; the device status screen is used to simulate the device status of the target interactive device.
[0048] In this embodiment, the auxiliary interaction device generates a corresponding auxiliary interaction interface according to whether the target interaction device has an operation interface, so as to adapt to the device interaction type of the target interaction device.
[0049] It is understandable that when there are interface elements of controls that are easy to operate in the operation interface of the target interactive device, the first interactive control is obtained by extracting the interface elements of the operation interface of the target interactive device. When there are no interface elements of controls that are easy to operate in the operation interface of the target interactive device, the first interactive control is an additionally added control and is associated with the interface elements of the operation interface of the target interactive device. For example, when the operation interface of the target interactive device is a graphical user interface, there are interface elements of controls that are easy to operate, and the operation interface of the target interactive device is replicated, and the corresponding interface elements (such as application icons and mouse icons, etc.) in the operation interface of the target interactive device are used as the first interactive control. When the operation interface of the target interactive device is a command line interface, there are no interface elements of controls that are easy to operate, and the first interactive control is additionally added and displayed in the auxiliary interactive interface.
[0050] When the target interactive device has an operation interface, the auxiliary interactive device interacts with the target interactive device to display the operation interface of the target interactive device in the auxiliary interactive interface, and generates a first interactive control according to the interface elements of the operation interface of the target interactive device. When the target interactive device does not have an operation interface, the auxiliary interactive device interacts with the target interactive device to simulate and display the device state of the target interactive device in the auxiliary interactive interface, and adds and displays the corresponding second interactive control to respond to the interactive operation on the auxiliary interactive interface.
[0051] Furthermore, it is possible to detect how many button-type icons and window events that can perceive cursor operations are in the current application window of the target interactive device, and reuse the somatosensory operations of the target interactive device to the auxiliary interactive device. For example, a video player application or a music player has similar icons for common buttons, and some somatosensory operations defined by common icons can be reused.
[0052] Furthermore, the corner elements are detected to identify the registered operation events of the boundaries of individual icons or application windows, and prompts to create a somatosensory gesture. By detecting the elements of the currently active small window, a somatosensory operation is applied to create a line to obtain the edge features of the upper right corner, and "press and hold" to pop up a prompt to reduce the window. For example, the coordinate capture of the somatosensory operation function statement is set to capture the features of the upper right corner of the active window, and the natural language is automatically disassembled into precise system operations through the large model to obtain the main window ID of the active window registered in the system's GUI, and then the size of the area in the upper right corner is obtained, and the pressing action is simulated to give a floating operation prompt, create a gesture animation, guide the user's gesture movement, and retain the function description near the generated operation statement for fuzzy matching. When the user creates a custom action function for a new application, whether there is similarity, the user can detect that the fuzzy matched action function can already meet the needs, and there is no need to create a new action function and somatosensory data. For another example, for alternative large-screen devices such as projectors, 3D shooting is performed through multiple cameras of auxiliary interactive devices and as a peripheral of alternative smart projectors, it is necessary to create a virtual space coordinate system with the distance and portrait size that the auxiliary interactive device can recognize, and bind it through a high-speed wireless network connection such as Star Flash. In addition, the relatively clear human activity space that the camera can recognize is mapped to the coordinate information of the virtual space. The gesture action information and posture information attached to the coordinate information are transmitted to the projector device (target interactive device), and according to the converted operation instructions, flexible somatosensory control of the application window is realized, which can realize some application operations, such as somatosensory control of games, and add some sports-type gameplay of full-body somatosensory operation for upgraded mobile games.
[0053] Based on the above embodiments, Figure 3 A schematic diagram of a method for displaying an auxiliary interactive interface provided by an exemplary embodiment of the present application is shown. In this embodiment, step 201 is replaced by at least one of step 201-1, step 201-2 and step 201-3: Step 201-1, when the operation interface is a command line interface, display an auxiliary interaction interface including the operation interface and a first interaction control for responding to a voice interaction operation.
[0054] Exemplarily, the operation interface of the target interactive device is displayed in the auxiliary interactive interface, and the operation interface is a command line interface. The first interactive control for responding to the voice interactive operation is also displayed in the auxiliary interactive interface. The auxiliary interactive device has a voice acquisition device for collecting voice interactive operations. When the voice acquisition device collects the corresponding voice interactive operation, the first interactive control responds to the voice interactive operation and feeds back the corresponding response screen. For example, the text content corresponding to the voice interactive operation is displayed. For example, the operation interface is divided into a first display area and a second display area. The interactive interface of the target operating device is displayed in the first display area, and the window content of the auxiliary interactive operation is displayed in the second display area. For the first display area, when auxiliary prompts are needed, the auxiliary content is displayed in a superimposed layer. If necessary, the overall generated window snapshot and voice conversation record are stored in the auxiliary display area, or stored for logging.
[0055] Step 201 - 2 , when the operation interface is a graphical user interface, displaying an auxiliary interaction interface including the operation interface and a first interaction control for responding to a somatosensory interaction operation.
[0056] Exemplarily, the operation interface of the target interactive device is displayed in the auxiliary interactive interface, and the operation interface is a graphical user interface. The corresponding interface element in the operation interface serves as the first interactive control. The auxiliary interactive device has a motion acquisition device for collecting somatosensory interactive operations. When the motion acquisition device collects the corresponding somatosensory interactive operations, the first interactive control responds to the somatosensory interactive operations and feeds back the corresponding response screen. For example, a screen showing the first interactive control moving, clicking, and / or page jumping following the somatosensory interactive operations is displayed.
[0057] Step 201-3, when the operation interface is a multimedia user interface, an auxiliary interaction interface including the operation interface and a first interaction control for responding to voice interaction operations and somatosensory interaction operations is displayed.
[0058] Exemplarily, the operation interface of the target interactive device is displayed in the auxiliary interactive interface, and the operation interface is a multimedia user interface. The corresponding interface elements in the operation interface serve as the first interactive controls, and the first interactive controls are additionally displayed. The auxiliary interactive device has a voice collection device and a motion collection device for collecting somatosensory interactive operations. When the motion collection device collects the corresponding somatosensory interactive operation, the first interactive control responds to the somatosensory interactive operation and feeds back a corresponding response screen. When the voice collection device collects the corresponding voice interactive operation, the first interactive control responds to the voice interactive operation and feeds back a corresponding response screen.
[0059] In some embodiments, the above step S202 specifically includes: in response to the somatosensory interaction operation collected by the motion acquisition device, extracting the somatosensory motion feature information in the somatosensory interaction operation; and / or in response to the voice interaction operation collected by the voice acquisition device, extracting the voice feature information in the voice interaction operation.
[0060] The auxiliary interaction device may be a device that collects somatosensory interaction operations through a motion acquisition device, obtains images of users performing somatosensory operations, identifies key points of the human body in the obtained images, extracts characteristic parts including joints, head, hands, etc. of the human body, estimates the posture of the human body using a mathematical model or algorithm, matches the estimated posture with a predefined posture template, thereby identifying the specific posture or action of the human body, extracts key gestures or directly transmits posture data, and calculates the somatosensory action feature information in the somatosensory interaction operation. The auxiliary interaction device may also be a device that collects voice interaction operations through a voice acquisition device, obtains the sound signal of the user performing the voice interaction operation, converts the sound signal from the time domain to the frequency domain, extracts the characteristic values of each frame of sound, such as frequency, energy, etc., and then sends these characteristic values to a neural network for training to predict the pronunciation rules of speech. A deep neural network is trained through a large amount of speech data to obtain a model that can predict the most likely pronunciation based on the characteristics of the speech, decodes the input speech based on the sound model and the language model, obtains the most likely text result, and obtains the speech feature information in the voice interaction operation.
[0061] In some embodiments, the human-computer interaction method also includes: identifying interface elements in the auxiliary interaction interface; displaying visual prompt content explaining the interface elements in the auxiliary interaction interface, or broadcasting voice prompt content explaining the interface elements.
[0062] Displaying visual prompts for explaining interface elements in the auxiliary interaction interface is done by first acquiring interface elements in the auxiliary interaction interface, performing visual entity recognition, and further identifying and feature analysis based on the UI and icon data sets of the interface construction of the auxiliary interaction interface, especially for user trials and cut-pictures of installed applications, and improving the RAG service interface of the UI, which can accurately identify interface elements of the auxiliary interaction interface for structured functional descriptions and generate visual prompts for explaining interface elements. For example, for necessary buttons, it is necessary to acquire interface content, acquire the corresponding element characteristics, and obtain a function that conforms to conventional visual button design, and generate visual prompts for explaining interface elements based on interface content and specific functions, such as the finger used for "clicking" and the gesture diagram layer of "swiping up". In specific implementation, the visual prompts for explaining interface elements can be a sidebar opened in the auxiliary interaction interface to prompt users of possible results when performing interactive operations.
[0063] The voice prompt content explaining the interface elements is broadcasted, specifically, after the interface elements of the auxiliary interactive interface are accurately identified and the structured functional description is performed, the voice prompt content explaining the interface elements is generated and broadcasted. In a specific implementation, the voice prompt content explaining the interface elements can be broadcasted by interactively asking the user's needs and broadcasting the voice prompt content explaining the interface elements.
[0064] Based on the above embodiments, Figure 4 The flowchart of step S203 provided by an exemplary embodiment of the present application is shown. In this embodiment, step S203 includes steps S401 to S403.
[0065] Step S401: Generate corresponding interactive control instructions according to the operation feature information.
[0066] Step S402: Map the interactive control instruction to the target interactive device according to a preset mapping relationship, so that the target interactive device executes the device behavior specified by the interactive operation.
[0067] Step S403: after the target interactive device executes the device behavior, the auxiliary interactive interface is updated.
[0068] In some embodiments, the human-computer interaction method also includes: obtaining device information of the target interactive device; the device information includes the device interaction type, device status and executable device behavior of the target interactive device; constructing a first mapping relationship and a second mapping relationship based on the device information; the first mapping relationship describes the mapping relationship between the operation feature information and the interactive control instructions, and the second mapping relationship describes the mapping relationship between the device behavior and the interactive control instructions.
[0069] Constructing the first mapping relationship specifically involves determining the interactive operations that the user can perform on the auxiliary interactive interface according to the device interaction type of the target interactive device. For example, if the operating interface of the target interactive device needs to be operated by a cursor, it can be determined that the user can control the cursor in the operating interface of the target interactive device through the corresponding interactive operation. After determining the interactive operations that the user can perform on the auxiliary interactive interface, a mapping relationship data set between the operation feature information of the interactive operation and the interactive control instruction is created and saved in the memory of the auxiliary interactive device, so that the auxiliary interactive device can generate the corresponding interactive control instruction according to the first mapping relationship after detecting the interactive operation. Constructing the second mapping relationship specifically involves determining the interactive control instruction that the auxiliary interactive device can convey to the target interactive device according to the device state and executable device behavior of the target interactive device. For example, when the target interactive device is an air conditioner, the auxiliary interactive device can send an interactive control instruction for adjusting the temperature to the target interactive device. After determining the interactive control instruction that the auxiliary interactive device can convey to the target interactive device, a mapping relationship data set between the device behavior and the interactive control instruction is created and saved in the memory of the auxiliary interactive device, so that the target interactive device can execute the corresponding device behavior after receiving the interactive control instruction sent by the target interactive device.
[0070] See also Figure 5 The embodiment of the present application also provides a human-computer interaction device, which can implement the above human-computer interaction method, and the device includes: The first module 501 is used to display an auxiliary interaction interface according to the device interaction type of the target interaction device; the auxiliary interaction interface is used to display the device state of the target interaction device, the target interaction device is used to execute a device behavior, and the device behavior is used to change the device state of the target interaction device; The second module 502 is configured to extract operation feature information in response to an interaction operation on the auxiliary interaction interface; The third module 503 is used to interact with the target interactive device in response to the operation characteristic information, so that the target interactive device executes the device behavior specified by the interactive operation.
[0071] The specific implementation of the human-computer interaction device is substantially the same as the specific implementation of the above-mentioned human-computer interaction method, and will not be described in detail herein.
[0072] Figure 6 It is a block diagram of an electronic device according to an exemplary embodiment.
[0073] Refer to the following Figure 6 The electronic device 600 according to this embodiment of the present disclosure is described. Figure 6The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0074] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.
[0075] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present disclosure described in the above human-computer interaction method section of this specification.
[0076] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .
[0077] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0078] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0079] The electronic device 600 may also communicate with one or more external devices 600' (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0080] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the human-computer interaction method described above is implemented.
[0081] The human-computer interaction method, apparatus, device and storage medium provided in the embodiments of the present application utilize an auxiliary interaction device to communicate and interact with a target interaction device. The auxiliary interaction device can display an auxiliary interaction interface for displaying the device status of the target interaction device according to the device interaction type of the target interaction device. When a user performs an interactive operation on the auxiliary interaction interface, the auxiliary interaction device can extract the operation feature information in the interactive operation in response to the interactive operation on the auxiliary interaction interface, and interact with the target interaction device in response to the operation feature information, so that the target interaction device performs the device behavior specified by the interactive operation. Since an auxiliary interaction device that can adapt to complex business scenarios is used to provide a human-computer interaction scenario, the auxiliary interaction device generates an auxiliary interaction interface according to the device status of the target interaction device for the user to perform interactive operations, so that the user can perform human-computer interaction with the target interaction device through the auxiliary interaction device that can adapt to complex business scenarios, thereby improving the convenience of human-computer interaction with old devices.
[0082] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation of the present disclosure.
[0083] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0084] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.
[0085] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.
[0086] The exemplary embodiments of the present disclosure are specifically shown and described above. It should be understood that the present disclosure is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present disclosure is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.
Claims
1. A human-computer interaction method, applied to an auxiliary interaction device, characterized in that: include: Displaying an auxiliary interaction interface according to the device interaction type of the target interaction device; The auxiliary interaction interface is used to display the device status of the target interaction device, the target interaction device is used to execute device behavior, and the device behavior is used to change the device status of the target interaction device; In response to an interactive operation on the auxiliary interactive interface, extracting operation feature information in the interactive operation; In response to the operation characteristic information, interact with the target interaction device so that the target interaction device executes the device behavior specified by the interaction operation.
2. The human-computer interaction method according to claim 1, characterized in that: The displaying of the auxiliary interaction interface according to the device interaction type of the target interaction device includes: When the target interactive device has an operation interface, displaying the auxiliary interactive interface including the operation interface of the target interactive device and a first interactive control for responding to the interactive operation; the first interactive control is generated according to the interface element of the operation interface; When the target interactive device does not have an operation interface, the auxiliary interactive interface including a device status screen and a second interactive control for responding to the interactive operation is displayed; the device status screen is used to simulate the device status of the target interactive device.
3. The human-computer interaction method according to claim 2, characterized in that: When the target interactive device has an operation interface, displaying the auxiliary interactive interface including the operation interface of the target interactive device includes at least one of the following: When the operation interface is a command line interface, displaying the auxiliary interaction interface including the operation interface and the first interaction control for responding to the voice interaction operation; When the operation interface is a graphical user interface, displaying the auxiliary interaction interface including the operation interface and the first interaction control for responding to the somatosensory interaction operation; When the operation interface is a multimedia user interface, the auxiliary interaction interface including the operation interface and the first interaction control for responding to voice interaction operations and somatosensory interaction operations is displayed.
4. The human-computer interaction method according to claim 1, characterized in that: The extracting, in response to the interactive operation on the auxiliary interactive interface, the operation feature information in the interactive operation includes: In response to the somatosensory interaction operation collected by the motion collection device, the somatosensory action feature information in the somatosensory interaction operation is extracted; and / or in response to the voice interaction operation collected by the voice collection device, the voice feature information in the voice interaction operation is extracted.
5. The human-computer interaction method according to claim 1, characterized in that: The method further comprises: Identifying interface elements in the auxiliary interaction interface; Display visual prompt content explaining the interface elements in the auxiliary interaction interface, or broadcast voice prompt content explaining the interface elements.
6. The human-computer interaction method according to claim 1, characterized in that: The step of interacting with the target interactive device in response to the operation characteristic information so that the target interactive device executes the device behavior specified by the interactive operation includes: generating corresponding interactive control instructions according to the operation characteristic information; According to a preset mapping relationship, the interactive control instruction is mapped to the target interactive device, so that the target interactive device executes the device behavior specified by the interactive operation; After the target interactive device executes the device behavior, the auxiliary interactive interface is updated.
7. The human-computer interaction method according to claim 1, characterized in that: The method further comprises: Acquire device information of the target interactive device; the device information includes the device interaction type, device status, and executable device behavior of the target interactive device; According to the device information, a first mapping relationship and a second mapping relationship are constructed; the first mapping relationship describes the mapping relationship between the operation feature information and the interactive control instruction, and the second mapping relationship describes the mapping relationship between the device behavior and the interactive control instruction.
8. A human-computer interaction device, characterized in that: include: The first module is used to display an auxiliary interaction interface according to the device interaction type of the target interaction device; The auxiliary interaction interface is used to display the device status of the target interaction device, the target interaction device is used to execute device behavior, and the device behavior is used to change the device status of the target interaction device; A second module is used to extract operation feature information in response to an interaction operation on the auxiliary interaction interface; The third module is used to interact with the target interactive device in response to the operation characteristic information, so that the target interactive device executes the device behavior specified by the interactive operation.
9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the human-computer interaction method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the human-computer interaction method according to any one of claims 1 to 7 is implemented.