Interaction method and device of terminal device

Abstract

The present application provides an interaction method and apparatus for a terminal device, which can improve the response quality of the terminal device, upgrade the companionship for the user, and enhance the active interaction effect. The method obtains the output data of the terminal device and / or the user multimodal data, wherein the user multimodal data includes at least one of voice data, image data, text data and touch data for the terminal device; determines the user behavior category based on the user multimodal data including at least one of voice data, image data, text data and touch data for the terminal device, the user behavior category includes the user complaint category about the terminal device, the user online class category, the user negative emotion category and / or the user behavior sensitivity category; and then actively interacts with the user based on the user behavior category.

Description

Technical Field

[0001] The present application relates to the field of Internet of Things technology, and in particular to an interaction method and apparatus for terminal devices. Background Art

[0002] Currently, passive user interaction with devices no longer meets people's needs. This means that devices only respond after users ask questions, making it difficult for them to perceive users' emotions and effectively address them. Therefore, proactive interaction is a key trend in the development of devices. Just like the give-and-take of human interaction, devices that proactively communicate with users can create a more trustworthy and intimate experience through appropriate proactive interaction.

[0003] It is worth studying how to further improve the effect of active interaction of terminal devices and enhance user satisfaction. Summary of the Invention

[0004] The present application provides an interactive method and apparatus for a terminal device, which can improve the effect of active interaction of the terminal device, thereby improving user satisfaction.

[0005] In a first aspect, an embodiment of the present application provides an interaction method of a terminal device, including:

[0006] Get the user's voice data;

[0007] Acquiring user's voice text data according to the user's voice data;

[0008] determining, based on the user's voice and text data, a user behavior category including a user complaint category regarding the terminal device;

[0009] determining, according to the user's complaint category regarding the terminal device, a first event matching the complaint category, the first event being a voice interaction or an application failure;

[0010] Actively interact with the user based on the first event.

[0011] Using this approach, the terminal device determines the user's behavior category, including the type of complaints about the terminal device, based on the user's voice data. Based on the type of complaints, the device then determines whether the event causing the complaint was a voice interaction or an application failure, thereby determining how to proactively interact with the user. This allows for rapid improvement of responses that may have caused complaints or addressing application failures, further improving the quality of the terminal device's responses and continuously enhancing user satisfaction.

[0012] In a second aspect, an embodiment of the present application provides an interactive device of a terminal device, including:

[0013] A data acquisition module is used to acquire user voice data;

[0014] A data processing module, configured to obtain user's voice text data based on the user's voice data;

[0015] A user behavior category determination module, configured to determine a user behavior category including a user complaint category regarding the terminal device based on the user's voice and text data;

[0016] a complaint event determination module, configured to determine, based on the category of the user's complaint about the terminal device, a first event matching the category of the complaint, wherein the first event is a voice interaction or an application failure;

[0017] An active interaction module is used to actively interact with the user based on the first event.

[0018] In a third aspect, an embodiment of the present application provides an interaction method of a terminal device, including:

[0019] Acquiring multimodal data of the user or touch data of the terminal device;

[0020] determining, based on the multimodal data of the user or the touch data of the terminal device, that the user behavior category includes a negative user emotion category;

[0021] Displaying a visual interface that matches the negative emotion category of the user, and / or determining and playing a voice that matches the negative emotion category of the user;

[0022] After a first preset time has passed, determining whether the current user behavior category is a user negative emotion category;

[0023] If the current user behavior category is a negative user emotion category, a first prompt message is sent to a contact associated with the user, where the first prompt message is used to remind the user of the emotional state.

[0024] Using the above approach, based on the user's multimodal data or touch data from the terminal device, the user behavior is classified as negative, and a matching visual interface or audio message is displayed to comfort the user. Furthermore, after a first preset time, the user's emotion is re-identified to determine whether it is negative. If it is still negative, family or friends can be promptly alerted to help improve the user's mood as quickly as possible, further enhancing the terminal device's proactive interaction and thus improving user satisfaction.

[0025] In a fourth aspect, an embodiment of the present application provides an interactive device of a terminal device, including:

[0026] A data acquisition module, configured to acquire multimodal data of the user or touch data of the terminal device;

[0027] A user behavior category determination module, configured to determine, based on the multimodal data of the user or the touch data of the terminal device, that the user behavior category includes a negative user emotion category;

[0028] An active interaction module, configured to display a visual interface that matches the negative emotion category of the user, and / or determine and play a voice that matches the negative emotion category of the user;

[0029] The user behavior category determination module is further configured to determine, after a first preset time, whether the current user behavior category is a user emotion negative category;

[0030] The active interaction module is further configured to send a first prompt message to a contact associated with the user if the current user behavior category is a negative user emotion category, wherein the first prompt message is used to remind the user of the emotional state.

[0031] In a fifth aspect, an embodiment of the present application provides an interaction method of a terminal device, including:

[0032] Acquiring output data of the terminal device and / or image data of the user;

[0033] Determining a user behavior category including an online course category of the user based on the output data of the terminal device and / or the image data of the user;

[0034] Determining that the user is a candidate for taking an online course;

[0035] Determine the type of the target time period, where the target time period is either online class or rest;

[0036] determining, based on the output data of the terminal device and / or the image data of the user, an action of the user within the target time period, the action of the user including a first action and a second action, the first action representing an action that the user is not allowed to perform during the online class, and the second action representing an action that the user is allowed to perform during the online class;

[0037] Actively interact with the user based on the user's actions and the type of the target period.

[0038] By adopting the above method, after determining that the user is an online course target, the user's online course status can be accurately determined by combining the type of target time period and the user's actions, thereby actively interacting with the user, which can further improve the effect of supervising the user's online course.

[0039] In a sixth aspect, an embodiment of the present application provides an interactive device of a terminal device, including:

[0040] Data acquisition means, for acquiring output data of the terminal device and / or image data of the user;

[0041] A user behavior category determination module, configured to determine a user behavior category including a user online course category based on output data of the terminal device and / or image data of the user;

[0042] An active interaction module, configured to determine that the user is a candidate for an online course;

[0043] The active interaction module is further used to determine the type of the target time period, which is either taking an online class or taking a break;

[0044] The active interaction module is further configured to determine an action of the user within the target time period based on the output data of the terminal device and / or the image data of the user, wherein the action of the user includes a first action and a second action, wherein the first action represents an action that the user is not allowed to perform during the online class, and the second action represents an action that the user is allowed to perform during the online class;

[0045] The active interaction module is further configured to actively interact with the user based on the user's actions and the type of the target time period.

[0046] In a seventh aspect, an embodiment of the present application provides an interaction method of a terminal device, including:

[0047] Acquiring output data of the terminal device and / or image data of the user;

[0048] determining a user behavior category including a user behavior sensitive category based on the output data of the terminal device and / or the image data of the user;

[0049] Displaying a visual interface that matches the user behavior sensitive category, and / or determining and playing a voice that matches the user behavior sensitive category;

[0050] Determining that the attribute information of the user meets a first preset condition;

[0051] A third prompt message is sent to a contact associated with the user, where the third prompt message is used to report the sensitive behavior of the user.

[0052] Using this approach, when user behavior categories include sensitive ones, reminders or corrections can be issued, ensuring the healthy physical and mental development of underage users. Furthermore, sensitive behaviors of underage users can be reported to their guardians, strengthening supervision of underage users and improving the effectiveness of proactive interaction with terminal devices, thereby increasing user satisfaction with the terminal devices.

[0053] In an eighth aspect, an embodiment of the present application provides an interactive device of a terminal device, including:

[0054] A data acquisition module, configured to acquire output data of the terminal device and / or image data of the user;

[0055] A user behavior category determination module, configured to determine a user behavior category including a user behavior sensitive category based on the output data of the terminal device and / or the image data of the user;

[0056] An active interaction module, configured to display a visual interface that matches the user behavior sensitive category, and / or determine and play a voice that matches the user behavior sensitive category;

[0057] The active interaction module is further configured to determine that the attribute information of the user satisfies a first preset condition;

[0058] The active interaction module is further configured to send a third prompt message to a contact associated with the user, where the third prompt message is used to report the sensitive behavior of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0060] Figure 1 An exemplary system architecture of the interaction method and the interaction apparatus of the terminal device of the present application is shown;

[0061] Figure 2 shows a hardware configuration block diagram of a terminal device 200 according to an exemplary embodiment;

[0062] Figure 3 An exemplary system architecture diagram of a terminal device whose operating system is the Android system is shown;

[0063] Figure 4 A schematic diagram of an interactive network architecture of a terminal device provided in an embodiment of the present application;

[0064] Figure 5 This is a schematic flow chart of an example of an interaction method of a terminal device provided in an embodiment of the present application;

[0065] Figure 6 This is a schematic diagram of an active interaction effect provided in an embodiment of the present application;

[0066] Figure 7 This is a schematic flow chart of another example of an interaction method of a terminal device provided in an embodiment of the present application;

[0067] Figure 8 This is a schematic diagram of an active interaction effect provided in an embodiment of the present application;

[0068] Figure 9 This is a schematic diagram of a first prompt information provided in an embodiment of the present application;

[0069] Figure 10 This is a schematic flow chart of another example of an interaction method of a terminal device provided in an embodiment of the present application;

[0070] Figure 11 This is a schematic diagram of an active interaction effect provided in an embodiment of the present application;

[0071] Figure 12 This is another example of an active interaction effect diagram provided in an embodiment of the present application;

[0072] Figure 13 This is a schematic diagram of a second prompt information provided in an embodiment of the present application;

[0073] Figure 14 This is another schematic diagram of the second prompt information provided in an embodiment of the present application;

[0074] Figure 15 This is another schematic flowchart of an interaction method for a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose and implementation of this application clearer, the exemplary implementation of this application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only part of the embodiments of this application, not all of the embodiments.

[0076] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0077] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or sequence, unless otherwise noted. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances.

[0078] Figure 1 The following shows an exemplary system architecture of the interactive method and interactive device of the terminal device to which the present application can be applied. Figure 1As shown, 10 is a server and 200 is a terminal device, which exemplarily includes (smart TV 200a, mobile device 200b, smart speaker 200c).

[0079] In this application, the server 10 and the terminal device 200 communicate data via various communication methods. The terminal device 200 may be connected to a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 10 may provide various content and interactions to the terminal device 200. For example, the terminal device 200 and the server 10 may send and receive information, as well as receive software program updates.

[0080] The server 10 can be a server that provides various services, such as a backend server that supports audio data collected by the terminal device 200. The backend server can analyze and process the received audio data and other data, and feed back the processing results (such as endpoint information) to the terminal device. The server 10 can be a server cluster or multiple server clusters, and can include one or more types of servers.

[0081] The terminal device 200 can be hardware or software. When the terminal device 200 is hardware, it can be various electronic devices with sound collection functions, including but not limited to smart speakers, smart phones, TVs, tablets, e-book readers, smart watches, players, computers, AI devices, robots, smart vehicles, etc. When the terminal devices 200, 201, and 202 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, for providing sound collection services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0082] It should be noted that the terminal device interaction method provided in the embodiment of the present application can be executed by the server 10, or by the terminal device 200, or by both the server 10 and the terminal device 200, and this application does not limit this.

[0083] Figure 2 FIG. 2 shows a hardware configuration block diagram of the terminal device 200 according to an exemplary embodiment. Figure 2 The terminal device 200 shown includes at least one of a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, an audio processor, a graphics processor, RAM, ROM, and first to nth interfaces for input / output.

[0084] The display 260 includes a display screen component for presenting images, and a driving component for driving image display, a component for receiving image signals output from a controller, and a component for displaying video content, image content, and a menu control interface and a user control UI interface.

[0085] The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

[0086] Communicator 220 is a component used to communicate with external devices or servers using various communication protocols. For example, the communicator may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chip, a near-field communication protocol chip, and an infrared receiver. Terminal device 200 can establish communication with server 10 via communicator 220 to send and receive control signals and data signals.

[0087] The user interface can be used to receive external control signals.

[0088] Detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or detector 230 includes an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or detector 230 includes a sound collector, such as a microphone, for receiving external sounds.

[0089] The sound collector can be a microphone, also known as a "microphone" or "microphone", which can be used to receive the user's voice and convert the sound signal into an electrical signal. The terminal device 200 can be provided with at least one microphone. In other embodiments, the terminal device 200 can be provided with two microphones, which can not only collect sound signals but also implement noise reduction functions. In other embodiments, the terminal device 200 can also be provided with three, four, or more microphones to collect sound signals, reduce noise, identify the sound source, implement directional recording functions, etc.

[0090] In addition, the microphone may be built into the terminal device 200, or the microphone may be connected to the terminal device 200 by wired or wireless means. Of course, the embodiment of the present application does not limit the position of the microphone on the terminal device 200. Alternatively, the terminal device 200 may not include a microphone, that is, the microphone is not provided in the terminal device 200. The terminal device 200 may be connected to an external microphone (also referred to as a microphone) through an interface (such as a USB interface 130). The external microphone may be fixed to the terminal device 200 by an external fixing member (such as a camera holder with a clip).

[0091] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the terminal device 200.

[0092] Exemplarily, the controller includes at least one of a central processing unit (CPU), an audio processor, a graphics processing unit (GPU), RAM Random Access Memory (RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus, etc.

[0093] Figure 3 An exemplary system architecture diagram of a terminal device whose operating system is an Android system is shown. In some examples, the operating system of the terminal device is an Android system as an example. Figure 3 As shown, the smart TV 200 - 1 can be logically divided into an application layer (abbreviated as “application layer”) 21 , a kernel layer 22 and a hardware layer 23 .

[0094] Among them, such as Figure 3 As shown, the hardware layer may include Figure 2 The controller 250, communicator 220, detector 230, etc. are shown. The application layer 21 includes one or more applications. The applications can be system applications or third-party applications. For example, the application layer 21 includes a voice recognition application that can provide a voice interaction interface and services for connecting the smart TV 200-1 to the server 10.

[0095] The kernel layer 22 serves as a software middleware between the hardware layer and the application layer 21 and is used to manage and control hardware and software resources.

[0096] In some examples, the kernel layer 22 includes a detector driver, which is used to send the voice data collected by the detector 230 to the voice recognition application. For example, when the voice recognition application in the terminal device 200 is started and a communication connection is established between the terminal device 200 and the server 10, the detector driver is used to send the user input voice data collected by the detector 230 to the voice recognition application. The voice recognition application then sends query information containing the voice data to the intent recognition module 102 in the server. The intent recognition module 102 is used to input the voice data sent by the terminal device 200 into the intent recognition model.

[0097] To clearly illustrate the embodiments of this application, Figure 4A speech recognition network architecture provided in an embodiment of the present application is described.

[0098] See also Figure 4 , Figure 4 A schematic diagram of an interactive network architecture of a terminal device provided in an embodiment of the present application. Figure 4 In the example, the terminal device is used to receive input information and output the processing results of the information. The speech recognition module is deployed with a speech recognition service for recognizing audio as text; the semantic understanding module is deployed with a semantic understanding service for semantically parsing text; the business management module is deployed with a business instruction management service for providing business instructions; the language generation module is deployed with a language generation service (NLG) for converting instructions for the terminal device to execute into text language; the speech synthesis module is deployed with a speech synthesis (TTS) service for processing the text language corresponding to the instruction and sending it to the speaker for broadcast. In one embodiment, Figure 4 The illustrated architecture may include multiple physical service devices deployed with different business services, or one or more physical service devices may integrate one or more functional services.

[0099] In some embodiments, the following Figure 4 The process of processing information input into a terminal device in the illustrated architecture is described by way of example, taking the information input into the terminal device as a query statement inputted via voice as an example:

[0100] [Speech Recognition]

[0101] After receiving a query statement input via voice, the terminal device may perform noise reduction processing and feature extraction on the audio of the query statement. The noise reduction processing here may include steps such as removing echoes and ambient noise.

[0102] [Semantic Understanding]

[0103] Using acoustic models and language models, the natural language understanding of the identified candidate text and the associated context information is performed, and the text is parsed into structured, machine-readable information, business domain, intent, word slots and other information to express semantics, etc. The executable intent is obtained and the intent confidence score is determined. The semantic understanding module selects one or more candidate executable intents based on the determined intent confidence score.

[0104] [Business Management]

[0105] Based on the semantic analysis results of the query statement text, the semantic understanding module sends query instructions to the corresponding business management module to obtain the query results given by the business service, and executes the actions required to "complete" the user's final request, and feeds back the device execution instructions corresponding to the query results.

[0106] [Language Generation]

[0107] Natural language generation (NLG) is configured to generate language text from information or instructions. Specifically, it can be divided into chat-type, task-based, knowledge question-and-answer (Q&A) and recommendation-based types. Among them, NLG in chat-type conversations is to perform intent recognition and sentiment analysis based on the context, and then generate open replies; in task-based conversations, conversation replies need to be generated based on learned strategies. Generally, replies include clarifying needs, guiding users, asking questions, confirming, and concluding conversations; in knowledge question-and-answer conversations, the knowledge required by users (knowledge, entities, fragments, etc.) is generated based on question type recognition and classification, information retrieval, or text matching; in recommendation-type conversation systems, interest matching and candidate recommendation content ranking are performed based on the user's hobbies, and then recommended content is generated for the user.

[0108] [Speech Synthesis]

[0109] The speech synthesis module is configured to present speech output to the user. The speech synthesis processing module synthesizes speech output based on the text provided by the digital assistant. For example, the generated dialogue response is in the form of a text string. The speech synthesis module converts the text string into audible speech output.

[0110] It should be noted that Figure 4 The architecture shown is only an example and does not limit the scope of protection of this application. In the embodiments of this application, other architectures can also be used to implement similar functions. For example, all or part of the above process can be completed by a smart terminal, which will not be described in detail here.

[0111] Compared to passive human interaction, where the user asks and the device responds, proactive interaction can identify underlying meanings like a human and proactively adjust responses to more appropriate ones. However, the effectiveness of proactive interaction currently needs to be further improved.

[0112] To improve the effect of proactive interaction, this application provides an example of a terminal device interaction method for scenarios where users complain about terminal devices. The method 300 includes:

[0113] S310: Acquire user's voice data.

[0114] Exemplarily, the terminal device obtains the user's voice data through a microphone.

[0115] S320: Acquire the user's voice text data according to the user's voice data.

[0116] The user's voice data is audio data. The audio data is converted into the user's voice text data, and the text in the voice data can be recognized through a neural network model.

[0117] S330: Determine, based on the user's voice and text data, a user behavior category including a user complaint category about the terminal device.

[0118] Among them, the user complaint category about the terminal device indicates that the user is dissatisfied with the response of the terminal device when using the terminal device.

[0119] For example, the chat reply from the terminal device does not solve the user's problem or makes the user feel unhappy.

[0120] Another example is when a user is using a certain function on a terminal device and the terminal device malfunctions, resulting in a poor user experience. For example, when a user is using an app on a terminal device and the app crashes, the user complains: "What the hell is this (specific name of the app)?"

[0121] S340: Determine a first event that matches the category of the user's complaint about the terminal device.

[0122] Among them, the first event is voice interaction, that is, the content of the voice response made by the terminal device to the voice interaction makes the user dissatisfied, thereby causing the user to complain about the terminal device, or the first event is an application (APP) failure, that is, the failure that occurs when the user uses the terminal device to operate the application makes the user dissatisfied.

[0123] S350: Actively interact with the user based on the first event.

[0124] Exemplarily, a visual interface that matches the category of the user's complaint about the terminal device is displayed, and / or a voice that matches the category of the user's complaint about the terminal device is determined and played.

[0125] Among them, a visual interface matching the apologetic smile mode is used to express the apology of the terminal device, such as using the bowing action of an animated character to express apology.

[0126] The voice matching the apologetic smile mode is used to express the apology of the terminal device, for example, playing "XX bows to you" in a gentle tone.

[0127] In the above method, the terminal device determines the user's behavior category, including the category of the user's complaint about the terminal device, based on the user's voice data. Then, based on the category of the user's complaint, it determines whether the event causing the user's complaint is a voice interaction or an application failure, thereby determining a proactive method for interacting with the user. This allows for rapid improvement of responses that may have caused complaints or handling of application failures, further improving the quality of the terminal device's responses and continuously enhancing user satisfaction.

[0128] In order to improve the effect of active interaction, for scenarios where users are in a negative mood, this application also provides an example of an interaction method for a terminal device. The method 400 includes:

[0129] S410: Acquire multimodal data of the user.

[0130] The user multimodal data includes at least one of voice data, image data, text data and touch data for the terminal device.

[0131] Terminal devices with multimodal data acquisition capabilities are usually equipped with a variety of different receiving devices or sensors, such as cameras, microphones, touch screens, pressure sensors, distance sensors, infrared sensors, etc., to ensure the effective collection of multimodal data.

[0132] Exemplarily, multimodal data is multimodal data for a terminal device when a user wants to interact with the terminal device; or it is multimodal data generated when the user has no intention of interacting with the terminal device and is only active in the spatial environment where the terminal device is located.

[0133] The voice data in the multimodal data can be obtained through the voice acquisition device of the terminal device, such as a microphone; the image data can be obtained through the image acquisition device of the terminal device, such as a camera; the text data is the data input by the user into the terminal device; the touch data can be collected through corresponding sensors, such as the pressure sensor of the touch screen, the pressure sensor of the touch button, or the pressure sensor at the beginning, when the interactive object touches or presses the terminal device.

[0134] S420: Determine, based on the multimodal data of the user, that the user behavior category includes a negative user emotion category.

[0135] Among them, the negative category of user emotions represents the user's emotional states such as sadness, grief, anger, tension, anxiety, pain, fear, hatred, etc., for example:

[0136] (1) The user said: “I’m so annoyed,” which indicates that he is in a bad mood.

[0137] (2) The user speaks normally, but the tone is not good. For example, "I want to listen to music", but the tone is angry or sad. This also shows that the user is in a bad emotional state.

[0138] (3) If the user’s expression is angry or sad, it can also indicate that the user is in a bad emotional state.

[0139] (4) The user angrily slams the keyboard, expressing his anger.

[0140] The negative user emotion category also includes the user's negative emotions reflected when inputting text data into the terminal device. For example, the user uses the terminal device to search for content such as "What to do if you are sad after breaking up" on an APP, which can show that the user is in a negative mood.

[0141] S430 , displaying a visual interface that matches the negative emotion category of the user, and / or determining and playing a voice that matches the negative emotion category of the user.

[0142] Specifically, a visual interface matching the user's negative emotion category is displayed, and / or a voice message matching the user's negative emotion category is determined and played to comfort the user. Examples of visual interfaces matching the user's negative emotion category include animated images of cartoon characters patting the head for comfort. Voice messages matching the user's negative emotion category include other voice messages that can cheer the user up, such as "Don't be sad, let me play a song for you," played in a gentle tone.

[0143] S440: After a first preset time has passed, determine whether the current user behavior category is a user negative emotion category.

[0144] The first preset time can be customized according to user needs. Optionally, the default time is 30 minutes.

[0145] S450: If the current user behavior category is a negative user emotion category, a first prompt message is sent to a contact associated with the user, where the first prompt message is used to remind the user of the emotional state.

[0146] The first prompt message is used to remind the contact person of the user's emotional state. The contact person can be set according to actual needs, for example, the contact person can be a family member or friend of the user.

[0147] The first prompt information includes text or a dynamic image that reminds the user's family or friends to pay attention to the user's emotional state, and optionally also includes the user's mood curve record. The mood curve record allows family or friends to check the user's historical emotional state at any time.

[0148] Using the above approach, based on the user's multimodal data or touch data from the terminal device, the user behavior is classified as negative, and a matching visual interface or audio message is displayed to comfort the user. Furthermore, after a first preset time, the user's emotion is re-identified to determine whether it is negative. If it is still negative, family or friends can be promptly alerted to help improve the user's mood as quickly as possible, further enhancing the terminal device's proactive interaction and thus improving user satisfaction.

[0149] In order to improve the effect of active interaction, the present application also provides an example of an interaction method of a terminal device for a scenario where a user takes an online course. The method 500 includes:

[0150] S510: Acquire output data of a terminal device and / or image data of a user.

[0151] The output data of the terminal device includes voice data, image data, text data, etc. output by the terminal device, for example, pictures, videos, audio, text, etc. played by the terminal device according to user needs.

[0152] S520: Determine, based on the output data of the terminal device and / or the image data of the user, the user behavior category, including the user taking an online course category.

[0153] Among them, the categories of users taking online courses include scenarios where users cannot go to school due to force majeure and can only take online courses at home, or scenarios where users spontaneously conduct online education in order to improve themselves.

[0154] S530: Determine that the user is a target for taking an online course.

[0155] Specifically, there may be more than one user using the terminal device, and it is necessary to determine whether the user using the terminal device is a registered subject who needs to take an online course.

[0156] S540: Determine the type of the target time period, which may be taking an online class or taking a break.

[0157] Exemplarily, the target time period type is determined by the set online class time period.

[0158] For example, the online course playback time period is customized, and the time period of the online course live broadcast is not fixed.

[0159] Exemplarily, whether the target time period is an online class time period is determined by taking a screenshot of the display interface.

[0160] S550, determining the user's actions within the target time period based on the output data of the terminal device and / or the user's image data, the user's actions including a first action and a second action, the first action indicating the actions that the user is not allowed to do when taking an online class, and the second action indicating the actions that the user is allowed to do when taking an online class.

[0161] S560 , actively interacting with the user based on the user's action and the type of the target time period.

[0162] In this example, after determining that the user is an online course candidate, the user's online course status is accurately determined by combining the type of target time period and the user's actions, thereby actively interacting with the user, which can further improve the effect of supervising the user's online course.

[0163] In order to improve the effect of active interaction, for scenarios sensitive to user behavior, this application also provides an example of an interaction method of a terminal device. The method 600 includes:

[0164] S610: Obtain output data of a terminal device and / or user multimodal data.

[0165] The output data of the terminal device and the content of the user's multimodal data are described above and will not be repeated here.

[0166] S620: Determine, based on the output data of the terminal device and / or the user multimodal data, a user behavior category including a user behavior sensitive category.

[0167] The user behavior sensitive category includes sensitive behaviors performed by the user himself, or the user causing the terminal device to display sensitive text information or image data, play sensitive videos (including sensitive voice), and other content.

[0168] Among them, sensitive behaviors performed by the user himself, such as the user's words or actions, involve sensitive behaviors.

[0169] The user causes the terminal device to display sensitive text information or image data, or play sensitive videos (including sensitive voice).

[0170] In one example, sensitive behaviors can also be customized by users according to their needs. For example, in order to prevent their children from playing games and affecting their studies, parents can set "minors playing games" as a sensitive behavior to supervise minors.

[0171] S630: Display a visual interface that matches the user behavior sensitive category, and / or determine and play a voice that matches the user behavior sensitive category.

[0172] Among them, the visual interface that matches the sensitive category of user behavior includes a visual interface that prompts that the user behavior involves sensitive topics, or a visual interface that has been occluded.

[0173] Voices that match sensitive categories of user behavior include voices that indicate that the user's behavior involves sensitive topics.

[0174] S640: Determine whether the attribute information of the user meets a first preset condition.

[0175] In one example, the user's attribute information includes age, and the first preset condition is that the user is not older than 18 years old.

[0176] In one example, the user's attribute information includes height, and the first preset condition is that the height is no more than 1.2 meters.

[0177] S650: Send a third prompt message to the contact associated with the user, where the third prompt message is used to report the user's sensitive behavior.

[0178] Exemplarily, the third prompt information includes the user's statements, actions, and other behaviors involving sensitive content, or the text, video, or audio content displayed or played by the user using a terminal device.

[0179] In the above embodiment, when user behavior categories include sensitive categories, reminders or corrections can be provided, ensuring the healthy physical and mental development of underage users. Furthermore, sensitive behaviors of underage users can be reported to their guardians, strengthening supervision of underage users and improving the effectiveness of proactive interaction with the terminal device, thereby increasing user satisfaction with the terminal device.

[0180] Based on the method 300, this application introduces a terminal device interaction method based on the user's complaint category about the terminal device through a specific embodiment. Figure 5 This is a schematic flow chart of an example of an interactive method of a terminal device provided in an embodiment of the present application. Figure 5 As shown in method 300a:

[0181] S310a, obtaining user statements.

[0182] Specifically, the user's speech data is converted into speech text data.

[0183] S320a, determining the user behavior category.

[0184] Specifically, the terminal device determines the user behavior category based on the user's statement.

[0185] In one example, the method of determining the user behavior category based on the user's statement includes:

[0186] Method 1

[0187] Perform text classification on user statements to determine the user behavior category.

[0188] For example, common user statements are labeled 0 and 1, with 0 representing non-complaints and 1 representing complaints. A binary text classification model is trained using a convolutional neural network. The model is fed with the acquired user statements to obtain the text classification result. If the result is 1, it is considered a user complaint about the terminal device.

[0189] Method 2

[0190] Perform regular expression matching on user statements to determine the user behavior category.

[0191] For example, keywords representing user complaints about terminal devices, such as "dislike" and "angry," can be configured in the database. When these keywords appear in a user's statement, it is considered a complaint about the terminal device. Keywords for user complaints about terminal devices can be manually maintained, added, or deleted based on actual conditions, making it easier to expand.

[0192] Method 3

[0193] The user's statement is sentimentally scored to determine the user behavior category. The sentiment score is set for each keyword, and then the keywords in the user's statement are identified. The scores of all keywords in the user's statement are summed. User statements that meet the preset criteria are classified as complaints, indicating that the current user behavior category is a user complaining about the terminal device.

[0194] For example, we assign sentiment scores to keywords in user statements, such as "hate" and "angry" (-3 points), "like" (3 points), "very" and "too" (2 points), "smart" (2 points), "silly" and "dumb" (-2 points), "no" (-1 points), and "furiating to death" (-8 points). The sentiment threshold is set to 0, and a user statement with a sentiment score below 0 is considered a complaint.

[0195] Then the sentiment score of the user's statement "I really enjoy chatting with you" is: 2*3=6, which is greater than 0 and is not a complaint;

[0196] The sentiment score of the user statement "I hate chatting with you" is: 2*(-3)=-6, which is complaining;

[0197] User statement: "You make me so mad" has a sentiment score of -8, which is a complaint.

[0198] By analogy, when using sentiment scores to analyze user statements, the lower the score, the higher the level of user complaints, and it is necessary to prioritize proactive interaction with users.

[0199] Optionally, if the user behavior category does not include a user complaining about the terminal device category, a user with negative emotions category, a user taking an online course category, or a user with sensitive behavior category, the terminal device activates normal mode. If the user behavior category includes a user complaining about the terminal device category, method S340a is performed. Normal mode indicates that the terminal device does not interact with the user based on the aforementioned user behavior categories.

[0200] S330a: Determine a first event.

[0201] Specifically, if the user behavior category does not include the user complaining about the terminal device category, the terminal device starts the normal mode; if the user behavior category includes the user complaining about the terminal device category, a first event matching the user complaining about the terminal device category is determined.

[0202] S340a: The terminal device actively interacts with the user based on the first event.

[0203] In a possible implementation, the first event is an application failure, and the method S340a further includes:

[0204] The displayed visual interface includes an animated image expressing the apology of the terminal device, and / or plays a voice expressing the apology of the terminal device, and then closes the application program, and restarts the application program after a first preset time, such as Figure 5 S341a in.

[0205] In another example, the first event is a voice reply, and the method S340a further includes:

[0206] The voice response made by the terminal device to the voice interaction is shielded so that the voice response is not included in the candidate response set that matches the content of the voice interaction. The content of the voice response is then verified and the verified voice response content is added to the candidate response set.

[0207] The content of the voice reply may be manually verified, or the content of the voice reply may be automatically verified by the server.

[0208] Among them, voice interaction includes other voice interaction methods such as users asking voice questions to the terminal device, the terminal device making voice replies to the user, or users having voice chats with the terminal device.

[0209] In the above example, after identifying the first event that causes the user to complain about the terminal device, timely processing can further improve user satisfaction.

[0210] The following example illustrates voice interaction:

[0211] The content of the voice question and answer is, for example: Are you fat?

[0212] After verification, the content of the voice reply is, for example: I am cute even though I am chubby.

[0213] Optionally, verify the content of the voice reply and add the verified content of the voice reply to a candidate reply set, including:

[0214] The terminal device sends a reminder message, which is used to prompt the user to verify the voice reply. The voice reply is then verified based on the reminder message and inserted into the candidate reply set.

[0215] Optionally, the terminal device sends the reminder information to a device operated by the staff.

[0216] Optionally, the terminal device sends the reminder information to the server.

[0217] The following combination Figure 5 The following is an example of the active interaction process when the first event is a voice reply:

[0218] S342a, find the most recent chat reply and put it into the shielding database.

[0219] The most recent chat reply is the chat reply that causes dissatisfaction among the user.

[0220] Specifically, the sentence corresponding to the most recent chat reply is modified to "enable=false" and stored in the shielding database. The chat replies in the shielding database cannot be called by the search-based chat model.

[0221] S343a, shielding the database from sending reminder information.

[0222] Specifically, when the most recent chat reply is received in the shielding database, a reminder message is sent to remind the staff to manually verify the most recent chat reply.

[0223] S344a, manually verify the latest chat reply.

[0224] Specifically, the most recent chat reply is manually modified into a suitable chat reply, and its corresponding statement is modified to "enable=true", and then put into its corresponding chat candidate set.

[0225] The chat replies after manual verification can be called again by the retrieval-based chat model, which combines information such as user query statements or historical records to give appropriate voice replies.

[0226] By adopting the above method, after receiving a chat reply that causes user dissatisfaction, the shielding database reminds the staff to perform manual verification, so that the staff can correct the chat reply that causes user dissatisfaction in time, thereby improving user satisfaction with the terminal device.

[0227] Furthermore, before putting the verified voice reply content into the candidate reply set, if the user performs the above-mentioned voice question and answer again, the terminal device will play other voice replies in the candidate reply set that match the content of the voice question and answer.

[0228] For example, the user asked again: Are you fat?

[0229] The content of the terminal device's voice reply is, for example: A happy heart and a fat body, hehe (the second sentence in the original candidate reply set).

[0230] In the above method, while avoiding replies that cause user dissatisfaction from being used again by the terminal device, it can also ensure high-quality replies to the user's questions and answers, further improving the active interaction effect of the terminal device.

[0231] The terminal device identifies the user complaint through the user's statement and determines that the current user behavior category is the user's complaint category about the terminal device, and then displays a matching visual interface and / or determines and plays a matching voice. Figure 6 This is a schematic diagram of an active interaction effect provided by an embodiment of the present application. Figure 6 As shown, an animated image expressing apology, such as a bowing lady, is displayed, accompanied by a voice reply of "I apologize to you, I will change it immediately", and the voice is played in a gentle tone.

[0232] The terminal device then quickly retrieves the historical records and adds the sentences that caused the user to complain to the blocked database. When the user asks the same question again, the inappropriate sentence will not be retrieved.

[0233] For example, the expected effect is: (the voice response that caused user complaints has been added to the blocking database, but has not been fixed yet)

[0234] User: Are you fat?

[0235] Terminal: A happy heart makes a fat body, hehe. (Original top 2 sentence)

[0236] At the same time, maintenance personnel will immediately receive a notification that an inappropriate statement has been added to the blocked database. They can then immediately repair it and add it back to the candidate list. By default, the candidate statement has enable=true. While the statement is in the blocked database, enable=false. Once repaired, the modified statement has enable=true and can be used again by the search-based chat model.

[0237] For example, the expected effect is: (after manual verification)

[0238] User: Are you fat?

[0239] Terminal: I'm cute even though I'm chubby. (Manually edited and annotated sentences)

[0240] The above-mentioned scheme of the present application can identify the category of user complaints about the terminal device, and determine the first event that causes the user to complain. Based on the first event, it actively interacts with the user, soothes the user's dissatisfaction with the terminal device, and expresses the apology of the terminal device. Furthermore, it can also timely verify the voice reply content that causes the user to complain, improve the voice reply quality of the terminal device, and further improve the effect of active interaction of the terminal device, thereby improving the user's satisfaction with the terminal device.

[0241] Based on method 400, this application introduces a method for interacting with a terminal device based on the negative emotion category of the user through a specific embodiment. Figure 7This is a schematic flow chart of another example of an interaction method of a terminal device provided in an embodiment of the present application. Figure 7 As shown in method 400a:

[0242] S410a: Acquire user multimodal data.

[0243] The user multimodal data includes one or more of user voice (audio), user speech (text), user video (video) and user touch data on the terminal device.

[0244] S420a, determining the user behavior category.

[0245] Specifically, sentiment analysis is performed based on the user multimodal data to determine that the user behavior category includes a user negative emotion category.

[0246] For example, when multimodal user data includes user voice, the decibel level or tone of the user's voice is used to determine whether the user's behavior categorizes them as negative. Alternatively, the user's voice is passed to the backend for VAD speech semantic detection and deep noise reduction with Steam ASR (supporting emotion recognition) processing to convert the user's voice into text and identify the user's emotion. For example, a sad voice indicates a negative emotion.

[0247] For example, when multimodal user data includes user statements, the user statements are used as input to the emotion recognition model, and the user's emotions are then identified based on the output. For example, if the user statement includes keywords such as "sad" or "break up," which indicate negative user emotions, the model may use these keywords to identify the user's emotions.

[0248] For another example, when user multimodal data includes user videos, the neural network model can identify user emotions by recognizing actions and expressions in the user videos. For example, actions or expressions indicating negative emotions, such as crying for more than one minute, can be used.

[0249] For another example, when the user multimodal data includes user touch data on the terminal device, the user's emotions are identified through the touch data. For example, the user throwing the mouse or keyboard, etc., indicates that the user has negative emotions when touching the terminal device.

[0250] S430a, actively interacting with the user based on the user behavior category.

[0251] If the user behavior category does not include the user complaint category about the terminal device, the user negative emotion category, the user online class category, and the user behavior sensitive category, the terminal device starts the normal mode; if the user behavior category includes the user negative emotion category, the terminal device actively interacts with the user based on the user behavior category.

[0252] Specifically, a visual interface matching the negative emotion category of the user is displayed, and / or a voice matching the negative emotion category of the user is determined and played.

[0253] When the terminal device asks for suggestions such as "Can I play a song for you?" or "Can I play a movie for you?", and the user agrees to the suggestion, the terminal device will perform corresponding actions, such as playing soothing songs, funny movies or sketches, telling jokes, and inspiring e-books.

[0254] In one example, the terminal device improves the comfort effect by continuously identifying the user's emotions, such as Figure 7 As shown, the method 400a further includes:

[0255] S440a: After a first preset time has passed, determine whether the current user behavior category is a user negative emotion category.

[0256] If the user's mood is not negative, that is, the user's mood has improved, the terminal device starts the normal mode. Furthermore, a picture indicating happiness may be displayed on the screen, such as a smiling face.

[0257] If the user's emotion is negative, proceed to S450a.

[0258] S450a: Send a first prompt message to a contact associated with the user.

[0259] The content of the first prompt information is described in method S400 and will not be repeated here.

[0260] Exemplarily, the first prompt information is sent via email, text message, multimedia message, etc.

[0261] Figure 8 This is another example of an active interaction effect diagram provided by an embodiment of the present application. Figure 9 This is a schematic diagram of a first prompt information provided in an embodiment of the present application. Figure 8 and Figure 9 The above method 400a is described in detail with an example.

[0262] User: I’m so sad.

[0263] The terminal device recognizes the user's negative emotions through the user's voice, activates the comfort mode, displays a comforting animation of patting the head on the display interface, and replies: Hug my dear, how about listening to a song?

[0264] User: OK.

[0265] The terminal device plays an inspiring song, and after 30 minutes, it recognizes that the user is crying through the user video, and sends a first prompt message to the user's family. The content of the first prompt message is as follows: Figure 9shown.

[0266] In method 400a, the user's negative emotions are identified through the user's multimodal data, and the user is actively interacted with based on the user's negative emotion category. The user is comforted by displaying pictures or playing voice, etc. to improve the user's mood. Furthermore, if the user's mood is still negative after a preset time, his family or friends are notified to pay attention to his emotional state to help the user recover his mood, further improving the effect of active interaction of the terminal device.

[0267] Based on the method 500, this application introduces a method for interacting with a terminal device based on the category of the user taking an online course through a specific embodiment. Figure 10 This is another schematic flow chart of an interactive method of a terminal device provided in an embodiment of the present application. Figure 10 As shown in method 500a:

[0268] S510a, determining user identity information.

[0269] Specifically, when the user turns on the terminal device, the user's facial data can be obtained through the camera, and facial recognition can be performed to determine whether the user is the subject of online course supervision (that is, the subject who has registered and needs to take online courses). If the user is not the subject of online course supervision, the terminal device starts normal mode; if the user is the subject of online course supervision, the terminal device performs method S520a.

[0270] S520a: Determine the type of the target period.

[0271] If the target time period is a rest period, the terminal device starts the normal mode; if the target time period is an online class period, method S530a is performed.

[0272] Among them, the methods for determining whether the target period is an online class period include:

[0273] Method 1

[0274] Determine whether the target period is an online class period by setting the online class period.

[0275] For example, for online classes at formal schools due to epidemic quarantine, the time periods are fixed. You can set online class time periods, such as Monday to Friday, 8:30-12:00 in the morning and 2:00-5:30 in the afternoon. For fixed app or live broadcast address access, you can set the online class time periods in advance.

[0276] Furthermore, when it is time to take an online class, the terminal device will send a reminder of the online class time and automatically open the corresponding app or website; in addition, you can also choose to manually connect to the online class and then cast the mobile phone screen.

[0277] Method 2

[0278] For online classes that can be reviewed at irregular time periods, you can customize the time period, for example, set the online class to take place at 6 p.m. every night.

[0279] Method 3

[0280] Determine whether the screenshot of the display interface is an online course screenshot, for example, by using convolutional neural network image classification to determine whether the target time period is a time period for taking online courses.

[0281] Correspondingly, other time periods outside of online class periods are rest periods.

[0282] S530a, actively interacting with the user based on the category of the online course the user is taking.

[0283] Specifically, the terminal device determines the user's action in the target period according to the output data of the terminal device and / or the user's image data, and then actively interacts with the user according to the user's action and the type of the target period.

[0284] Exemplarily, based on the user's action and the type of the target time period, a visual interface matching the category of the user's online class is displayed, and / or a voice matching the category of the user's online class is determined and played.

[0285] Among them, the visual interface that matches the user's online course category is used to report to the user the status of his or her online course. For example, it reports to the user that he or she is serious about taking the online course, and is accompanied by an animated picture of praise; or, it reports to the user that he or she has engaged in behaviors such as "wandering off" during the online course, and is accompanied by an animated picture reminding the user to pay attention next time he or she takes an online course.

[0286] The voice message that matches the user's online course category is used to report the user's online course status. For example, the voice message "You did a great job today" is played to report to the user that the user is taking the online course seriously. Alternatively, the voice message "You were absent-minded" is played to report to the user that the user was absent-minded during the online course.

[0287] In this example, by combining the type of target time period and the user's actions, the user's online class status can be accurately determined, thereby determining the content of the corresponding visual interface, or determining and playing the corresponding voice, which can further improve the effect of supervising the user's online class.

[0288] In one example, the terminal device determines the user's actions within the target period based on the output data of the terminal device and / or the user's image data, including:

[0289] The terminal device determines the user's actions during the target period based on the user's image data, including:

[0290] The terminal device obtains the user's video stream within the target time period and determines the user's action based on the user's video stream data. The method of determining the user's action includes:

[0291] Method 1

[0292] Perform action recognition on user video stream data based on swin-transformer.

[0293] For example, when a terminal device recognizes the start of an online class, the camera automatically turns on and continuously captures the user's live learning video stream data. After the class ends, the camera automatically turns off. The captured video stream is edited into multiple short video clips, each of which contains a clear action, is short in length, and has a unique action category. The action category includes a primary action and a secondary action. The primary action can be, for example, "playing with the phone" or "wandering off"; the secondary action can be, for example, "attentive listening," "doing homework," or "speaking." Finally, the swin-transformer is used to identify the user's primary and secondary actions.

[0294] In this way, during class, the duration of the user's first action or second action can be identified based on the real-time acquired video stream data, thereby achieving the effect of real-time supervision of the user's online class.

[0295] Method 2

[0296] Temporal Action Localization (TAL), also known as Temporal Action Detection (TAD), is used to identify user actions during online classes. TAD can be considered to consist of two subtasks: one predicting the start and end time intervals of an action, and the other predicting the action category. For example, if a video of a student slacking off during an online class, the "slacking off" action will be predicted and the corresponding time interval will be given, such as 10:30 AM to 10:46 AM.

[0297] The terminal device continuously captures the user's live learning video stream data. After a certain period of class (for example, 9:00-17:00) ends, the user's action categories contained in the video, as well as the start and end times of the user's actions, are predicted based on temporal action positioning. User actions typically occur only within a short period of time in the video. The monitored video may contain multiple action categories or no action at all. The action category in the video is the background category.

[0298] Taking the online class from 9:00 to 12:00 on a certain day as an example, the prediction results are as follows:

[0299] How is the class of child XX today?

[0300] 9:00-10:30 "Study hard";

[0301] 10:30-10:46 "desertion";

[0302] 10:46-11:10 “Other (Background)”

[0303] 11:10-12:00 “Study hard.”

[0304] In this way, the specific online course situation of the supervised subject within a period of time can be determined through the prediction results, so as to generate a detailed online course situation report within the period, which is conducive to the guardian of the supervised subject to clearly understand the online course situation of the supervised subject.

[0305] The terminal device determines the user's actions during the target period based on the output data of the terminal device, including:

[0306] The terminal device determines whether to switch to other content unrelated to the online course based on the display content of the terminal device to determine the user's action.

[0307] For example, when students switch to a TV program during an online class, such as exiting the online class interface or watching cartoons, it will be considered the user's first action.

[0308] Optionally, the terminal device determines the user's actions during the target period based on the terminal device's output data and the user's image data, i.e., combining the two aforementioned solutions to determine the user's actions during the target period. For example, the terminal device performs action recognition on the user's video stream data using a swin-transformer. Although the terminal device recognizes that the user is performing the second action, it determines from the terminal device's display interface that the user has switched to content unrelated to the online course, thus determining that the user is not actively attending the online course.

[0309] In one example, based on the user's action and the type of the target time period, a visual interface matching the user's online course category is displayed, and / or a voice matching the user's online course category is determined and played, including:

[0310] If the target time period is a break between classes, a visual interface that matches the category of the user's online class is displayed based on the ratio of the duration of the user's first action in the previous class to the total duration of the class, and / or a voice that matches the category of the user's online class is determined and played.

[0311] For example, if the duration of the user's first action in the previous class accounts for more than 30% of the total duration of the class, a visual interface for reporting that the user's online class status is poor will be displayed; or a voice reporting that the user's online class status is poor will be played; or both a visual interface for reporting that the user's online class status is poor will be displayed and a voice reporting that the user's online class status is poor will be played.

[0312] For another example, if the duration of the user's first action in the previous class does not exceed 30% of the total duration of the class, a visual interface for reporting the user to take the online class seriously and praising the user will be displayed; or a voice message for reporting the user to take the online class seriously and praising the user will be played; or both a visual interface for reporting the user to take the online class seriously and praising the user and a voice message for reporting the user to take the online class seriously and praising the user will be displayed.

[0313] In one example, displaying a visual interface that matches the online course supervision mode based on the user's action and the type of the target period, and / or determining and playing a voice that matches the online course supervision mode, further includes:

[0314] If the target time period is an online class, action recognition is performed based on the swin-transformer according to the real-time video stream data of the user. If the user's first action is recognized, the duration of the first action is recorded through the video stream data, and an interface to prompt the user is displayed according to the duration and / or a voice to prompt the user is determined and played.

[0315] For example, if the duration exceeds 5 minutes, a pop-up prompting the user to study hard will be played with a custom voice. This custom voice can be the voice of a parent or teacher, or even the voice of the student's idol. After the prompt is played, the user can exit within 2 seconds and return to the online course interface.

[0316] For another example, if the first action lasts for no more than 5 minutes and ends after 2 minutes, and then the second action is performed, the terminal device continues to play the online class-related video and no prompt interface will pop up.

[0317] In one possible implementation, the method 500a further includes:

[0318] S540a: Determine whether to send a second prompt message to a contact associated with the user according to the first duration.

[0319] Specifically, the first duration is the total duration of the user's first action during the online class period. Whether to send the second prompt information to the contact associated with the user is determined based on the first duration. The second prompt information is used to report the user's online class status.

[0320] The first duration is the total duration of the first action of the user during the online class period. The online class period can be understood as the period of one online class, or the period of multiple online classes, or the period of all online classes on the day.

[0321] For example, if the first duration exceeds 50 minutes, a second reminder message is sent to the contact, notifying the user that the user is not taking the online course seriously. The second reminder message can be sent to the contact after the first duration meets the preset conditions, which can be customized.

[0322] In one example, the method further includes determining whether to send a second prompt message to a contact associated with the user based on the ratio of the first duration to the second duration, where the second duration is the total duration of the user's online class.

[0323] For example, if the ratio exceeds 30% but does not exceed 50%, a second reminder message is sent to the parent, notifying the user that they are not taking the online class seriously. Alternatively, if the ratio exceeds 50%, a second reminder message is sent to the teacher, notifying the user that they are not taking the online class seriously. If the ratio does not exceed 30%, the terminal device enters normal mode. The above ratios can be set according to actual needs.

[0324] In one example, the second prompt information is sent via SMS, email, MMS, etc.

[0325] In one example, if the total duration of the user's first action on that day is 0, that is, the user has been taking the online class seriously on that day, an interface praising the user is displayed to the user.

[0326] In the above example, determining whether to send the second prompt information to the contact associated with the user based on the first duration can effectively report the user's online class situation to the contact in a timely manner, thereby improving the terminal device's supervision effect on the user's online class.

[0327] Figure 11 This is another example of an active interaction effect diagram provided in the embodiment of the present application. Figure 12 This is another example of an active interaction effect diagram provided in the embodiment of the present application. Figure 13 This is a schematic diagram of a second prompt information provided in an embodiment of the present application. Figure 14 This is another example of the second prompt information provided by the embodiment of the present application. Figures 11 to 13 The above method 500a is described in detail with examples.

[0328] During the online class period, the terminal device uses swin-transformer to identify the user's action category in real time. When it is recognized that the duration of the user's first action exceeds 5 minutes, an interface prompting the user to take the online class seriously will pop up, such as Figure 11As shown, the teacher's voice plays "Don't play too much, study hard". After the voice is played, it returns to the online class interface. After the online class ends on the same day, if the first duration does not exceed 50 minutes, an encouragement interface will pop up, such as Figure 12 As shown, the user-defined voice plays "Your performance today is great, you have the potential to be a top student." If the online class ends that day and the first duration does not exceed 50 minutes, the terminal device can send a second reminder message to the parents to report the user's online class status, such as Figure 13 As shown, the animated image and sentence used to praise the user are displayed; if the first duration exceeds 50 minutes after the end of the online class on the same day, the terminal device can send a second reminder message to the parents, reporting that the user is not serious about the online class, such as Figure 14 As shown, it shows a cartoon image of a person who does not take online classes seriously and a sentence reminding parents to supervise users to take online classes.

[0329] Furthermore, the terminal device can also obtain the video stream data of the user's online class on that day based on timing and motion positioning, identify the user's movements and corresponding time conditions, and thus generate a user learning status report so that the user's parents can understand the user's online class status.

[0330] In method 500a, by identifying the action categories of the user when taking online classes, the user is supervised to take the online classes seriously. Furthermore, the user's online class status can be reported to the user's teacher or parents, thereby strengthening the supervision of the user's online classes and further improving the effect of active interaction of the terminal device.

[0331] Based on method 600, this application introduces an interaction method of a terminal device based on user behavior sensitive categories through specific embodiments. Figure 15 This is another schematic flow chart of an interactive method of a terminal device provided in an embodiment of the present application. Figure 15 As shown in method 600a:

[0332] S610a: Acquire output data of the terminal device and / or user multimodal data.

[0333] The description of the output data of the terminal device and / or the user multimodal data is given in method 300 and will not be repeated here.

[0334] S620a: Determine the user behavior category.

[0335] Specifically, the user behavior category is determined based on the output data of the terminal device, and / or the user behavior category is determined based on the user multimodal data.

[0336] Exemplarily, a convolutional neural network model or other methods are used to identify the content of text, video, audio and other data displayed on the terminal device to determine whether the user is watching sensitive content. If so, the user behavior category is determined to be a sensitive user behavior category.

[0337] Exemplarily, a convolutional neural network model is used to identify whether the user behavior involves sensitive content based on user statements, user actions, and other user behaviors. If so, the user behavior category is determined to be a sensitive user behavior category.

[0338] Exemplarily, based on the text, video, audio and other data displayed by the terminal device and user behaviors such as user statements and user actions, a convolutional neural network model is used to identify whether the user behavior involves sensitive content. If so, the user behavior category is determined to be a sensitive user behavior category.

[0339] For example, keywords are extracted from user statements to identify whether the user behavior involves sensitive content. If the user statement contains sensitive keywords, it is considered that the user behavior involves sensitive content.

[0340] For example, by obtaining image data containing user actions through a camera, a motion recognition model can be used to determine whether there is fighting behavior (such as swin-transformer).

[0341] Another example is analyzing the information of a currently playing film or TV series to determine whether it contains sensitive content. Furthermore, for short videos or mobile screencasts, where it's difficult to obtain the title and corresponding tags, screenshots of the terminal device's display screen can be analyzed, such as using image classification methods to determine whether it contains sensitive content. If so, the user behavior is classified as sensitive.

[0342] S630a, actively interacting with the user based on the user behavior sensitive category.

[0343] If the user behavior category includes the user behavior sensitive category, the system actively interacts with the user based on the user behavior sensitive category. If the behavior category does not include the user complaint about the terminal device category, the user online class category, the user behavior sensitive category, and the user negative emotion category, the normal mode is started.

[0344] Specifically, a visual interface that matches the user behavior sensitive category is displayed, and / or a voice that matches the user behavior sensitive category is determined and played.

[0345] In one example, the visual interface is blocked, and / or a voice prompt is given to the user to change the playback content.

[0346] In one example, the user's attribute information is determined based on the user's multimodal data. If the user's attribute information meets a first preset condition, the visual interface is blocked and a voice prompt is given to the user to change the playback content; if the user's attribute information does not meet the first preset condition, a voice prompt is given to the user that the content he is watching is sensitive content.

[0347] In the above example, by using the user's attribute information and actively interacting with the user based on the sensitive protection model, it can effectively help the healthy physical and mental development of underage users.

[0348] In one possible implementation, the method 600a further includes:

[0349] When the user's attribute information meets the first preset condition, if the user behavior category includes a user behavior sensitive category, a third prompt message is sent to a contact associated with the user, where the third prompt message is used to report the user's sensitive behavior.

[0350] Exemplarily, the third prompt information is sent via SMS, MMS, email, etc.

[0351] The following uses method 600a as an example to illustrate the active interaction process based on user behavior sensitive categories.

[0352] Underage user A uses a terminal device to search for and play a movie. After obtaining the name of the movie, the terminal device determines that the user behavior category is a sensitive user behavior category, and then activates the sensitive backup mode. Through facial recognition, it is determined that user A is a minor, and the image on the display screen is mosaiced, and a voice message is played saying "You are a minor user, please do not watch this movie, it is recommended that you watch another movie." At the same time, the name of the movie watched by the user is sent to his guardian so that the guardian can supervise user A.

[0353] In method 600a, when the user behavior category includes a user behavior sensitive category, the behavior can be reminded or corrected, ensuring the healthy physical and mental development of the underage user. Furthermore, the sensitive behavior of the underage user can be reported to his or her guardian, strengthening the supervision of the underage user, and improving the effect of active interaction of the terminal device, thereby improving the user's satisfaction with the use of the terminal device.

[0354] In combination with the above embodiments, the present application further provides an interactive device of a terminal device, including:

[0355] A data acquisition module is used to acquire user voice data;

[0356] A data processing module is used to obtain the user's voice text data based on the user's voice data;

[0357] A user behavior category determination module is used to determine the user behavior category including the user's complaint category about the terminal device based on the user's voice and text data;

[0358] A complaint event determination module, configured to determine a first event matching a user's complaint category about a terminal device, the first event being a voice interaction or an application failure;

[0359] The active interaction module is used to actively interact with the user based on the first event.

[0360] For other implementations, please refer to method 300 and method 300a, which will not be described in detail here.

[0361] In combination with the above embodiments, the present application further provides an interactive device of a terminal device, including:

[0362] A data acquisition module is used to acquire multimodal data of the user or touch data of the terminal device;

[0363] A user behavior category determination module, configured to determine the user behavior category including a negative user emotion category based on the user's multimodal data or touch data for the terminal device;

[0364] An active interaction module, configured to display a visual interface that matches the user's negative emotion category, and / or determine and play a voice that matches the user's negative emotion category;

[0365] The user behavior category determination module is further configured to determine, after a first preset time, whether the current user behavior category is a user emotion negative category;

[0366] The active interaction module is further configured to send a first prompt message to a contact associated with the user if the current user behavior category is a negative user emotion category, where the first prompt message is used to remind the user of the emotional state.

[0367] For other implementations, please refer to method 400 and method 400a, which will not be described in detail here.

[0368] In combination with the above embodiments, the present application further provides an interactive device of a terminal device, including:

[0369] Data acquisition means, for acquiring output data of a terminal device and / or image data of a user;

[0370] A user behavior category determination module is used to determine the user behavior category including the user's online course category based on the output data of the terminal device and / or the user's image data;

[0371] Active interaction module, used to determine whether the user is suitable for taking online courses;

[0372] The active interaction module is also used to determine the type of target period, which can be online classes or rest;

[0373] The active interaction module is further configured to determine the user's actions within the target period based on the output data of the terminal device and / or the user's image data, where the user's actions include a first action and a second action, where the first action represents an action that the user is not allowed to perform during the online class, and the second action represents an action that the user is allowed to perform during the online class;

[0374] The active interaction module is further used to actively interact with the user based on the user's actions and the type of the target period.

[0375] For other implementations, please refer to method 500 and method 500a, which will not be described in detail here.

[0376] In combination with the above embodiments, the present application further provides an interactive device of a terminal device, including:

[0377] A data acquisition module, used to acquire output data of a terminal device and / or image data of a user;

[0378] A user behavior category determination module, configured to determine a user behavior category including a user behavior sensitive category based on output data of a terminal device and / or image data of a user;

[0379] An active interaction module, used to display a visual interface that matches the user behavior sensitive category, and / or determine and play a voice that matches the user behavior sensitive category;

[0380] The active interaction module is further configured to determine whether the attribute information of the user satisfies a first preset condition;

[0381] The active interaction module is further used to send a third prompt message to a contact associated with the user, where the third prompt message is used to report the user's sensitive behavior.

[0382] For other implementations, please refer to method 600 and method 600a, which will not be described in detail here.

[0383] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0384] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A terminal device interaction method, characterized in that: include: Get the user's voice data; Acquiring user's voice text data according to the user's voice data; performing a first processing on the user's voice and text data to determine a user behavior category including a user complaint category regarding the terminal device; wherein the first processing includes one or more of text classification, regular expression matching, and sentiment scoring, and the user complaint category regarding the terminal device indicates that the user is dissatisfied with a response of the terminal device when using the terminal device; determining, according to the user's complaint category regarding the terminal device, a first event matching the complaint category, the first event being a voice interaction or an application failure; Displaying a visual interface that matches the category of the user's complaint about the terminal device, and / or determining and playing a voice that matches the category of the user's complaint about the terminal device; Shielding a voice reply made by the terminal device in response to the voice interaction so that the voice reply is not included in a set of candidate replies that match the content of the voice interaction; Sending a reminder message, wherein the reminder message is used to prompt the user to verify the voice reply; Verifying the voice reply according to the reminder information; The verified voice response is inserted into the candidate response set.

2. The method according to claim 1, characterized in that The first event is an application failure, and actively interacting with the user based on the first event includes: Displaying a visual interface that matches the category of the user's complaint about the terminal device, and / or determining and playing a voice that matches the category of the user's complaint about the terminal device; Close the application program; Restart the application program after a first preset time.

3. An interactive device for a terminal device, characterized in that: include: A data acquisition module is used to acquire user voice data; A data processing module, configured to obtain the user's voice text data from the user's voice data; A user behavior category determination module is configured to perform a first process on the user's voice and text data to determine a user behavior category including a user complaint category regarding the terminal device; wherein the first process includes one or more of text classification, regular matching, and sentiment scoring, and the user complaint category regarding the terminal device indicates that the user is dissatisfied with a response of the terminal device when using the terminal device; a complaint event determination module, configured to determine, based on the category of the user's complaint about the terminal device, a first event matching the category of the complaint, wherein the first event is a voice interaction or an application failure; An active interaction module is used to display a visual interface that matches the category of the user's complaint about the terminal device based on the first event, and / or determine and play a voice that matches the category of the user's complaint about the terminal device; shield the voice response made by the terminal device to the voice interaction, so that the voice response is not included in the set of candidate responses that match the content of the voice interaction; send a reminder message, the reminder message is used to prompt the verification of the voice response; verify the voice response according to the reminder message; and insert the verified voice response into the set of candidate responses.

Images