Voice interaction method and display device

Abstract

The embodiment of the invention discloses a voice interaction method and display equipment, and the method comprises the steps: processing voice data after receiving the voice data inputted by a user, and obtaining voice information; constructing a first cue word based on the voice information, and inputting the first cue word into a label prediction model to obtain at least one prediction label; combining the at least one prediction label to obtain a combined label; acquiring historical memory data corresponding to the combined tag from a tag database, and screening out target memory data from the historical memory data; and inputting the target memory data and the request text into the intelligent agent to obtain feedback information. According to the embodiment of the invention, the multi-dimensional prediction tag corresponding to the voice data input by the user can be identified, then the current user demand and personal and crowd-level preferences are accurately identified according to the memory data corresponding to the prediction tag, information conforming to the preferences is fed back, and comprehensive and personalized interaction experience is provided for the user.

Description

Technical Field

[0001] This application relates to the field of voice interaction technology, and in particular to a voice interaction method and display device. Background Technology

[0002] With the continuous advancement of artificial intelligence technology, smart home appliances have become a core technological focus in the consumer industry. Currently, most smart home appliances offer convenient voice-based interaction, but this interaction often feels generic, with only a few devices and scenarios providing a personalized experience. More importantly, unlike smartphones and computers, which are more personal devices, home appliances are typically used by multiple people. This results in chaotic interaction data, especially in scenarios with diverse user groups (elderly / children / adults) and varying preferences (sports / watching TV / gaming).

[0003] In intelligent voice interaction devices designed for multiple users, personalized services can be achieved by accurately distinguishing different users. Personalized services can precisely match the preferences and needs of different users, avoiding command confusion and data sharing conflicts. One method to distinguish different users is to use the device or account as a unique label, unifying all interactive behaviors under a single device or login account. This model cannot effectively distinguish multiple actual users under the same device or account, leading to confusion in personalized data and misremembering user preferences. Another method is to rely on voiceprint recognition technology to identify users, but this is greatly affected by the environment, resulting in low accuracy. Furthermore, voiceprint recognition is often registration-based, cumbersome to use, and has poor user coverage. Summary of the Invention

[0004] Some embodiments of this application provide a voice interaction method and display device. After receiving user voice data, the method identifies multi-dimensional prediction tags corresponding to the voice data, and then accurately identifies the current user needs and individual and group-level preferences based on the memory data corresponding to the prediction tags, and provides feedback information that matches the user's preferences, thereby providing the user with a comprehensive and personalized interactive experience.

[0005] Firstly, some embodiments of this application provide a voice interaction method, including:

[0006] After receiving the user's voice input, the voice data is processed to obtain voice information, which includes the request text, user age, user gender, and user voice characteristics.

[0007] The first prompt word is constructed based on the request text, user age, user gender and user voice characteristics, and then the first prompt word is input into the label prediction model to obtain at least one predicted label. The predicted label is used to represent the user's identity and preferences.

[0008] Combine at least one predicted label to obtain a combined label;

[0009] Retrieve historical memory data corresponding to combined tags from a pre-built tag database, and filter out target memory data from the historical memory data. Target memory data includes historical interaction data related to the request text or key information in the historical interaction data.

[0010] The target memory data and request text are input into the agent so that the agent can generate feedback information corresponding to the request text.

[0011] The above technical solution has the following advantages or beneficial effects: After receiving user voice data, it can identify multi-dimensional predictive labels from the voice data, and then accurately identify the current user needs, as well as individual and group-level preferences based on the memory data corresponding to the predictive labels, and provide feedback with information that matches the user's preferences, thus providing the user with a comprehensive and personalized interactive experience.

[0012] In some embodiments, the step of filtering target memory data from historical memory data includes:

[0013] The request text is used to extract vectors to obtain the first feature vector;

[0014] Vector extraction is performed on historical memory data to obtain a second feature vector. Historical memory data includes historical interaction data or key information related to combined tags.

[0015] If the similarity between the first feature vector and the second feature vector is greater than a preset threshold, then the historical memory data is marked as the target memory data.

[0016] The above technical solution has the following advantages or beneficial effects: through vectorized semantic representation and similarity calculation, it is possible to identify memory data that is semantically similar to the request text, thereby accurately identifying the current user needs and individual and group preferences based on the memory data.

[0017] In some embodiments, the step of inputting target memory data and request text into an agent to generate feedback information corresponding to the request text includes:

[0018] If the amount of target memory data exceeds the preset amount, a second prompt word is constructed based on the target memory data and the request text;

[0019] The second prompt word is input into the ranking model to obtain the ranking of the target memory data. The ranking model is used to rank the target memory data from strong to weak according to the degree of correlation between the target memory data and the request text.

[0020] Input the target memory data and request text of the top-ranked preset position into the agent, so that the agent can generate feedback information corresponding to the request text.

[0021] The above technical solution has the following advantages or beneficial effects: When the number of matched target memory data is too large, inputting all of them into the agent would lead to problems such as wasted computing resources, excessively long context causing information dilution, and response delays. This application can filter out the memories that best match the user's intent through a sorting model, and avoid interference from invalid data by filtering through a preset quantity threshold, thus ensuring the efficient operation of the system.

[0022] In some embodiments, the step of processing voice data to obtain voice information includes:

[0023] Perform speech recognition on the speech data to obtain the request text;

[0024] The voice data is input into the age and gender prediction model to obtain the user's age and gender;

[0025] The voice data is input into the timbre analysis model to obtain the user's timbre characteristics.

[0026] The above technical solution has the following advantages or beneficial effects: it can simultaneously extract text content, age, gender and timbre features in a single voice input, achieving deep, seamless and multi-dimensional analysis of voice signals. It can complete user attribute recognition without requiring users to actively register, log in or declare their identity, and automatically distinguish user identities when multiple people share the device, avoiding cumbersome operations, so as to obtain the preferences of the current user in the future.

[0027] In some embodiments, the step of combining at least one predicted label to obtain a combined label includes:

[0028] At least one predicted label is input into the label combination model, which combines the predicted labels according to the logical relationship between them to obtain a combined label.

[0029] The above technical solution has the following advantages or beneficial effects: it automatically combines frequently occurring and logically related tags through a tag combination model, avoiding the need for manual definition of a large number of composite tags, reducing the maintenance cost of the tag system, and improving the automation and scalability of tag management.

[0030] In some embodiments, the step of combining at least one predicted label to obtain a combined label includes:

[0031] Obtain a set of label categories and a preset combination rule. The set of label categories includes the first label category corresponding to the predicted label, and the prediction combination rule consists of at least one second label category.

[0032] If all the second label categories are in the label category set, the predicted labels are combined according to the prediction combination rules to obtain the combined labels.

[0033] The above technical solution has the following advantages or beneficial effects: rule matching is a lightweight computation, which is faster and consumes fewer resources compared to complex model reasoning. It can also avoid manually defining a large number of composite tags, reduce the maintenance cost of the tag system, and improve the automation and scalability of tag management.

[0034] In some embodiments, the step of constructing a tag database includes:

[0035] Extract historical voice information from historical voice data, and / or obtain historical operation information. Historical voice information includes historical request text, user age, user gender, and user voice characteristics. Historical operation information includes historical application type, historical page, and historical operation.

[0036] A third prompt word is constructed based on historical speech information and historical operation information, and the third prompt word is input into the label prediction model to obtain the historical predicted label;

[0037] The historical prediction labels are combined to obtain historical combined labels, and the historical combined labels include at least one historical prediction label.

[0038] Based on historical voice information and / or historical operation information, create or update historical memory data corresponding to historical combination tags;

[0039] Save the historical memory data corresponding to the created or updated historical combination tags to the tag database.

[0040] The above technical solution has the following advantages or beneficial effects: First, it utilizes multi-dimensional information mapped from the user's historical voice and operational behavior; then, it uses a large model to perform multi-label recognition on the aforementioned multi-dimensional information; finally, it uses the large model to learn the user's interactive behavior under each label and writes it into the label database. The label database constructed in this application can discretize people and behaviors in the form of labels, ensuring both the personalization and comprehensiveness of the remembered information.

[0041] In some embodiments, the step of updating the historical memory data corresponding to the historical combination tag based on historical voice information and / or historical operation information includes:

[0042] If the historical memory data corresponding to the historical combination tags is obtained, a fourth prompt word is constructed based on the historical memory data, as well as historical voice information and / or historical operation information.

[0043] The fourth prompt word is input into the database update model to obtain the historical memory data corresponding to the updated historical combination tags. The database update model is used to create or update the historical memory data corresponding to the historical combination tags.

[0044] The above technical solution has the following advantages or beneficial effects: inputting prompt words constructed based on historical memory data, historical voice information and historical operation information into a large model to learn user interaction behavior under each label and update the label database can discretize people and behaviors in the form of labels, which ensures both the personalization and comprehensiveness of memory information.

[0045] In some embodiments, the step of creating historical memory data corresponding to historical combination tags based on historical voice information and / or historical operation information includes:

[0046] In the absence of historical memory data corresponding to historical combination tags, a fifth prompt word is constructed based on historical voice information and / or historical operation information;

[0047] Input the fifth prompt word into the database update model to obtain the historical memory data corresponding to the created historical combination tags.

[0048] The above technical solution has the following advantages or beneficial effects: inputting prompt words constructed based on historical voice information and historical operation information into a large model to learn user interaction behavior under each tag, adding historical memory data to the tag database, and discretizing people and behaviors in the form of tags can ensure both the personalization and comprehensiveness of memory information.

[0049] Secondly, some embodiments of this application provide a display device, including:

[0050] The display is configured to show the user interface.

[0051] The controller, coupled to the display, is configured to:

[0052] After receiving the user's voice input, the voice data is processed to obtain voice information, which includes the request text, user age, user gender, and user voice characteristics.

[0053] Based on the request text, user age, user gender, and user voice characteristics, a first prompt word is constructed and input into the label prediction model to obtain at least one predicted label. The predicted label is used to represent the user's identity and preferences.

[0054] Combine at least one predicted label to obtain a combined label;

[0055] Retrieve historical memory data corresponding to combined tags from a pre-built tag database, and filter out target memory data from the historical memory data. Target memory data includes historical interaction data related to the request text or key information in the historical interaction data.

[0056] The target memory data and request text are input into the agent so that the agent can generate feedback information corresponding to the request text.

[0057] The above technical solution has the following advantages or beneficial effects: After receiving user voice data, it can identify multi-dimensional predictive labels from the voice data, and then accurately identify the current user needs, as well as individual and group-level preferences based on the memory data corresponding to the predictive labels, and provide feedback with information that matches the user's preferences, thus providing the user with a comprehensive and personalized interactive experience.

[0058] This embodiment of the application processes the received voice data to obtain voice information. The voice information includes the request text, user age, user gender, and user vocal characteristics. Then, a first prompt word is constructed based on the voice information and input into a label prediction model to obtain at least one predicted label. This predicted label represents the user's identity and preferences. The at least one predicted label is then combined to obtain a combined label. Historical memory data corresponding to the combined label is retrieved from a pre-built label database, and target memory data is selected from this historical memory data. The target memory data includes historical interaction data related to the request text or key information from historical interaction data. Finally, the target memory data and the request text are input into the intelligent agent to obtain feedback information. This embodiment of the application, upon receiving user voice data, can identify multi-dimensional predicted labels from the voice data, and then accurately identify the current user's needs and individual / group-level preferences based on the memory data corresponding to the predicted labels, providing feedback information that matches the user's preferences and offering a comprehensive and personalized interactive experience. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application;

[0061] Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application;

[0062] Figure 3 This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application;

[0063] Figure 4 A flowchart illustrating a voice interaction method provided in some embodiments of this application;

[0064] Figure 5 A flowchart illustrating a voice data processing method provided in some embodiments of this application;

[0065] Figure 6 A flowchart illustrating a method for filtering target memory data provided in some embodiments of this application;

[0066] Figure 7 A flowchart illustrating a method for acquiring target memory data provided in some embodiments of this application;

[0067] Figure 8 A timing diagram of a voice interaction method provided for some embodiments of this application;

[0068] Figure 9 A flowchart illustrating a method for constructing a tag database provided in some embodiments of this application;

[0069] Figure 10 A flowchart illustrating a method for writing historical memory data, provided for some embodiments of this application;

[0070] Figure 11 This is a timing diagram of a method for constructing a tag database provided in some embodiments of this application. Detailed Implementation

[0071] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0072] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0073] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0074] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0075] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0076] In this embodiment, display device 200 generally refers to a device with screen display and data processing capabilities. For example, display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0077] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, a user can operate the display device 200 via touch operation, a mobile terminal 300, and a control device 100. The control device 100 receives user input commands and converts them into control commands that the display device 200 can recognize and respond to. For example, the control device 100 can be a remote control, a stylus, a gamepad, etc. The mobile terminal 300 includes mobile phones, tablets, laptops, and smartwatches, etc.

[0078] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0079] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application that controls the display device 200.

[0080] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0081] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0082] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0083] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0084] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0085] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0086] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0087] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0088] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0089] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0090] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0091] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0092] In some embodiments, the user input interface 280 can be used to receive instructions from user input. For example, the user input interface 280 can receive confirmation instructions from the user regarding controls in the user interface.

[0093] In some embodiments, to enable user interaction, the display device 200 may run an operating system. An operating system is a computer program that manages and controls the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running applications. The operating system also allows users to interact with the display device 200.

[0094] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0095] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 3As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0096] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0097] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0098] like Figure 3 As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0099] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0100] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 3 As shown, hardware drivers can be configured in the kernel layer. The drivers included in the kernel layer can be at least one of the following: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0101] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of layers and the specific type of the operating system may take other forms.

[0102] With the continuous advancement of artificial intelligence technology, smart home appliances have become a core technological focus in the consumer industry. Currently, most smart home appliances offer convenient voice-based interaction, but this interaction often feels generic, with only a few devices and scenarios providing a personalized experience. More importantly, unlike smartphones and computers, which are more personal devices, home appliances are typically used by multiple people. This results in complex interaction data, especially in scenarios with diverse user groups (elderly / children / adults) and varying preferences (sports / watching TV / gaming).

[0103] In smart voice interaction devices designed for multiple users, personalized services can be achieved by accurately distinguishing between different users. Personalized services can precisely match the preferences and needs of different users, avoiding command confusion and data sharing conflicts. For example, each user can receive services such as music recommendations and home settings tailored to their preferences, improving the accuracy of interaction. Personalized control enhances the user experience's sense of exclusivity and convenience, making the device smarter and more user-friendly.

[0104] One method to distinguish different users is by using a device or account as a unique identifier, grouping all interactions under a single device or login account. This approach fails to effectively differentiate between multiple actual users under the same device or account, leading to confusion in personalized data and misremembered user preferences. Another method is to rely on voiceprint recognition as a single technology for user identification, but this is highly susceptible to environmental influences, resulting in low accuracy. Furthermore, voiceprint recognition is often registration-based, cumbersome to use, and has poor user coverage.

[0105] Personalized interaction primarily focuses on building a mature memory system and how to use it. There is little work on interaction scenarios involving multiple users. The main reasons are as follows: 1. Technological Maturity: For single-user interaction scenarios, current implementation solutions for personalized interaction are still developing and maturing. Research on such issues will only begin after the basic memory system has matured and stabilized. 2. Scenario Complexity: Multi-user scenarios are complex, not only because the interaction data involves multiple different users, but also because: a) different people may appear alternately in a set of dialogue interaction data; b) the interaction data of different people cannot be identified solely by who spoke, as there are scenarios such as substitute interactions (adults searching for videos for children) and supplementary interactions (A supplementing or clarifying B's points, not necessarily based on A's intentions); c) because the interaction memory is built imperceptibly by the user, distinguishing between different people is a challenge, especially since interests often overlap within families.

[0106] To provide users with a comprehensive and personalized interactive experience, this application provides a voice interaction method that can be applied independently to the display device 200 or the server 400, or it can be implemented by the display device 200 and the server 400 in conjunction. The structure and functions of each part of the display device 200 can be found in the above embodiments. Based on the display device 200 and server 400 shown in the above embodiments, this application can further improve some functions of the display device 200 and server 400. For example... Figure 4 As shown, the voice interaction method includes the following steps:

[0107] Step S401: After receiving the voice data input by the user, the voice data is processed to obtain voice information. The voice information includes the request text, user age, user gender, and user voice characteristics.

[0108] In some embodiments, the implementation of receiving user-inputted voice data may include: the sound collector of the display device 200 can collect ambient sounds in real time and identify the collected ambient sounds; if the ambient sounds are identified to include a voice wake-up word, a wake-up prompt tone can be played, and the user-inputted voice data can be collected through the sound collector.

[0109] In some embodiments, the implementation of the display device 200 establishing a Bluetooth connection with the control device 100 and receiving voice data input by the user may further include: after detecting that the user presses the voice key on the control device 100, displaying a prompt page to start collecting voice data, and using the control device 100 to collect the voice data input by the user, and the display device 200 receiving the voice data sent by the control device 100 through the Bluetooth connection.

[0110] In some embodiments, such as Figure 5 As shown, the implementation methods for processing voice data to obtain voice information may include:

[0111] Step S501: Perform speech recognition on the speech data to obtain the request text.

[0112] The Automatic Speech Recognition (ASR) mini-model is used to translate and correct the speech data into text, resulting in the requested text. Automatic speech recognition refers to the technology of automatically converting human speech signals into corresponding text or commands.

[0113] Step S502: Input the voice data into the age and gender prediction model to obtain the user's age and gender.

[0114] The age and gender prediction mini-model is used to recognize speech data and output speech dimensions including the user's age and gender. The age and gender prediction mini-model analyzes acoustic features in speech, such as fundamental frequency, formants, speech rate, pauses, and articulation, and inputs these acoustic features into a classification or regression model for prediction.

[0115] Step S503: Input the speech data into the timbre analysis model to obtain the user's timbre characteristics.

[0116] The timbre analysis model is invoked to identify the timbre of speech data, obtaining textual information about the user's timbre characteristics. A timbre analysis model is a computational model used to extract, identify, and understand timbre features in speech or music and convert them into textual descriptions.

[0117] For example, after processing the voice data, the following voice information is obtained: Request text: I want to see a video of a construction vehicle; User age: 55 years old (elderly); User gender: female; User voice characteristics: accented, voice is relatively high-pitched and clear.

[0118] It should be noted that, to ensure user privacy, if the display device 200 has sufficient memory and computing power, the display device 200 can process the voice data. To speed up data processing, the voice data can also be sent to the server 400 for processing. Furthermore, this application does not limit the execution order of steps S501-S503.

[0119] Step S402: Construct a first prompt word based on the voice information, and input the first prompt word into the label prediction model to obtain at least one predicted label. The predicted label is used to represent the user's identity and preferences.

[0120] The process involves constructing prompt word templates by filling the corresponding prompt templates with voice information extracted from the voice data, thus forming the first prompt word. The prompt word template includes character settings, task objectives, contextual information, input data, output format, example demonstrations, and instruction reinforcement.

[0121] Role setting is used to define the role or identity of the artificial intelligence and guide its behavioral style. For example, you are a senior tag prediction expert. Task objective is used to clarify the specific task to be completed. Context information is used to provide background information or constraints. In this embodiment, tag prediction needs to refer to various information, targeting the final audience of the interaction, including situations like "adults searching for resources for children." When the target audience changes, information such as age, gender, and timbre extracted from the speech needs to be discarded. Input data consists of variable placeholders that are filled at runtime. In this embodiment, speech information is filled into the input data. Output format is used to specify the structure or format of the returned result. Example demonstration is used to provide 1-2 input / output examples to enhance understanding. Instruction reinforcement is used to clarify requirements, such as conciseness, bullet points, and avoiding speculation.

[0122] The constructed first prompt word is input into the label prediction model, which outputs predicted labels representing user identity and preferences. The memory data is the request text or memory information extracted from the request text. For example, if the voice information is: Request text: I want to see videos of construction vehicles; User age: 55 (elderly); User gender: female; User voice characteristics: accented, high-pitched and clear (elderly person searching for resources for a child), the output predicted labels would be: Age: Child; Interest: Construction vehicles.

[0123] It should be noted that predictive labels should be defined in advance based on actual needs. In principle, different products have different labeling systems, and the system design should be tailored to specific requirements. The specific design of the system will not be elaborated here.

[0124] Step S403: Combine at least one predicted label to obtain a combined label. The combined label includes single labels and multiple labels.

[0125] In some embodiments, one implementation of combining at least one predicted label to obtain a combined label may include: inputting at least one predicted label into a label combination model, so that the label combination model combines the predicted labels according to the logical relationships between the predicted labels to obtain a combined label. The label combination model is used to combine predicted labels according to the logical relationships between the predicted labels. Logical relationships include functional, causal, subordinate, and modifying relationships. The label combination model can also be used to combine predicted labels according to the co-occurrence strength, semantic similarity, etc. of multiple predicted labels. Co-occurrence strength refers to the probability that multiple predicted labels co-occur in historical data.

[0126] For example, the predicted labels "child," "male," and "construction vehicle" are input into the label combination model to obtain the combined labels "child," "male," "child + male," and "child + construction vehicle."

[0127] In some embodiments, combining at least one predicted label to obtain a combined label may include: obtaining a set of label categories and a preset combination rule, wherein the set of label categories includes a first label category corresponding to the predicted label, and the prediction combination rule consists of at least one second label category. If all second label categories in the preset combination rule are in the set of label categories, the predicted labels are combined according to the prediction combination rule to obtain a combined label. If any second label category in the preset combination rule is not in the set of label categories, then it is not necessary to combine the predicted labels according to the prediction combination rule; instead, the system continues to determine whether the second label category in the next preset combination rule is in the set of label categories. The second label categories in the preset combination rule can be one or more, meaning the combined label can be a single predicted label or multiple predicted labels.

[0128] For example, the predicted labels are "child" and "male," and the first label category is "age" and "gender," meaning the label category set is {age and gender}. If the preset combination rule is "age + gender," and the second label category is "age" and "gender," and both the second label category are in the label category set, then the combined label is "child + male." If the preset combination rule is "age + hobby," and the second label category is "age" and "hobby," and the second label category "hobby" is not in the label category set, then no combined label can be generated.

[0129] Step S404: Obtain historical memory data corresponding to the combined tags from the pre-built tag database, and filter out the target memory data from the historical memory data.

[0130] The system iterates through the combined tags and performs a hybrid vector retrieval based on the memory data of the request text and the combined tags to obtain the target memory data. This target memory data includes historical interaction data related to the request text or key information from historical interaction data.

[0131] In some embodiments, such as Figure 6 As shown, one implementation method for filtering target memory data from historical memory data may include:

[0132] Step S601: Extract vectors from the request text to obtain the first feature vector.

[0133] The purpose of vector extraction is to transform the request text into a fixed-length numerical vector, i.e., an embedded vector, so that machine learning models can process and understand its semantics.

[0134] Step S602: Extract vectors from the historical memory data to obtain the second feature vector.

[0135] The tag database includes multiple tags constructed based on user voice and behavioral interactions, along with their corresponding memory data. The construction of the tag database is described in detail below. Historical memory data corresponding to combined tags can be obtained from the tag database, and then vector extraction is performed on the historical memory data to obtain a second feature vector. The historical memory data includes historical interaction data related to the combined tags or key information within historical interaction data. Historical interaction data refers to the request text corresponding to the user's historical voice input data.

[0136] It should be noted that, in order to improve the speed of filtering target memory data, the memory data of each tag in the tag database can be extracted into vectors in advance, and then the tag and its corresponding second feature vector can be saved. The second feature vector corresponding to the combined tags can be obtained from the tag database.

[0137] Step S603: Calculate the similarity between the first feature vector and the second feature vector.

[0138] Calculate the cosine similarity between the first and second feature vectors. Cosine similarity reflects the semantic or thematic similarity between two texts. A similarity greater than a preset threshold indicates that the requested text is similar to the stored data, while a similarity less than or equal to the preset threshold indicates that the requested text is not similar to the stored data.

[0139] Step S604: Determine whether the similarity is greater than the preset threshold.

[0140] If the similarity is greater than a preset threshold, then proceed to step S605: mark the historical memory data as the target memory data.

[0141] If the similarity is less than or equal to a preset threshold, then proceed to step S606: Do not mark historical memory data as target memory data.

[0142] For example, the target memory data is as follows:

[0143] "Children": "Memory content, including interactive raw data and extracted memory information, such as tag-based memory information: construction vehicles, Paw Patrol, nursery rhymes";

[0144] "Children + Construction Vehicles": "Memory content includes interactive raw data and extracted memory information, such as tag-based memory information: excavator, fire truck, rescue vehicle."

[0145] Step S405: Input the target memory data and request text into the agent so that the agent can generate feedback information corresponding to the request text.

[0146] In some embodiments, the implementation of inputting target memory data and request text into an agent to generate feedback information corresponding to the request text may include: determining whether the amount of target memory data exceeds a preset amount. If the amount of target memory data exceeds the preset amount, then constructing a second prompt word based on the target memory data and the request text. Specifically, constructing a prompt word template involves filling the current request text or voice information, as well as the target memory data, into the corresponding prompt template to form the second prompt word.

[0147] The second prompt word is then input into the ranking model to obtain the ranking of the target memory data. The ranking model is used to sort the target memory data from strongest to weakest according to the correlation between the target memory data and the request text. Finally, the target memory data with the highest preset ranking and the request text are input into the agent to generate feedback information corresponding to the request text.

[0148] For example, if the request text is "Recommend a movie for me" and the target memory data is team XX, then the agent will output movies similar to team XX.

[0149] This application's embodiments first use a small model to extract multi-dimensional information mapped from speech, then use a large model to perform label recognition on the aforementioned multi-dimensional information, and finally perform parallel multi-label memory reading based on the user request text. Furthermore, considering performance and the extremely long length of memory information to be read, an automatic sorting function is added to the memory reading program, allowing for more accurate memory sorting when necessary. The intelligent agent performs targeted task understanding and arrangement based on the read interaction memory, bringing a comprehensive and personalized interactive experience.

[0150] In some embodiments, steps S402-S406 may be partially or entirely performed by the server 400. The display device 200 sends voice data and a device identifier to the server 400, which can obtain a tag database corresponding to the device identifier in order to filter out target memory data from the tag database corresponding to the current device.

[0151] In some embodiments, the target memory data acquisition method may be as follows: Figure 7 As shown, after receiving the user's voice input, a small voice information extraction model is used to extract the voice information from the voice data. This model includes an automatic speech recognition model, an age recognition prediction model, and a timbre analysis model. The voice information includes the request text, user age, user gender, and timbre characteristics. Then, the first prompt word constructed based on the voice information is input into the label prediction model to obtain predicted labels: child, male, and construction vehicle. The predicted labels are then combined and a parallel hybrid retrieval is performed based on the request text and the memory data of the combined labels in the label database to obtain the target memory data. If the target memory data is too long, it needs to be comprehensively sorted, and the memory data ranked first is taken as the final input data into the agent.

[0152] In some embodiments, the timing diagram of the voice interaction method can be as follows: Figure 8 As shown. The display device 200 includes a voice information extraction module, a tag prediction module, a tag combination module, a memory data filtering module, and a business processing module. After receiving voice data input by the user, the voice information extraction module processes the voice data to obtain voice information and sends it to the tag prediction module, and sends the request text to the business processing module. The voice information includes the request text, user age, user gender, and user voice characteristics. The tag prediction module constructs a first prompt word based on the request text, user age, user gender, and user voice characteristics, inputs the first prompt word into the tag prediction model to obtain a predicted tag, and then sends the predicted tag to the tag combination module. The tag combination module combines the predicted tags to obtain a combined tag, and then sends the combined tag to the memory data filtering module. The memory data filtering module filters target memory data from a pre-built tag database based on the combined tag, and then sends the target memory data to the business processing module. The business processing module inputs the target memory data and the request text into the intelligent agent, obtains feedback information, and then sends the feedback information to the display 260, which displays the feedback information.

[0153] In some embodiments, the voice information extraction module, tag prediction module, tag combination module, memory data filtering module, and business processing module may be partially or entirely deployed on the server 400, that is, the server 400 performs these functions independently, or the display device 200 and the server 400 cooperate to perform voice information extraction, tag prediction, tag combination, memory data filtering, and business processing.

[0154] In some embodiments, such as Figure 9 As shown, an implementation method for constructing a tag database may include the following steps:

[0155] Step S901: Extract historical voice information from historical voice data, and / or obtain historical operation information.

[0156] The historical voice information includes historical request text, user age, user gender, and user voice characteristics. Historical operation information includes historical application types, historical pages, and historical actions. Historical actions include remote control operations, while historical application types and historical pages refer to the application types and pages displayed in response to remote control operations. Extracting historical voice information is the same as processing the voice data described above, and will not be repeated here.

[0157] For example, historical operation information: "Application":"XX application","Page":"Children","Operation":"Scroll down".

[0158] In some embodiments, after the total amount of statistical interaction data (the total amount of historical voice data and historical operation information) reaches a specified amount, historical voice information and / or historical operation information are extracted from the accumulated historical voice data.

[0159] In some embodiments, historical voice information and / or historical operation information are periodically extracted from historical voice data. For example, the reception time of the most recent interaction data is obtained, and if the difference between the reception time of the most recent interaction data and the reception time of the first stored interaction data is greater than a preset duration, then historical voice information and / or historical operation information are extracted from the stored historical voice data. Stored historical interaction data can be deleted after the tag database is updated.

[0160] Step S902: Construct a third prompt word based on historical speech information and / or historical operation information, and input the third prompt word into the label prediction model to obtain historical predicted labels.

[0161] Construct prompt word templates and fill in historical voice information and historical operation information into the corresponding prompt templates to form third prompt words.

[0162] Step S903: Combine the historical prediction labels to obtain historical combined labels.

[0163] Historical combined labels include at least one historical prediction label. The method for combining historical prediction labels is the same as that for combining prediction labels, and will not be repeated here.

[0164] Step S904: Create or update historical memory data corresponding to historical combination tags based on historical voice information and / or historical operation information.

[0165] Determine if historical memory data corresponding to historical combination tags can be obtained. If historical memory data corresponding to historical combination tags can be obtained, construct a fourth prompt word based on the historical memory data, as well as historical voice information and / or historical operation information. Then, input the fourth prompt word into the database update model to obtain updated historical memory data corresponding to historical combination tags. The database update model is used to create or update historical memory data corresponding to historical combination tags.

[0166] In the absence of historical memory data corresponding to historical combination tags, a fifth prompt word is constructed based on historical voice information and / or historical operation information. The fifth prompt word is then input into the database update model to obtain the historical memory data corresponding to the created historical combination tags.

[0167] For example, historical memory data is as follows: "Children": "Memory content, including original interactive data and extracted memory information, such as tag-based memory information: construction vehicle, Paw Patrol, nursery rhymes"; "Children + Construction Vehicles": "Memory content, including original interactive data and extracted memory information, such as tag-based memory information: excavator, fire truck, rescue vehicle".

[0168] This application embodiment traverses all combined tags and recalls the existing memory data under the current combined tag. Using a large model, historical memory data and current interaction data are fused to update the memory data of this tag combination.

[0169] Step S905: Save the historical memory data corresponding to the created or updated historical combination tags to the tag database.

[0170] This application first uses a small model to extract multi-dimensional information mapped from speech and user actions. Then, a large model is used to perform multi-label recognition on this multi-dimensional information. Finally, the large model is used to learn user interaction behaviors under each label and write them into a label database. The label database constructed in this application can discretize people and behaviors through labels, ensuring both the personalization and comprehensiveness of the remembered information.

[0171] In this embodiment, the memory data is stored at a mixed granularity in the form of discrete tag combinations, such as male, elderly, sports, and TV series watching. However, global device-based memory data still exists, and global memory is more general data, such as membership purchase preferences.

[0172] In some embodiments, the historical memory data writing method can be as follows: Figure 10As shown, a small speech information extraction model is used to extract historical speech information from historical speech data. This historical speech information includes historical request text, user age, user gender, and voice characteristics. Historical operation information is also extracted, including historical application type, historical pages, and historical actions. Then, a third prompt word constructed based on the historical speech and operation information is input into a label prediction model to obtain predicted labels: child, male, sports, opera, and war. Finally, the predicted labels are combined, and historical memory data corresponding to the combined historical labels is created or updated based on the historical speech and operation information.

[0173] In some embodiments, the sequence diagram of the method for constructing the tag database can be as follows: Figure 11 As shown, the display device 200 includes a voice information extraction module, an operation information acquisition module, a tag prediction module, a tag combination module, a memory data learning module, and a tag database. The voice information extraction module extracts historical voice information from historical voice data and sends it to the tag prediction module and the memory data learning module. The historical voice information includes historical request text, user age, user gender, and user voice characteristics. The operation information extraction module acquires historical operation information and sends it to the tag prediction module and the memory data learning module. The tag prediction module constructs a third prompt word based on the historical voice information and historical operation information, inputs the third prompt word into the tag prediction model to obtain historical predicted tags, and then sends the historical predicted tags to the tag combination module. The tag combination module combines the historical predicted tags to obtain historical combined tags, and then sends the combined tags to the memory data learning module. The memory data update module creates or updates the historical memory data corresponding to the historical combined tags based on the historical voice information and historical operation information, and then saves the created or updated historical memory data corresponding to the historical combined tags to the tag database.

[0174] In some embodiments, the voice information extraction module, operation information acquisition module, tag prediction module, tag combination module, memory data learning module, and tag database may also be partially or fully deployed on the server 400, that is, the server 400 alone or the display device 200 and the server 400 cooperate to complete historical voice information extraction, historical operation information acquisition, tag prediction, tag combination, memory data update, etc.

[0175] This application applies to display devices 200 with multi-user characteristics, such as smart TVs in homes. Based on a size model and the interaction data generated between the user and the display device 200, it automatically completes multi-tag interaction memory. During real-time voice interaction, it combines voice, remote control operation, and multi-dimensional information mapped from language to accurately identify current user needs and individual / group preferences, providing users with a comprehensive and personalized interactive experience.

[0176] Some embodiments of this application also provide a computer-readable storage medium that can store a program. When the computer storage medium is configured in a display device or server, the program, when executed, can include the program steps involved in the voice interaction method in the above embodiments. The computer storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0177] This application provides an electronic device, which includes a processor and a memory for storing processor-executable instructions. The processor is configured to read executable instructions from the memory and execute the instructions to implement the voice interaction method described in the above embodiments.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A voice interaction method, characterized in that, include: After receiving the voice data input by the user, the voice data is processed to obtain voice information, which includes the request text, user age, user gender and user voice characteristics. Based on the request text, the user's age, the user's gender, and the user's voice characteristics, a first prompt word is constructed, and the first prompt word is input into a tag prediction model to obtain at least one predicted tag, which is used to characterize the user's identity and preferences; At least one of the predicted labels is combined to obtain a combined label; The historical memory data corresponding to the combined tags is obtained from the pre-built tag database, and the target memory data is filtered out from the historical memory data. The target memory data includes historical interaction data related to the request text or key information in the historical interaction data. The target memory data and the request text are input into the agent so that the agent can generate feedback information corresponding to the request text.

2. The method according to claim 1, characterized in that, The step of filtering target memory data from the historical memory data includes: The request text is subjected to vector extraction to obtain a first feature vector; Vector extraction is performed on the historical memory data to obtain a second feature vector. The historical memory data includes historical interaction data or key information related to the combined tags. If the similarity between the first feature vector and the second feature vector is greater than a preset threshold, then the historical memory data is marked as the target memory data.

3. The method according to claim 1, characterized in that, The step of inputting the target memory data and the request text into the intelligent agent to generate feedback information corresponding to the request text includes: If the amount of target memory data exceeds a preset amount, a second prompt word is constructed based on the target memory data and the request text; The second prompt word is input into the ranking model to obtain the ranking of the target memory data. The ranking model is used to rank the target memory data from strong to weak according to the degree of correlation between the target memory data and the request text. The target memory data ranked at the top of the preset order and the request text are input into the intelligent agent, so that the intelligent agent can generate feedback information corresponding to the request text.

4. The method according to claim 1, characterized in that, The step of processing the voice data to obtain voice information includes: The voice data is subjected to speech recognition to obtain the request text; The voice data is input into the age and gender prediction model to obtain the user's age and gender; The voice data is input into the timbre analysis model to obtain the user's timbre characteristics.

5. The method according to claim 1, characterized in that, The step of combining at least one of the predicted labels to obtain a combined label includes: At least one of the predicted labels is input into the label combination model, so that the predicted labels are combined by the label combination model according to the logical relationship between the predicted labels to obtain a combined label.

6. The method according to claim 1, characterized in that, The step of combining at least one of the predicted labels to obtain a combined label includes: Obtain a set of label categories and a preset combination rule, wherein the set of label categories includes a first label category corresponding to the predicted label, and the prediction combination rule consists of at least one second label category; If all of the second label categories are in the set of label categories, the predicted labels are combined according to the prediction combination rules to obtain combined labels.

7. The method according to claim 1, characterized in that, The steps for constructing the tag database include: Extract historical voice information from historical voice data, and / or obtain historical operation information, wherein the historical voice information includes historical request text, user age, user gender and user voice characteristics, and the historical operation information includes historical application type, historical page and historical operation; A third prompt word is constructed based on the historical voice information and the historical operation information, and the third prompt word is input into the label prediction model to obtain the historical predicted label; The historical prediction labels are combined to obtain historical combined labels, wherein the historical combined labels include at least one of the historical prediction labels; Based on the historical voice information, and / or the historical operation information, the historical memory data corresponding to the historical combination tag is created or updated; The historical memory data corresponding to the created or updated historical combination tags is saved to the tag database.

8. The method according to claim 7, characterized in that, The step of updating the historical memory data corresponding to the historical combination tag based on the historical voice information and / or the historical operation information includes: If the historical memory data corresponding to the historical combination tag is obtained, a fourth prompt word is constructed based on the historical memory data, the historical voice information and / or the historical operation information; The fourth prompt word is input into the database update model to obtain the updated historical memory data corresponding to the historical combination tag. The database update model is used to create or update the historical memory data corresponding to the historical combination tag.

9. The method according to claim 8, characterized in that, The step of creating historical memory data corresponding to the historical combination tag based on the historical voice information and / or the historical operation information includes: In the absence of historical memory data corresponding to the historical combination tags, a fifth prompt word is constructed based on the historical voice information and / or the historical operation information; The fifth prompt word is input into the database update model to obtain the historical memory data corresponding to the created historical combination tag.

10. A display device, characterized in that, include: monitor; Configured to display the user interface; The controller, coupled to the display, is configured to: After receiving the voice data input by the user, the voice data is processed to obtain voice information, which includes the request text, user age, user gender and user voice characteristics. Based on the request text, the user's age, the user's gender, and the user's voice characteristics, a first prompt word is constructed, and the first prompt word is input into a tag prediction model to obtain at least one predicted tag, which is used to characterize the user's identity and preferences; At least one of the predicted labels is combined to obtain a combined label; The historical memory data corresponding to the combined tags is obtained from the pre-built tag database, and the target memory data is filtered out from the historical memory data. The target memory data includes historical interaction data related to the request text or key information in the historical interaction data. The target memory data and the request text are input into the agent so that the agent can generate feedback information corresponding to the request text.

Images