Display device and man-machine interaction method

Abstract

The invention provides a display device and a man-machine interaction method. In the process that the display device interacts with the user, the historical interaction data and the associated data of the current round of user request can be combined to perform semantic analysis on the current round of user request, so that errors can be corrected in time by enhancing context awareness, the rewriting accuracy of the current round of user request is improved, and continuous transmission of the errors is avoided. Moreover, the display device can cooperatively process a plurality of semantic analysis tasks in the same semantic space through the semantic analysis model so as to realize the overall semantic analysis of the user request and ensure that the analysis result has integrity. Moreover, the analysis result can be further structured by adding a semantic tag to the analysis result, so that the understanding and expansibility of the display equipment on the final result are effectively improved, and the accuracy and efficiency of final response can be further improved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a display device and a human-computer interaction method. Background Technology

[0002] Display devices refer to terminal devices capable of outputting specific display images, such as smart TVs, mobile terminals, smart advertising screens, and projectors. Taking smart TVs as an example, smart TVs are television products based on Internet application technologies, possessing open operating systems and chips, and having open application platforms. They enable two-way human-computer interaction and integrate multiple functions such as audio-visual, entertainment, and data to meet diverse and personalized user needs.

[0003] Display devices can be configured with human-computer interaction capabilities. Based on this capability, the display device can perform semantic analysis on user requests, understand user intent, and respond to that intent, thus achieving interaction with the user. It is evident that the accuracy of the display device's understanding of user intent directly affects the accuracy of the response.

[0004] Typically, display devices employ different semantic analysis modules to independently analyze different intents. Each module performs semantic analysis only on the current user request and relies on the analysis results of the previous module. This fragmented and layer-by-layer dependent approach to semantic analysis of single-turn user requests leads to a disconnect between the final analysis results and the context, lacking overall coherence. Furthermore, it propagates and amplifies errors layer by layer, making it difficult to guarantee the accuracy of the analysis results and consequently affecting the accuracy of the final response. Summary of the Invention

[0005] This application provides a display device and a human-computer interaction method that can effectively solve the problem of difficulty in ensuring the accuracy of semantic analysis.

[0006] In a first aspect, this application provides a display device, comprising: The display is configured to show the user interface; The input device is configured to receive user requests input by the user in each round of interaction; The controller is configured as follows: Based on the user request received in the current round of interaction, obtain the historical interaction data before the current round of interaction and the associated data of the user request; The control semantic parsing model combines the historical interaction data and the associated data to perform overall semantic parsing on the user request, so as to parse at least one conditional semantic unit and at least one action semantic unit with a preset logical relationship in the request data, and obtain the first structured data; the first structured data includes the at least one conditional semantic unit and the at least one action semantic unit; The semantic units in the first structured data are labeled with corresponding semantic tags to obtain the second structured data; The user request is responded to based on the second structured data.

[0007] The above technical solution has the following beneficial effects or advantages: During user interaction, display devices can combine historical interaction data with related data from the current user request to perform semantic analysis. By enhancing context awareness, errors can be corrected promptly, improving the accuracy of rewriting the current user request and preventing the propagation of errors. Furthermore, display devices can use semantic parsing models to collaboratively process multiple semantic analysis tasks within the same semantic space, achieving holistic semantic parsing of user requests and ensuring the integrity of the parsing results. Adding semantic tags to the parsing results further structures them, effectively improving the display device's understanding and scalability of the final result, thereby enhancing the accuracy and efficiency of the final response.

[0008] In some embodiments of this application, the associated data includes candidate entities, and the controller, based on the received user request in the current round of interaction, obtains historical interaction data prior to the current round of interaction and associated data of the user request, specifically configured as follows: The user request is segmented into multiple words. Identify the specified entity from the multiple word segments; Based on a preset entity library, candidate entities that match the specified entity are determined.

[0009] The above technical solution has the following beneficial effects or advantages: For specified entities in user requests that are prone to ambiguity, the display device can identify candidate entities that match these specified entities from a preset entity library. These candidate entities can then be used as associated data for the subsequent semantic parsing model to further determine the entity actually indicated by the user, which can effectively improve the parsing accuracy of the semantic parsing model.

[0010] In some embodiments of this application, the associated data includes status information, and the controller, based on the received user request in the current round of interaction, obtains historical interaction data prior to the current round of interaction and associated data of the user request, specifically configured as follows: After receiving the user request, determine the target object related to the user request; Obtain the state information of the target object.

[0011] The above technical solution has the following beneficial effects or advantages: For user requests related to device status, the display device can identify the target objects associated with these requests and obtain the status information of the target objects. This status information can then be used as associated data for the subsequent semantic parsing model to parse the user's intent, effectively improving the parsing accuracy of the semantic parsing model.

[0012] In some embodiments of this application, the associated data includes user characteristics, and the controller, based on the received user request in the current round of interaction, obtains historical interaction data prior to the current round of interaction and associated data of the user request, specifically configured as follows: After receiving the user request, the user's user characteristics are obtained.

[0013] The above technical solution has the following beneficial effects or advantages: After receiving a user request, the display device can obtain the user's characteristics. Using these characteristics as related data for subsequent semantic parsing models to analyze user intent allows the parsing results to better match the current user's intent, thereby effectively improving the accuracy of the semantic parsing model.

[0014] In some embodiments of this application, the controller controls the semantic parsing model to perform overall semantic parsing of the user request by combining the historical interaction data and the associated data, so as to parse at least one conditional semantic unit and at least one action semantic unit with a preset logical relationship in the request data, thereby obtaining first structured data, which is specifically configured as follows: Obtain preset prompt information, the prompt information including the requirement to perform the overall semantic parsing; Based on the user request, the historical interaction data, the associated data, and the prompt information, a calling instruction is generated; The invocation instruction is sent to the semantic parsing model to control the semantic parsing model to combine the historical interaction data and the associated data, and perform overall semantic parsing of the user request according to the prompt information to obtain the first structured data.

[0015] The above technical solution has the following beneficial effects or advantages: The display device can obtain prompts including requirements for performing overall semantic parsing, and generate invocation instructions for the semantic parsing model. It can then control the semantic parsing model to accurately perform the overall semantic parsing task according to the requirements indicated by the prompts, thereby ensuring the accuracy and standardization of the first structured data.

[0016] In some embodiments of this application, the controller sends the invocation instruction to the semantic parsing model to control the semantic parsing model to combine the historical interaction data and the associated data, and perform overall semantic parsing of the user request according to the prompt information to obtain the first structured data, which is specifically configured as follows: The user request is rewritten by combining the semantic parsing model with the historical interaction data and the associated data to obtain the first text; Split the first text into multiple text objects; Determine the feature type corresponding to each text object, as well as the slot parameters it contains; The conditional semantic unit is constructed based on the slot parameters contained in the text object whose feature type is condition, and the action semantic unit is constructed based on the slot parameters contained in the text object whose feature type is action.

[0017] The above technical solution has the following beneficial effects or advantages: During the overall semantic parsing of a user request, the semantic parsing model combines historical interaction data and related data to rewrite the request, ensuring that the rewritten initial text accurately matches the true user intent. The semantic parsing model then breaks down the initial text; the split text objects retain their original logical relationships, ensuring that subsequent semantic units built based on different text objects also retain these logical relationships. This guarantees that the initial structured data accurately reflects the logical relationships expressed by the user's intent.

[0018] In some embodiments of this application, the first structured data further includes thought process data generated by the semantic parsing model when performing the overall semantic parsing.

[0019] The above technical solution has the following beneficial effects or advantages: The semantic parsing model can also output thought process data, allowing users or display devices to identify deviations in the semantic parsing model during the thought process based on this data, and thus promptly correct the next round of user requests or modify the semantic parsing model.

[0020] In some embodiments of this application, the controller labels the semantic units in the first structured data with corresponding semantic tags to obtain second structured data, which is specifically configured as follows: Identify the unit information contained in each semantic unit and the slot information contained in each semantic unit from the first structured data; The specified unit information and the specified slot information are labeled with the corresponding semantic tags to obtain the second structured data.

[0021] The above technical solution has the following beneficial effects or advantages: When a display device tags the first structured data, it filters out specified cell information and specified slot information. This specified cell and slot information can affect the accuracy of the response to user intent. Therefore, by labeling this cell and slot information with corresponding semantic tags, the display device can accurately determine the slot information required to perform the corresponding task in the second structured data based on these semantic tags, thereby ensuring that the response matches the user intent.

[0022] In some embodiments of this application, the controller responds to the user request based on the second structured data, and is specifically configured as follows: Based on the semantic tag in the second structured data, obtain the first slot information corresponding to the semantic tag in the slot information included in the conditional semantic unit, and obtain the second slot information corresponding to the semantic tag in the slot information included in the action semantic unit. The user request is responded to based on the first slot information and the second slot information.

[0023] The above technical solution has the following beneficial effects or advantages: The display device can accurately determine the slot information required to perform the corresponding task in the second structured data based on semantic tags, and respond to the user request according to the logical relationship between each semantic unit, so as to ensure that the response can match the user's intent.

[0024] Secondly, this application also provides a human-computer interaction method, the method comprising: Based on the user request received in the current round of interaction, obtain the historical interaction data before the current round of interaction and the associated data of the user request; The control semantic parsing model combines the historical interaction data and the associated data to perform overall semantic parsing on the user request, so as to parse at least one conditional semantic unit and at least one action semantic unit with a preset logical relationship in the request data, and obtain the first structured data; the first structured data includes the at least one conditional semantic unit and the at least one action semantic unit; The semantic units in the first structured data are labeled with corresponding semantic tags to obtain the second structured data; The user request is responded to based on the second structured data.

[0025] The above technical solution has the following beneficial effects or advantages: During user interaction, display devices can combine historical interaction data with related data from the current user request to perform semantic analysis. By enhancing context awareness, errors can be corrected promptly, improving the accuracy of rewriting the current user request and preventing the propagation of errors. Furthermore, display devices can use semantic parsing models to collaboratively process multiple semantic analysis tasks within the same semantic space, achieving holistic semantic parsing of user requests and ensuring the integrity of the parsing results. Adding semantic tags to the parsing results further structures them, effectively improving the display device's understanding and scalability of the final result, thereby enhancing the accuracy and efficiency of the final response. Attached Figure Description

[0026] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram illustrating the interaction scenario between the display device and the user in an embodiment of this application; Figure 2 This is a hardware configuration block diagram of the display device 200 in this embodiment of the application; Figure 3 This is a diagram showing the operating system configuration of the display device 200 in this embodiment of the application; Figure 4 This is a flowchart illustrating the human-computer interaction between the display device 200 and the user in this embodiment of the application. Figure 5 This is a timing diagram showing the human-computer interaction between the display device 200 and the user in an embodiment of this application; Figure 6 This is a flowchart illustrating the process of the display device 200 acquiring candidate entities in an embodiment of this application; Figure 7This is a flowchart illustrating how the display device 200 acquires status information in an embodiment of this application. Figure 8 This is a flowchart illustrating how the display device 200 invokes the semantic parsing model in an embodiment of this application. Figure 9 This is a flowchart illustrating how the display device 200 obtains the first structured data through a semantic parsing model in an embodiment of this application. Figure 10 This is a flowchart illustrating how the display device 200 adds tags to the first structured data in an embodiment of this application. Figure 11 This is a flowchart illustrating how the display device 200 responds to a user request in an embodiment of this application. Detailed Implementation

[0028] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0029] In this application embodiment, "terminal device" generally refers to a device with image data processing capabilities. For example, terminal devices include, but are not limited to, smart TVs, mobile terminals, computers, monitors, wearable devices, virtual reality devices, augmented reality devices, etc.

[0030] Figure 1 These are schematic diagrams illustrating human-computer interaction scenarios between a display device and a user, provided in some embodiments of this application. Figure 1 As shown, the display device 200 can communicate with control devices, such as a remote control 100, a mouse, a keyboard, etc. The display device 200 can also communicate with smart devices, such as smartphones and wearable devices. Of course, the display device 200 can also interact with the user via voice.

[0031] Figure 1 It is also shown that the display device 200 can communicate with the server 300 via various communication methods. The display device 200 may be allowed to communicate with the server 300 via a local area network (LAN), a wireless local area network (WLAN), and other networks to provide richer support through the server 300, such as network media resources provided by the server 300.

[0032] In some embodiments, the display device 200 may also provide broadcast television reception functionality, and may also provide intelligent network television functionality with computer support, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0033] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0034] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a user input interface 280, a memory, and a power supply.

[0035] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 300 according to various communication protocol types. The display device 200 may be equipped with multiple communication devices 220 depending on the supported communication methods. The communication devices 220 can enable the display device 200 to communicate with the external devices or the server 300 via wireless or wired connections.

[0036] In some embodiments, the detector 230 is used to collect signals from the external environment or to interact with the outside world.

[0037] In some embodiments, device interface 240 is used to connect to an external device.

[0038] In some embodiments, the controller 250 is used to control the overall operation of the display device 200. The controller 250 may include at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device 200 and responds to user operations through various software control programs stored in memory.

[0039] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0040] In some embodiments, the display 260 is used to receive and display image signals output from the controller 250. The display 260 may include display function components for presenting images and driving components for driving image display.

[0041] In some embodiments, a user can input user commands on a graphical user interface (GUI) displayed on a display 260, and a user input interface 280 can receive user commands through the GUI.

[0042] In some embodiments, the audio output device 270 may be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200.

[0043] In some embodiments, the user input interface 280 can be used to receive instructions requested by a user.

[0044] In some embodiments, components such as communication device 220, detector 230, device interface 240, and user input interface 280 that can be requested by the user are collectively referred to as input devices.

[0045] In some embodiments, the display device 200 may run an operating system to perform user interaction. An operating system is a computer program used to manage and control the hardware and software resources of the display device 200.

[0046] The operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices 200.

[0047] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 3 As shown, in some embodiments, the system can be divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Runtime Library layer, and the Kernel Layer.

[0048] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with users based on the applications. For example, the application layer may include a voice assistant, which provides voice interaction functionality, allowing users to interact with the display device via voice.

[0049] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Applications can access system resources and obtain system services during execution through the API interface.

[0050] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0051] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management.

[0052] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0053] The display device 200 can receive user requests through an input device, and based on human-computer interaction functions, perform semantic analysis on the user requests to understand the user's intent and respond to that intent, thus achieving interaction with the user. It is evident that the accuracy of the display device 200's understanding of the user's intent directly affects the accuracy of the response.

[0054] Typically, the display device 200 is equipped with different semantic analysis modules to independently analyze different intents. Each module performs semantic analysis only on the current user request and depends on the analysis results of the previous module.

[0055] In one example, independent modules (such as subsystems and analyzers) are assigned to different functions (e.g., task management, reminders), logical relationship analysis, and different text processing tasks (e.g., sentence segmentation, error correction, rewriting). These modules process the current user request accordingly. Consider a user request: "If I'm watching TV at 8 PM on Friday, remind me to watch this program on channel A1." Each module only processes this specific request. Therefore, it cannot accurately identify which day "Friday" refers to (i.e., the exact date), nor can it accurately determine the specific program referred to by "this program." Furthermore, it cannot know that the seemingly accurate "channel A1" is actually "channel A2." Based on this, the display device 200 will pass these uncertainties and errors to the subsequent modules that perform semantic analysis tasks. Moreover, when performing semantic parsing on the user request, each module mechanically segments the text based on keywords or punctuation, resulting in text fragments lacking inherent logical connections and structural hierarchy. Therefore, it can be said that the semantic analysis results obtained based on the segmented text fragments are only the results of isolated processing of each text fragment, and cannot accurately reflect the logical relationship between each text fragment, that is, it is difficult to accurately reflect the overall user intent.

[0056] Therefore, this fragmented and layer-by-layer dependent semantic analysis of single-round user requests will result in the final analysis results being disconnected from the context and lacking wholeness. Moreover, it will propagate and amplify errors layer by layer, making it difficult to guarantee the accuracy of the analysis results, and thus affecting the accuracy of the final response.

[0057] To address the aforementioned issues, the display device 200 provided in this application embodiment can offer an end-to-end unified semantic understanding architecture, which can accurately parse the user intent corresponding to the user request and improve the accuracy of responding to the user request.

[0058] In some embodiments, such as Figure 3The operating system shown has a user interface with an input device at the application layer for receiving user requests. At the framework layer, it has a semantic parsing service that schedules relevant modules to perform semantic parsing tasks. At the runtime layer, it has a context and association data acquisition module for acquiring historical interaction data and associated data of user requests. At the runtime layer, it also has a semantic parsing model for semantically parsing user requests. Furthermore, it has a semantic tagging module at the runtime layer for adding semantic tags to the structured data output by the semantic parsing model. Finally, it has a response module at the runtime layer for responding based on the structured data output by the semantic tagging module.

[0059] In some embodiments, the context and associated data acquisition module, the semantic parsing model, and the semantic tagging module can be integrated into a single module.

[0060] Figure 4 This is a flowchart illustrating the human-computer interaction between the display device 200 and the user in this embodiment of the application. The specific steps are as follows: Step S401: Based on the user request received in this round of interaction, obtain the historical interaction data and the associated data of the user request before this round of interaction.

[0061] In some embodiments, based on human-computer interaction functions, the user and the display device 200 can engage in at least one round of interaction. A single user request and the display device 200's response to that request are considered as one round of interaction data.

[0062] In some embodiments, users can flexibly choose how to input data.

[0063] In one example, the user can use voice input, and the display device 200 can collect the user's request (voice data) through the microphone in the input device.

[0064] In another example, the user can use text input, and the display device 200 can receive the user request (string) through the corresponding interface in the input device.

[0065] The display device 200 will convert the received user request into corresponding text data.

[0066] In some embodiments, the user request that the display device 200 needs to respond to is defined as the user request in the current round of interaction, or the user request in the current round of interaction.

[0067] In some embodiments, when a user has an interaction request, they can control the display device 200 to start a dialogue and interact within that dialogue. The user can control the display device 200 to start a completely new dialogue or to initiate a previously created historical dialogue. A dialogue includes at least one round of interaction data; within a dialogue, the interaction data prior to the current round of interaction is referred to as historical interaction data.

[0068] It is understandable that, as the display device 200 interacts with the user, the current interaction data and historical interaction data are dynamically changing.

[0069] Figure 5 This is a timing diagram of human-computer interaction between the display device 200 and the user in an embodiment of this application.

[0070] Combination Figure 5 The display device 200 can receive user requests input by the user in each round of interaction via the input device. In this embodiment, the user request in this round of interaction is "If I am watching TV at 8 pm on Friday, remind me to watch this program on channel A1" as an example for illustrative purposes.

[0071] In some embodiments, the display device 200 obtains historical interaction data after receiving a user request in the current round of interaction.

[0072] Combination Figure 5 The display device 200 can use the semantic parsing service to call the context and related data acquisition module to obtain historical interaction data.

[0073] In some embodiments, the display device 200 may acquire all historical interaction data of the current conversation to ensure the integrity of the data base.

[0074] In other embodiments, the display device 200 may filter out a portion of the historical interaction data from all historical interaction data of the current dialogue in order to minimize the amount of data while ensuring the validity of the data basis.

[0075] In one example, historical interaction data may include timestamps corresponding to user requests and system responses. The display device 200 can filter historical interaction data within a preset time range whose timestamps are prior to the current system time.

[0076] In another example, historical interaction data may include round identifiers, such as pair_id, used to sequentially record the order of corresponding rounds of interaction within the dialogue. The display device 200 can filter historical interaction data based on a preset number, selecting those where the absolute value of the difference between the round identifier and the round identifier of the current interaction is less than or equal to that preset number.

[0077] In this embodiment of the application, the following examples are used to illustrate the historical interaction data, including "User request: When does Program A on Channel A2 air? System response: Program A on Channel A2 airs every Friday at 8 pm" and "User request: What are some popular contestants on this program recently? System response: Zhang XX and Li XX are very popular on Program A recently".

[0078] In some embodiments, after receiving a user request in the current interaction, the display device 200 also obtains associated data of the user request. The data dimensions of the associated data of the user request can be flexibly changed according to parsing requirements.

[0079] Combination Figure 5 The display device 200 can call the context and associated data acquisition module through the semantic parsing service to obtain the associated data requested by the user.

[0080] In some embodiments, the data dimension of the associated data may include candidate entities. An entity is an important component in natural language processing, representing a specific object that can be referred to and manipulated. Candidate entities are entities related to the entities in the user's request, and can be used to help clarify the specific meaning of the entities in the user's request.

[0081] Figure 6 This is a flowchart illustrating the process of the display device 200 acquiring candidate entities in this embodiment of the application. The specific steps are as follows: Step S601: Perform word segmentation on the user request to obtain multiple words.

[0082] Based on the example, after processing the user request "If I am watching TV at 8 p.m. on Friday night, remind me to go to channel A1 to watch this program", we can get multiple word segments, including "if", "Friday", "night", "8 p.m.", "I", "at", "watch", "TV", "then", "remind", "I", "go", "channel A1", "watch", "this", "program".

[0083] Step S602: Identify the specified entity from multiple word segments.

[0084] In some embodiments, entities include multiple categories, such as value entities (also known as literal entities), named entities, and referential entities. Value entities have a clear and unambiguous meaning; therefore, there is no need to further identify their specific meaning using candidate entities. Named entities, however, may have uncertain or ambiguous meanings, thus requiring further identification using candidate entities. Referential entities have ambiguous meanings and require referential elimination.

[0085] Based on this, the display device 200 can identify entities from multiple word segments, and further identify a specified entity whose entity category is named entity from the identified entities.

[0086] As illustrated in the example, display device 200 can identify a specified entity from the word segmentation, including: "channel A1".

[0087] Step S603: Based on the preset entity library, determine the candidate entities that match the specified entity.

[0088] The pre-defined entity library includes entities that can serve as candidate entities for other entities (for ease of distinction, entities in the entity library are referred to as first entities).

[0089] In some embodiments, different entity libraries can be created for different application scenarios. When matching candidate entities, the display device 200 can determine the target application scenario to which the entity to be matched (the named entity in the user request) belongs, and match the candidate entity from the entity library corresponding to that target application scenario. In this way, the display device 200 can match a wider range of candidate entities with less workload.

[0090] In some embodiments, an association relationship can be pre-established between an entity and its candidate entities. The display device 200 can then associate the entity to be matched with the candidate entity in a preset entity library based on the association relationship.

[0091] In some embodiments, the display device 200 can determine the candidate entity corresponding to the entity to be matched by calculating the semantic similarity between the entity to be matched and each entity in the entity library.

[0092] As illustrated in the example, the display device 200 can determine candidate entities matching "Channel A1" from a preset entity library, such as "Channel A2", "Channel A3", and "Channel A4". In other words, the channel entity that the user actually wants to indicate is one of "Channel A1", "Channel A2", "Channel A3", and "Channel A4", which needs to be further determined.

[0093] Based on this, for specified entities in user requests that are prone to ambiguity, the display device 200 can determine candidate entities that match these specified entities from a preset entity library, and use these candidate entities as the associated data for the subsequent semantic parsing model to further determine the entity actually indicated by the user, thereby effectively improving the parsing accuracy of the semantic parsing model.

[0094] In some embodiments, the data dimension of the associated data may include state information. State information primarily refers to the device state of the display device 200, which includes software and hardware states. In one example, hardware state may include the state of hardware components, such as the state of a hardware switch. In another example, software state may include the system's operating state, such as device operating state (e.g., power-on state, standby state), user interface focus state, media playback state, application operating state, and system operating parameters (e.g., system time, brightness, volume). Of course, device state may also include the external environmental state, such as ambient brightness and ambient volume. These device states may affect the recognition of user intent requests; to improve the accuracy of user intent parsing, state information can be used as associated data.

[0095] Figure 7 This is a flowchart illustrating how the display device 200 acquires status information in this embodiment of the application. The specific steps are as follows: Step S701: After receiving the user request, determine the target object related to the user request.

[0096] If a user request contains requirements related to device status, after receiving the user request, the display device 200 can determine the target device status requested in the user request, and thus determine that the target device status is the target object related to the user request.

[0097] Step S702: Obtain the status information of the target object.

[0098] Based on the example, the user request includes a requirement for system time, namely "Friday evening at 8 PM," which identifies the target as the system time zone. The display device 200 needs to obtain the current system time zone, such as "System Time Zone: ×× Continent / ×× Province / ×× City," and use this system time zone as the basis for determining the system time; in other words, it needs to determine the system time within this system time zone.

[0099] To illustrate further, consider a user request: "Recommend TV series starring the person whose focus indicator is displayed." In this case, the user request includes a requirement related to the focus state of the user interface, namely "the person whose focus indicator is displayed." Therefore, the target object can be identified as the focus state of the user interface. The display device 200 needs to obtain the currently focused character element or control on the user interface as associated data to determine "this person."

[0100] Based on this, for user requests related to device status, the display device 200 can identify the target objects related to these requests and obtain the status information of the target objects. The status information of the target objects can be used as associated data for subsequent semantic parsing models to parse user intent, which can effectively improve the parsing accuracy of semantic parsing models.

[0101] In some embodiments, the data dimensions of the associated data may include user characteristics. User characteristics can be used to represent a user's identity, preferences, habits, etc. In one example, user characteristics may include user profiles, voiceprint features, etc.

[0102] After receiving a user request, the display device 200 can obtain the user characteristics of the user who entered the request, and then obtain information such as the user's identity, preferences, and habits.

[0103] The following examples illustrate this further. User characteristics can include user profiles and voiceprint features. A user profile represents a user's preference for viewing audiovisual media in picture-in-picture mode. Voiceprint features can represent the user X who inputs the user request.

[0104] Based on this, after receiving a user request, the display device 200 can obtain the user's characteristics. Using these user characteristics as related data for subsequent semantic parsing model analysis of user intent can make the analysis results of the semantic parsing model more closely match the current user's intent, thereby effectively improving the parsing accuracy of the semantic parsing model.

[0105] Step S402: The control semantic parsing model combines historical interaction data and related data to perform overall semantic parsing of the user request, so as to parse at least one conditional semantic unit and at least one action semantic unit with a preset logical relationship in the request data, and obtain the first structured data.

[0106] Semantic parsing models support the collaborative processing of multiple semantic analysis tasks within the same semantic space during a single reasoning process. They accurately identify various intents and can not only correct and rewrite user requests by combining historical interaction data and related data to ensure the correctness of the content, but also break down user requests into logically related fragments according to various intents, so as to preserve logical relationships while facilitating subsequent processing. Furthermore, they can accurately extract key parameters for the response.

[0107] In some embodiments, in order for the semantic parsing model to possess the above capabilities, it is necessary to train the semantic parsing model.

[0108] In one example, sample data for training the semantic parsing model is obtained. This sample data is the data from this paper and includes the request sample data to be parsed, historical interaction sample data associated with the request sample data, and related sample data. The request sample data contains at least one user intent. The sample data is input into the semantic parsing model, and the model outputs the parsing result. Annotated ground truth values ​​are used as supervision signals for model training. The labeled ground truth values ​​are the annotated request sample data, i.e., standardized structured data. The loss between the parsing result and the labeled ground truth values ​​is calculated, and this loss is used to iteratively optimize the semantic parsing model until it converges, thus completing the training of the semantic parsing model.

[0109] In some embodiments, the semantic parsing model can be a generative pre-trained model (such as the GPT series models, LLaMA models, ChatGLM models, etc.), a pre-trained model with an encoder-decoder architecture (such as T5, mT5, BART, etc.), an encoder-based model (such as BERT, RoBERTa, etc.), a dedicated semantic parsing model, or a large language model that can call external tools, etc.

[0110] In some embodiments, the semantic parsing model can be deployed locally or on a cloud server. If the semantic parsing model is deployed on a cloud server, the display device 200 deploys the corresponding API locally.

[0111] Figure 8 This is a flowchart of the display device 200 calling the semantic parsing model in this embodiment of the application. The specific steps are as follows: Step S801: Obtain the preset prompt information.

[0112] The prompt message includes the requirements for performing overall semantic parsing.

[0113] In some embodiments, the prompt information may include the execution requirements for the task to be performed, the structural requirements for the output results, etc.

[0114] In one example, the prompt could be: "Your task is to classify the user's current input text based on contextual semantics and rewrite it if necessary. Please divide the output into two parts: a list of condition dictionaries and a list of action dictionaries. The condition dictionary list contains: (details not provided here). The action dictionary list contains: (details not provided here). Rewrite other information directly in the output. Please refer to the following output format: (details not provided here)."

[0115] In some embodiments, the prompt information can be adjusted according to actual needs.

[0116] Combination Figure 5 The display device 200 can obtain preset prompt information through a semantic parsing service. For example, it can read pre-stored prompt information from memory.

[0117] Step S802: Generate a calling instruction based on the user request, historical interaction data, related data, and prompt information.

[0118] The invocation command is used to control the semantic parsing model. It takes user requests, historical interaction data, and related data as joint input data, and performs overall semantic parsing on the joint input data according to the prompt information. It outputs structured data that meets the format requirements, ensuring the accuracy and standardization of the output results, and thus ensuring the effectiveness of subsequent processing based on the output results.

[0119] Combination Figure 5 The display device 200 can generate invocation instructions based on user requests, historical interaction data, related data, and prompt information through semantic parsing services.

[0120] Step S803: Send a call instruction to the semantic parsing model to control the semantic parsing model to combine historical interaction data and related data, and perform overall semantic parsing of the user request according to the prompt information to obtain the first structured data.

[0121] Combination Figure 5 The display device 200 can send a call instruction to the semantic parsing model through the semantic parsing service to call the semantic parsing model to perform semantic parsing tasks.

[0122] Figure 9 This is a flowchart illustrating how the display device 200 obtains the first structured data through a semantic parsing model in this embodiment of the application. The specific steps are as follows: Step S901: The semantic parsing model combines historical interaction data and related data to rewrite the user request, resulting in the first text.

[0123] The semantic parsing model responds to the invocation command and performs overall semantic parsing of the user request according to the requirements in the prompt message.

[0124] Combination Figure 5 The semantic parsing model combines historical interaction data and related data to rewrite the user request, resulting in the rewritten text data, i.e., the first text.

[0125] With specific examples, the semantic parsing model, combining historical interaction data and candidate entities, can determine that the named entity to be specified, “Channel A1”, should be “Channel A2”, and that the referent entity to be specified, “this program”, should be “Program a”.

[0126] By combining the semantic parsing model with the associated data "system time zone: ×× continent / ×× province / ×× city", it can be determined that "Friday at 8 pm" is triggered according to the system time zone, and the channel name is matched according to the system time zone.

[0127] The rewritten first text could be: "If I am watching TV at 8 p.m. on Friday, remind me to watch Program A on Channel A2."

[0128] Based on this, the rewritten text data can more accurately match the true user intent and ensure the validity of the content.

[0129] In some embodiments, the user request can be rewritten based on the responder's identity to make the rewritten text more consistent with the responder's response style and more human-like. For example, the pronouns in the user request can be changed, such as rewriting "I" as "you" to match the responder's response content. Another example is adding supplementary content to the user request, such as adding "I will" before "remind you".

[0130] The first text, after further rewriting, could be: "If you are watching TV at 8 p.m. on Friday, I will remind you to switch to channel A2 to watch 'Program A'."

[0131] Based on this, the rewritten text data can better meet the requirements of being used as response content, thereby improving the user's interactive experience when using the rewritten text data as response content in the future.

[0132] Step S902: Split the first text into multiple text objects.

[0133] Combination Figure 5 The semantic parsing model splits the first text into multiple text objects.

[0134] In some embodiments, the first text can be split into multiple text objects according to sentence components, such as conditions, actions, and goals. Since there are logical relationships between the sentence components, there are also corresponding logical relationships between the text objects.

[0135] In some embodiments, the first text may contain logical relationships, such as conditional, adversative, causal, and parallel relationships. The first text can be split into multiple text objects according to the logical relationships contained therein, and these text objects have corresponding logical relationships with each other.

[0136] Step S903: Determine the feature type corresponding to each text object, as well as the slot parameters it contains.

[0137] Combination Figure 5 The semantic parsing model determines the feature type corresponding to each text object, as well as the slot parameters it contains.

[0138] In some embodiments, feature types include condition types and action types. Condition types may include time, status, event, etc. Action types may include reminders, channel switching, queries, etc. Semantic parsing models can classify text objects into corresponding feature types.

[0139] To illustrate with a specific example, the multiple text objects resulting from the splitting of the first text can include: "Friday at 8 PM", "I'm watching TV", and "Remind me to watch 'Program A' on Channel A2". Among these, the conditional text objects include "Friday at 8 PM" and "I'm watching TV", while the action-type text object includes: "Remind me to watch 'Program A' on Channel A2".

[0140] The slot parameter (slot_value), also known as the slot, slot value, or key slot, is a key parameter contained in a text object.

[0141] For specific examples, the slot parameters for "Friday at 8 PM" include: "Friday at 8 PM"; the slot parameters for "I'm watching TV" include: "Watching TV"; and the slot parameters for "Remind me to watch Program A on Channel A2" include: "Channel A2" and "Program A".

[0142] Step S904: Based on the slot parameters contained in the text object whose feature type is condition, construct a conditional semantic unit, and based on the slot parameters contained in the text object whose feature type is action, construct an action semantic unit.

[0143] Based on the structural requirements of the output results according to the prompt information, the semantic parsing model structures the parsing results and outputs the structured parsing results, namely the first structured data.

[0144] In some embodiments, a semantic unit can be constructed for each text object. A semantic unit includes at least one semantic slot, and the slot information of each semantic slot includes a slot parameter (slot_value).

[0145] In some embodiments, semantic units may further include unit information. In one example, unit information may include: unit identifiers (condition semantic unit identifiers, such as condition_id, action semantic unit identifiers, such as action_id), specific types in condition type and action type (specific types in condition type, such as condition_type, specific types in action type, such as action_type), corresponding text objects (such as text objects in condition type, condition_content, text objects in action type, action_content), and attributes at execution time (affirmation).

[0146] In some embodiments, slot information may also include: slot name (slot_name), entity type (entity_type), entity identifier (entity_id), and attributes at execution time (affirmation), etc.

[0147] If the attribute during execution is affirmative, it means that the corresponding action needs to be performed under the corresponding conditions, and the target of the execution is the corresponding slot parameter; if the attribute during execution is negative, it means that the action is not performed under the corresponding conditions, the corresponding action is not performed, and the target of the execution is not the corresponding slot parameter.

[0148] In one example, the slot parameter "Watching TV" means that if the attribute is affirmative when executing, the action needs to be performed while watching TV; if the attribute is negative when executing, the action does not need to be performed while watching TV.

[0149] Based on the attributes at execution time, the logical relationships between semantic units can be effectively defined.

[0150] Combination Figure 5 The semantic parsing model constructs conditional semantic units based on the slot parameters contained in text objects whose feature type is condition, and constructs action semantic units based on the slot parameters contained in text objects whose feature type is action, thus obtaining the first structured data.

[0151] In one example, the conditional semantic unit `cond_1` includes: `condition_id`: `cond_1`; `condition_type`: time; `condition_content`: Friday evening at 8 PM; `slot_name`: time point; `slot_value`: Friday evening at 8 PM; `affirmation`: affirmation. This conditional semantic unit indicates that an action needs to be performed at Friday evening at 8 PM.

[0152] In another example, the action semantic unit `act_1` includes: `action_id`: `act_1`; `action_type`: reminder; `action_content`: remind me to watch program a on channel A2; `affirmation`: affirmative; `slot_name`: target channel; `slot_value`: channel A2; `entity_type`: channel; `entity_id`: channel A2 id; `affirmation`: affirmative; `slot_name`: program name; `slot_value`: program a; `entity_type`: program; `entity_id`: program a id; `affirmation`: affirmative. This action semantic unit indicates that an action reminder needs to be executed, with the target being channel A2 and program a.

[0153] In some embodiments, if the prompt message indicates that thought process data should be output, the semantic parsing model may also output the thought process data generated during the overall semantic parsing.

[0154] To illustrate with a specific example, the thought process data input to the semantic parsing module could be: "Based on your historical conversation, you have been asking about 'Program A' on Channel A2. Although you just mentioned 'Channel A1,' in the context, 'this program' refers to 'Program A,' which only airs on Channel A2. To ensure accurate execution, I have rewritten your instruction as 'If you are watching TV at 8 PM on Friday, I will remind you to switch to Channel A2 to watch 'Program A.' ​​Please correct me if there are any errors."

[0155] Based on this, users or display devices 200 can use the thinking process data to discover deviations in the semantic parsing model during the thinking process, and then promptly correct the next round of user requests or modify the semantic parsing model.

[0156] Step S403: Label the semantic units in the first structured data with corresponding semantic tags to obtain the second structured data.

[0157] Combination Figure 5 After receiving the first structured data output by the semantic parsing model, the semantic parsing service can generate a calling instruction based on the first structured data and send the calling instruction to the semantic tag module to call the semantic tag module to label the semantic units in the first structured data with corresponding semantic tags, thereby obtaining structured data with semantic tags, i.e., the second structured data.

[0158] Based on this, semantic tags can be added to the parsing results to further structure the results, effectively improving the display device's understanding and scalability of the final results, and further improving the accuracy and efficiency of the final response.

[0159] Figure 10 The flowchart for adding tags to the first structured data by the display device 200 in this embodiment of the application is as follows: Step S1001: Identify the unit information contained in each semantic unit and the slot information contained in each semantic unit from the first structured data.

[0160] As can be seen from step S904, the semantic tag module can quickly decompose each semantic unit based on the first structured data, and then determine the unit information and slot information in the semantic unit.

[0161] Step S1002: Mark the specified unit information and specified slot information with corresponding semantic tags to obtain the second structured data.

[0162] To further facilitate subsequent models' understanding of the first structured data, semantic tags can be added to describe the specified cell information and the specified slot information, which also makes it easier for subsequent modules to quickly locate the relevant information based on the semantic tags.

[0163] In some embodiments, semantic tags may be displayed in the format " / / semantic tags:".

[0164] In one example, after adding a semantic tag to the conditional semantic unit cond_1 in the first structured data, the resulting structured data includes: condition_id: cond_1; condition_type: time; / / semantic tag: time; condition_content: Friday evening at 8 pm; slot_name: time point; slot_value: Friday evening at 8 pm; affirmation: affirmation.

[0165] In another example, after adding semantic tags to the action semantic unit act_1 in the first structured data, the resulting structured data includes: action_id: act_1; action_type: reminder; / / Semantic tag: reminder; action_content: remind me to watch program a on channel A2; affirmation: affirmative; slot_name: target channel; slot_value: channel A2; entity_type: channel; / / Semantic tag: channel; entity_id: channel A2 id; affirmation: affirmative; slot_name: program name; slot_value: program a; entity_type: program; / / Semantic tag: program; entity_id: program a id; affirmation: affirmative.

[0166] Step S404: Respond to the user request based on the second structured data.

[0167] Combination Figure 5 The display device 200 transmits the second structured data to the response module via a semantic parsing service, so that the response module can respond to the user request based on the second structured data.

[0168] Figure 11 This is a flowchart of the display device 200 responding to a user request in an embodiment of this application. The specific steps are as follows: Step S1101: Based on the semantic tags in the second structured data, obtain the first slot information corresponding to the semantic tag in the slot information included in the conditional semantic unit, and obtain the second slot information corresponding to the semantic tag in the slot information included in the action semantic unit.

[0169] The display device 200 can use the response module to identify semantic tags in the second structured data, thereby quickly and accurately understanding the meaning of the corresponding semantic units of the semantic tags. After understanding the meaning of each part in the second structured data, the response module can also accurately locate the slot information required to perform the action, such as slot parameters and attributes during execution, based on the identified semantic tags.

[0170] Of course, the response module can use the semantic unit containing the semantic tag as the data basis for the response.

[0171] Step S1102: Respond to the user request based on the first slot information and the second slot information.

[0172] The display device 200 can respond to user requests by performing corresponding actions under appropriate conditions based on the extracted first slot information and second slot information through the response module.

[0173] As illustrated in the example, display device 200 can be set to time the system to display a prompt message or play a voice prompt message when it detects that the system time has reached 8 p.m. on Friday and that the TV is on, reminding the user to "switch to channel A2 to watch program a".

[0174] Based on the above embodiments, during interaction with the user, the display device 200 can combine historical interaction data and related data of the current user request to perform semantic analysis on the current user request. By enhancing context awareness, errors can be corrected in a timely manner, improving the accuracy of rewriting the current user request and preventing the further propagation of errors. Furthermore, the display device can use a semantic parsing model to collaboratively process multiple semantic analysis tasks within the same semantic space to achieve holistic semantic parsing of the user request, ensuring the integrity of the parsing results. Moreover, by adding semantic tags to the parsing results, the parsing results can be further structured, effectively improving the display device's understanding and extensibility of the final result, and further enhancing the accuracy and efficiency of the final response.

[0175] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the discussion in some embodiments is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the contents of this disclosure, thereby enabling those skilled in the art to better utilize the embodiments.

Claims

1. A display device, characterized in that, include: The display is configured to show the user interface; The input device is configured to receive user requests input by the user in each round of interaction; The controller is configured as follows: Based on the user request received in the current round of interaction, obtain the historical interaction data before the current round of interaction and the associated data of the user request; The control semantic parsing model combines the historical interaction data and the associated data to perform overall semantic parsing on the user request, so as to parse at least one conditional semantic unit and at least one action semantic unit with a preset logical relationship in the request data, and obtain the first structured data; the first structured data includes the at least one conditional semantic unit and the at least one action semantic unit; The semantic units in the first structured data are labeled with corresponding semantic tags to obtain the second structured data; The user request is responded to based on the second structured data.

2. The display device according to claim 1, characterized in that, The associated data includes candidate entities. The controller, based on the user request received in the current round of interaction, obtains historical interaction data prior to the current round of interaction and the associated data of the user request, specifically configured as follows: The user request is segmented into multiple words. Identify the specified entity from the multiple word segments; Based on a preset entity library, candidate entities that match the specified entity are determined.

3. The display device according to claim 1, characterized in that, The associated data includes status information. The controller, based on the received user request in the current round of interaction, obtains historical interaction data prior to the current round of interaction and the associated data of the user request, specifically configured as follows: After receiving the user request, determine the target object related to the user request; Obtain the state information of the target object.

4. The display device according to claim 1, characterized in that, The associated data includes user characteristics. The controller, based on the user request received in the current interaction, obtains historical interaction data prior to the current interaction and associated data of the user request, specifically configured as follows: After receiving the user request, the user's user characteristics are obtained.

5. The display device according to claim 1, characterized in that, The controller controls the semantic parsing model to perform overall semantic parsing of the user request by combining the historical interaction data and the associated data, so as to parse at least one conditional semantic unit and at least one action semantic unit with a preset logical relationship in the request data, thereby obtaining the first structured data, which is specifically configured as follows: Obtain preset prompt information, the prompt information including the requirement to perform the overall semantic parsing; Based on the user request, the historical interaction data, the associated data, and the prompt information, a calling instruction is generated; The invocation instruction is sent to the semantic parsing model to control the semantic parsing model to combine the historical interaction data and the associated data, and perform overall semantic parsing of the user request according to the prompt information to obtain the first structured data.

6. The display device according to claim 5, characterized in that, The controller sends the invocation instruction to the semantic parsing model to control the semantic parsing model to perform overall semantic parsing of the user request according to the prompt information, combining the historical interaction data and the associated data, to obtain the first structured data, which is specifically configured as follows: The user request is rewritten by combining the semantic parsing model with the historical interaction data and the associated data to obtain the first text; Split the first text into multiple text objects; Determine the feature type corresponding to each text object, as well as the slot parameters it contains; The conditional semantic unit is constructed based on the slot parameters contained in the text object whose feature type is condition, and the action semantic unit is constructed based on the slot parameters contained in the text object whose feature type is action.

7. The display device according to any one of claims 1-6, characterized in that, The first structured data also includes the thought process data generated by the semantic parsing model when performing the overall semantic parsing.

8. The display device according to claim 1, characterized in that, The controller labels the semantic units in the first structured data with corresponding semantic tags to obtain the second structured data, which is specifically configured as follows: Identify the unit information contained in each semantic unit and the slot information contained in each semantic unit from the first structured data; The specified unit information and the specified slot information are labeled with the corresponding semantic tags to obtain the second structured data.

9. The display device according to claim 8, characterized in that, The controller, based on the second structured data, responds to the user request and is specifically configured as follows: Based on the semantic tag in the second structured data, obtain the first slot information corresponding to the semantic tag in the slot information included in the conditional semantic unit, and obtain the second slot information corresponding to the semantic tag in the slot information included in the action semantic unit. The user request is responded to based on the first slot information and the second slot information.

10. A human-computer interaction method, characterized in that, The method includes: Based on the user request received in the current round of interaction, obtain the historical interaction data before the current round of interaction and the associated data of the user request; The control semantic parsing model combines the historical interaction data and the associated data to perform overall semantic parsing on the user request, so as to parse at least one conditional semantic unit and at least one action semantic unit with a preset logical relationship in the request data, and obtain the first structured data; the first structured data includes the at least one conditional semantic unit and the at least one action semantic unit; The semantic units in the first structured data are labeled with corresponding semantic tags to obtain the second structured data; The user request is responded to based on the second structured data.

Images