Display device and semantic understanding method

Abstract

Some embodiments of the present application provide a display device and a semantic understanding method. The display device can obtain speech text according to a voice instruction of a user, and obtain text segmentation and text entity based on the speech text. The display device obtains a target entity label corresponding to the text entity, and a device state of the display device. The display device obtains a user intent and a domain type based on the speech text, the target entity label and the device state, and obtains slot filling information based on the text segmentation and the target entity label. The display device performs parameter packaging on the user intent, the domain type and the slot filling information to obtain packaging information, which can represent semantics of the voice instruction. The display device can determine user semantics according to the packaging information, so as to perform corresponding operations, thereby improving the user experience.

Description

Technical Field

[0001] This application relates to the field of display device technology, and more particularly to a display device and a semantic understanding method. Background Technology

[0002] With the development of artificial intelligence technology, voice interaction is gradually entering various aspects of people's lives. People can use voice interaction to control display devices. They can use voice interaction to watch videos, listen to music, check the weather, control devices, and perform a series of other operations.

[0003] For display devices, in implementing voice interaction functions, a voice recognition module typically recognizes the user's voice commands as text. Then, a semantic analysis module performs lexical, syntactic, and semantic analysis on this text to determine the user's intent. Finally, the control unit controls the smart electronic device to perform the corresponding operations based on the user's intent.

[0004] When performing lexical, syntactic, and semantic analysis on text, related technologies typically extract keywords and analyze their corresponding semantics. However, relying solely on keyword analysis only allows for a superficial understanding of user intent, failing to accurately capture the full semantic meaning of the text. Consequently, it cannot accurately respond to user voice commands, severely impacting the user experience. Summary of the Invention

[0005] This application provides a display device and a semantic understanding method. It addresses the problem in related technologies where the semantic meaning of text cannot be accurately obtained, thus failing to accurately respond to user voice commands and severely impacting the user experience.

[0006] In a first aspect, some embodiments of this application provide a display device, including a display and a controller. The controller is configured to perform the following steps:

[0007] Get speech text;

[0008] Based on the speech text, text segmentation and text entities are obtained. The text segmentation is the segmentation obtained after segmenting the speech text, and the text entities are the entities obtained after extracting entities from the speech text.

[0009] Obtain the target entity label corresponding to the text entity;

[0010] User intent and domain type are obtained based on the voice text and the target entity tags; slot filling information is obtained based on the text segmentation and the target entity tags.

[0011] The user intent, the domain type, and the slot filling information are encapsulated with parameters to obtain encapsulation information, which is used to characterize the semantics of the speech text.

[0012] Secondly, some embodiments of this application provide a semantic understanding method applied to a display device, including:

[0013] Get speech text;

[0014] Based on the speech text, text segmentation and text entities are obtained. The text segmentation is the segmentation obtained after segmenting the speech text, and the text entities are the entities obtained after extracting entities from the speech text.

[0015] Obtain the target entity label corresponding to the text entity;

[0016] User intent and domain type are obtained based on the voice text and the target entity tags; slot filling information is obtained based on the text segmentation and the target entity tags.

[0017] The user intent, the domain type, and the slot filling information are encapsulated with parameters to obtain encapsulation information, which is used to characterize the semantics of the speech text.

[0018] As can be seen from the above technical solutions, some embodiments of this application provide a display device and a semantic understanding method. The display device can obtain speech text based on a user's voice commands, and obtain text segmentation and text entities based on the speech text. The display device obtains the target entity tags corresponding to the text entities. The display device obtains the user intent and domain type based on the speech text and target entity tags, and obtains slot filling information based on text segmentation and target entity tags. The display device encapsulates the user intent, domain type, and slot filling information into encapsulated information, which can represent the semantics of the voice command. The display device can determine the user's semantics based on this encapsulated information, thereby performing corresponding operations and improving the user experience. Attached Figure Description

[0019] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This illustrates a use case of a display device according to some embodiments;

[0021] Figure 2 A hardware configuration block diagram of a control device 100 according to some embodiments is shown;

[0022] Figure 3 A hardware configuration block diagram of a display device 200 according to some embodiments is shown;

[0023] Figure 4 A software configuration diagram of a display device 200 according to some embodiments is shown;

[0024] Figure 5 A schematic diagram of the voice interaction network architecture of the display device is shown in some embodiments;

[0025] Figure 6 Schematic diagrams of the display device system settings UI interface are shown in some embodiments;

[0026] Figure 7 The diagram illustrates a display showing confirmation information for the voice interaction mode in some embodiments;

[0027] Figure 8 The diagrams show the interaction flowcharts of various components of the display device in some embodiments;

[0028] Figure 9 A schematic diagram of slot filling information is shown in some embodiments;

[0029] Figure 10 The diagram shows the interaction flowchart between the display device and the server in some embodiments;

[0030] Figure 11 The illustrations show some scenarios of users interacting with display devices via voice in some embodiments;

[0031] Figure 12 A schematic diagram of a display device showing a search interface is shown in some embodiments;

[0032] Figure 13 A schematic diagram of a media asset details page is shown in some embodiments;

[0033] Figure 14 The diagram shows a display device showing prompt information in some embodiments;

[0034] Figure 15 A flowchart illustrating an embodiment of the semantic understanding method is shown. Detailed Implementation

[0035] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0036] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0037] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0038] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0039] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0040] Figure 1 This is a schematic diagram illustrating a usage scenario of the display device according to an embodiment. For example... Figure 1 As shown, the user can operate the display device 200 through the terminal device 300 and the control device 100.

[0041] In some embodiments, the control device 100 may be a remote control. Communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via other wired means. The wireless method can be direct or indirect, routed or unrouted. Users can input user commands through buttons on the remote control, voice input, control panel input, etc., to control the display device 200. For example, users can input corresponding control commands through volume up / down buttons, channel control buttons, up / down / left / right movement buttons, voice input buttons, menu buttons, power on / off buttons, etc., to achieve the function of controlling the display device 200.

[0042] In some embodiments, the smart device 300 may include any one of a mobile terminal, tablet computer, computer, laptop computer, AR / VR device, and stylus.

[0043] In some embodiments, the smart device 300 may also be used to control the display device 200. For example, the smart device 300 may be used to control an application running on the display device 200, or the application running on the smart device 300 may be used to control the display device 200. This application, through configuration, can provide the user with various controls on the screen associated with the smart device, within an intuitive user interface (UI).

[0044] In some embodiments, the smart device 300 and the display device may also be used for data communication.

[0045] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, it can be controlled by directly receiving the user's voice commands through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving the user's voice commands through a voice control device set outside the display device 200.

[0046] In some embodiments, the display device 200 also communicates with the server 400. The display device 200 may communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactive features to the display device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

[0047] In some embodiments, software steps executed by one execution entity can be migrated to another execution entity with which it communicates data, as needed. For example, software steps executed by a server can be migrated to a display device with which it communicates data, and vice versa.

[0048] Figure 2 An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0049] In some embodiments, the communication interface 130 is used for external communication and includes at least one of a WIFI chip, a Bluetooth module, an NFC module, or an alternative module.

[0050] In some embodiments, the user input / output interface 140 includes at least one of a microphone, touchpad, sensor, button, or alternative module.

[0051] Figure 3 A hardware configuration block diagram of a display device 200 according to an exemplary embodiment is shown.

[0052] In some embodiments, the display device 200 includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0053] In some embodiments, the controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to an nth interface for input / output.

[0054] In some embodiments, the display 260 includes a display screen component for presenting an image, a driving component for driving image display, a component for receiving image signals output from a controller, and a user control UI interface, etc.

[0055] In some embodiments, the display 260 may be at least one of a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

[0056] In some embodiments, the tuner 210 receives broadcast television signals via wired or wireless reception and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0057] In some embodiments, the communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 200 can establish the transmission and reception of control signals and data signals with the control device 100 or the server 400 through the communicator 220.

[0058] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0059] In some embodiments, the external device interface 240 may include, but is not limited to, one or more interfaces such as: High Definition Multimedia Interface (HDMI), analog or data high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by multiple interfaces mentioned above.

[0060] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0061] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations via various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object to display on the display 260, the controller 250 can perform operations related to the object selected by the user command.

[0062] In some embodiments, the object can be any of the optional objects, such as a hyperlink, an icon, or other operable controls. Operations related to the selected object include: displaying links to hyperlinked pages, documents, images, etc., or performing operations corresponding to the program associated with the icon.

[0063] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0064] A CPU (CPU) processor is used to execute operating system and camera application instructions stored in memory, as well as various interactive instructions received from external input, to perform various camera applications, data, and content, ultimately displaying and playing various audio and video content. A CPU processor may include multiple processors, such as a main processor and one or more sub-processors.

[0065] In some embodiments, a graphics processor is used to generate various graphical objects, such as at least one of icons, operation menus, and user-input command-displayed graphics. The graphics processor includes an arithmetic logic unit (ALU) that performs calculations based on various user-input interactive commands and displays various objects according to display attributes; it also includes a renderer that renders the various objects obtained from the ALU, the rendered objects being displayed on a monitor.

[0066] In some embodiments, a video processor is configured to receive an external video signal and perform at least one of the following video processing operations according to a standard encoding and decoding protocol of the input signal: decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image synthesis, etc., to obtain a signal that can be directly displayed or played on a display device 200.

[0067] In some embodiments, the video processor includes at least one of a demultiplexing module, a video decoding module, an image compositing module, a frame rate conversion module, and a display formatting module. The demultiplexing module demultiplexes the input audio and video data streams. The video decoding module processes the demultiplexed video signal, including decoding and scaling. The image compositing module, such as an image synthesizer, overlays and blends a GUI signal generated by a graphics generator based on user input or its own generation with the scaled video image to generate a displayable image signal. The frame rate conversion module converts the input video frame rate. The display formatting module modifies the received frame rate-converted video output signal to conform to a display format, such as outputting RGB data signals.

[0068] In some embodiments, an audio processor is configured to receive external audio signals, perform decompression and decoding according to a standard codec protocol of the input signals, and at least one of the following processes: noise reduction, digital-to-analog conversion, and amplification, to obtain a sound signal that can be played in a speaker.

[0069] In some embodiments, the user can input user commands through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input commands through the GUI. Alternatively, the user can input user commands by inputting specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0070] In some embodiments, a "user interface" is the medium through which a camera application or operating system interacts and exchanges information with a user, enabling the conversion between the internal form of information and a form acceptable to the user. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include at least one of the visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0071] In some embodiments, the user interface 280 is an interface that can be used to receive control input (e.g., physical buttons on the display device body, or others).

[0072] In some embodiments, the display device's system may include a kernel, a command interpreter (shell), a file system, and a camera application. The kernel, shell, and file system together form the basic operating system structure, allowing users to manage files, run programs, and use the system. Upon power-up, the kernel starts, activates the kernel space, abstracts hardware, initializes hardware parameters, and runs and maintains virtual memory, the scheduler, signals, and inter-process communication (IPC). After the kernel starts, the shell and the user's camera application are loaded. The camera application is compiled into machine code after startup, forming a process.

[0073] like Figure 4 As shown, a display device's system can include a kernel, a command interpreter (shell), a file system, and applications. The kernel, shell, and file system together form the basic operating system structure, allowing users to manage files, run programs, and use the system. Upon power-up, the kernel starts, activates the kernel space, abstracts hardware, initializes hardware parameters, and runs and maintains virtual memory, the scheduler, signals, and inter-process communication (IPC). After the kernel starts, the shell and user applications are loaded. Applications are compiled into machine code after startup, forming a process.

[0074] like Figure 4 As shown, the display device system is divided into three layers, from top to bottom: the application layer, the middleware layer, and the hardware layer.

[0075] The application layer mainly includes commonly used applications on TVs, as well as the application framework. The commonly used applications are mainly browser-based applications, such as HTML5 apps, and native apps.

[0076] An application framework is a complete program model that has all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interface for these functions (toolbar, status bar, menu, dialog box).

[0077] Native apps can support online or offline access, push notifications, or access to local resources.

[0078] The middleware layer includes various television protocols, multimedia protocols, and system components. Middleware can use the basic services (functions) provided by system software to connect different parts of application systems or different applications on the network, achieving resource sharing and function sharing.

[0079] The hardware layer mainly includes the HAL interface, hardware, and drivers. The HAL interface is a unified interface for all TV chips, with the specific logic implemented by each chip. The drivers mainly include: audio drivers, display drivers, Bluetooth drivers, camera drivers, Wi-Fi drivers, USB drivers, HDMI drivers, sensor drivers (such as fingerprint sensors, temperature sensors, pressure sensors, etc.), and power drivers.

[0080] Figure 5 A schematic diagram of the voice interaction network architecture of a display device is shown in some embodiments. For example... Figure 5 As shown, the display device 200 is used to receive input information such as sound and output the processing results of that information. The speech recognition module deploys an Automatic Speech Recognition (ASR) service to recognize audio as text; the semantic understanding module deploys a Natural Language Understanding (NLU) service to perform semantic parsing on the text; the business management module deploys a business instruction management service such as Dialog Management (DM) to provide business instructions; the language generation module deploys a Natural Language Understanding (NLG) service to convert instructions to be executed by the display device into text language; and the speech synthesis module deploys a Text-to-Speech (TTS) service to process the text language corresponding to the instructions and send it to the speaker for playback. The voice interaction network architecture can contain multiple entity service devices deploying different business services, or one or more entity service devices can combine one or more functional services.

[0081] In some embodiments, the following describes the basis Figure 5 The process of processing information input to the display device 200 in the architecture shown is described with an example, taking a query statement input via voice as an example:

[0082] Voice recognition: After receiving a query statement input by voice, the display device 200 can perform noise reduction processing and feature extraction on the audio of the query statement. The noise reduction processing may include steps such as removing echoes and environmental noise.

[0083] Semantic understanding: Natural language understanding is performed on the identified candidate text and associated contextual information. The text is parsed into structured, machine-readable information, business domain, intent, slots, etc., to express semantics, and an executable intent is determined. An intent confidence score is obtained, and the semantic understanding module selects one or more candidate executable intents based on the determined intent confidence score.

[0084] Business Management: Based on the semantic parsing results of the query statement text, the semantic understanding module sends query instructions to the corresponding business management module to obtain the query results provided by the business service, as well as the actions required to "complete" the user's final request, and feeds back the device execution instructions corresponding to the query results.

[0085] Language generation: This is configured to generate spoken text from information or instructions. Specifically, it can be divided into casual conversation, task-oriented, knowledge-based question-answering, and recommendation-oriented systems. In casual conversation, NLG (Natural Language Generation) performs intent recognition and sentiment analysis based on context, then generates open-ended responses. In task-oriented conversations, learned strategies are used to generate responses, typically including clarifying needs, guiding the user, asking questions, confirming, and closing remarks. In knowledge-based question-answering conversations, the required knowledge (knowledge, entities, fragments, etc.) is generated based on question type identification and classification, information retrieval, or text matching. In recommendation-oriented conversation systems, user interests are matched, candidate recommendations are ranked, and then recommended content is generated for the user.

[0086] Speech synthesis: The voice output configured to be presented to the user. The speech synthesis processing module synthesizes speech output based on text provided by the digital assistant. For example, the generated dialogue response is in the form of a text string. The speech synthesis module converts the text string into audible speech output.

[0087] It should be noted that, Figure 5 The architecture shown is merely an example and is not intended to limit the scope of protection of this application. Other architectures can also be used to achieve similar functions in the embodiments of this application. For example, all or part of the above process can be performed by the display device 200, which will not be elaborated here.

[0088] In some embodiments, the voice recognition function can be implemented by a sound acquisition device and a controller 250 set on the display device, and the semantic function can be implemented by the controller 250 of the display device.

[0089] Users can use control devices, such as remote controls, to control the display device 200. For example, with a smart TV, users can use the remote control to control the TV to play media or adjust the volume, thereby controlling the smart TV.

[0090] In some embodiments, the display device 200 has a voice interaction function. When the display device 200 enables the voice interaction function, the user can send voice commands to the display device 200 via voice input, thereby enabling the display device 200 to perform the corresponding functions. Therefore, the display device 200 may be equipped with a voice interaction mode.

[0091] In some embodiments, a user can send a voice interaction mode command to the display device 200 by operating a designated button on the remote control. In practical applications, a correspondence between the voice interaction mode command and the remote control button is pre-defined. For example, a voice interaction mode button is provided on the remote control. When the user touches this button, the remote control sends a voice interaction mode command to the controller 250, at which point the controller 250 controls the display device 200 to enter voice interaction mode. When the user touches the button again, the controller 250 can control the display device 200 to exit voice interaction mode.

[0092] In some embodiments, a pre-defined mapping relationship between voice interaction mode commands and multiple remote control buttons can be established. When a user touches the buttons bound to the voice interaction mode commands, the remote control issues the voice interaction mode commands. In one feasible embodiment, the buttons bound to the voice interaction mode commands are, in sequence, the directional keys (left, down, left, down). That is, the remote control only sends the voice interaction mode commands to the controller 250 when the user continuously touches the buttons (left, down, left, down) within a preset time. Using the above binding method can prevent voice interaction mode commands from being issued due to user error. This application embodiment only provides several exemplary binding relationships between voice interaction mode commands and buttons. In actual applications, the binding relationships between voice interaction mode commands and buttons can be set according to user habits, and no limitations are imposed here.

[0093] In some embodiments, the user can use the sound collector of the display device 200, such as a microphone, to send a voice interaction mode instruction to the display device 200 in the form of voice input, so as to control the display device 200 to enter the voice interaction mode. An intelligent voice system can be set in the display device 200, and the intelligent voice system can recognize the user's voice to extract the instruction content input by the user. The user can input a preset wake-up word through the microphone to start the intelligent voice system, so that the controller 250 can respond to the instruction input by the user. And input a voice interaction mode instruction within a certain period of time, so that the display device 200 enters the voice interaction mode. For example, the user can input "XX classmate" to start the intelligent voice system, and at this time the display device 200 enters the voice interaction mode.

[0094] In some embodiments, the user can also send a voice interaction mode instruction to the display device 200 through a preset gesture. The display device 200 can detect the user's behavior through an image collector, such as a camera. When the user makes a preset gesture, it can be considered that the user has sent a voice interaction mode instruction to the display device 200. For example, it can be set that when it is detected that the user draws a V shape, it is determined that the user has input a voice interaction mode instruction to the display device 200. The user can also send a voice interaction mode instruction to the display device 200 through a preset action. For example, it can be set that when it is detected that the user raises the left foot and the right hand at the same time, it is determined that the user has input a voice interaction mode instruction to the display device 200.

[0095] In some embodiments, when the user uses the display device 200 to control the display device 200, such as using a mobile phone, a voice interaction mode instruction can also be sent to the display device 200. In the actual application process, a control can be set in the mobile phone, and whether to enter the voice interaction mode can be selected through this control, so as to send a voice interaction mode instruction to the controller 250. At this time, the controller 250 can control the display device 200 to enter the voice interaction mode.

[0096] In some embodiments, when the user uses the mobile phone to control the display device 200, a continuous click instruction can be issued to the mobile phone. The continuous click instruction means that within a preset period, the number of times the user clicks on the same area of the mobile phone touch screen exceeds a preset threshold. For example, when the user continuously clicks on a certain area of the mobile phone touch screen 3 times within 1 s, it is regarded as a continuous click instruction. After the mobile phone receives the continuous click instruction, it can send a voice interaction mode instruction to the display device 200, so that the controller 250 controls the display device 200 to enter the voice interaction mode.

[0097] In some embodiments, when a user controls the display device 200 using a mobile phone, the mobile phone can be configured to send a voice interaction mode command to the display device 200 when the user's touch pressure value on a certain area of ​​the mobile phone touch screen exceeds a preset pressure threshold.

[0098] The user can also set a voice interaction mode option in the UI interface of the display device 200. When the user clicks this option, the user can control the display device 200 to enter or exit the voice interaction mode. Figure 6 Schematic diagrams of the system settings UI interface of the display device 200 in some embodiments are shown. For example... Figure 6 As shown, the system settings include screen settings, sound settings, voice interaction settings, network settings, and factory reset. Users can click the voice interaction control to control the display device 200 to enter or exit voice interaction mode.

[0099] In some embodiments, to prevent users from accidentally triggering the voice interaction mode, when the controller 250 receives a voice interaction mode instruction, it can control the display to show voice interaction mode confirmation information, thereby allowing the user to confirm whether to control the display device 200 to enter the voice interaction mode. Figure 7 A schematic diagram showing the display of voice interaction mode confirmation information on the display in some embodiments is shown.

[0100] In some embodiments, after the display device 200 enters the voice interaction mode, the user can directly send commands to the display device 200 via voice input. Upon receiving the user's voice command, the display device 200 can respond to the command and execute corresponding operations to achieve the desired function.

[0101] In some embodiments, the controller 250 can control the sound acquisition device to collect voice commands input by the user. After the sound acquisition device collects the voice commands, the controller 250 can parse the voice commands to obtain the voice text.

[0102] The controller 250 can send the received voice data to a voice recognition service, which converts it into text information, resulting in voice-to-text. The voice recognition service is a web service that can be deployed on the display device 200 and may include a voice recognition module and a semantic analysis module. The voice recognition service is used to recognize audio as text, while the semantic service is used to perform semantic parsing on the text. For example, the voice recognition module can parse user-input voice commands to recognize the voice-to-text. The semantic analysis module then performs lexical, syntactic, and semantic analysis on the voice-to-text to understand the user's intent and execute the voice command to achieve the corresponding function.

[0103] In some embodiments, the display device 200 may further include a third-party voice recognition interface. After receiving a voice command input by a user, the controller 250 may send the voice data to the third-party voice recognition interface, and use a third-party voice recognition device or the like to convert the user's voice command into a voice text. After obtaining the voice text, the controller 250 may parse the voice text and execute the voice command.

[0104] In some embodiments, the controller 250 may also send the voice command to a server. The server may generate a voice text according to the voice command and feedback the voice text to the display device.

[0105] Figure 8 The interaction flowchart of each component of the display device in some embodiments is shown.

[0106] In some embodiments, after obtaining the voice text, the controller 250 may perform semantic analysis on the voice text. The voice text is the texturized user input voice command and may be in the form of a sentence. In the embodiments of the present application, the display device 200 is introduced as a smart TV. For example, when a user searches for a movie to watch, the user may send a corresponding voice command "Search for movie XXX" to the smart TV.

[0107] The controller 250 may first perform word segmentation on the voice text to obtain a word segmentation result including multiple words. Each word is a word segmentation, so as to obtain all the word segmentations included in the voice text. In the embodiments of the present application, the obtained word segmentations are referred to as text word segmentations.

[0108] In some embodiments, for a voice text in Chinese, the controller 250 may perform word segmentation on the voice text in units of characters, so as to obtain the text word segmentation corresponding to the voice text. For example, for the voice text "I want to watch TV drama A", after performing word segmentation, the word segmentation result may be seven text word segmentations: "I, want, to, watch, TV, drama, A".

[0109] In some embodiments, based on the current multi-language voice interaction environment, the voice command input by the user may not only be Chinese, but may also be a language in English or other languages, or even a multi-language mixed form, such as a Chinese-English mixed voice command. According to their own language habits, users may send a Chinese command "Search for the third season of XXX movie", an English command "search for XXX movie", a French command "Rechercher xxx Films", or other voice commands in different languages to the smart TV.

[0110] There are significant differences in grammar habits and writing structures among different languages. For example, when writing Chinese, Japanese, and Thai, they are based on characters, and there are no spaces between characters, while English, French, etc. use words as the smallest unit, and there are spaces between words. If there are multiple languages in a sentence, for example, "Has the Dow Jones index dropped?", there are both Chinese and English in the current sentence. In order to accurately obtain the text segmentation corresponding to the speech text, the controller 250 can first identify the languages to distinguish different languages.

[0111] The controller 250 can use regular matching of Unicode encoding, word lists of each language, and the python language detection toolkit langdetect to identify the languages in the speech text. The identification process is as follows: First, the controller 250 uses regular matching of Unicode encoding to detect the original data in units of characters. Since the Unicode encodings of letters in different languages are different, the Unicode encoding can be used to roughly judge some languages, such as Chinese, Japanese, Thai, and Arabic. However, for English, French, etc. that share the Latin alphabet and have the same letter system, they cannot be distinguished by Unicode encoding. So after detecting with regular matching of Unicode encoding, if there is original data in the detection result that cannot be identified as a language due to having the same letter system, the controller 250 can then use the word lists of each language to detect the original data in units of words. Since the word lists of each language are incomplete and there are common words in each language, there are also cases where they cannot be completely distinguished in a few situations. At this time, if there is original data that cannot be identified as a language due to having the same words, the controller 250 can use the python language detection toolkit langdetect for detection and give the final detection result in combination with the previous detection results.

[0112] After detecting the languages, the controller 250 can segment them in the smallest units of each language (for example, words for English and characters for Chinese). For example, for "Has the Dow Jones index dropped?", it will be segmented into [Dow, Jones, 的, 指, 数, 跌, 了, 吗]. Each smallest unit is a text segmentation, thus obtaining all the text segmentations corresponding to the speech text.

[0113] In some embodiments, the controller 250 can also perform entity extraction on the speech text to obtain all the entities included in the speech text, which are called text entities in the embodiments of the present application.

[0114] In the embodiments of the present application, an entity is set as a noun phrase included in the speech text, such as a movie name, a person's name, etc. These entities are all existing nouns, and the controller 250 can extract all the entities corresponding to the speech text.

[0115] The controller 250 can first acquire all the words contained in the spoken text. It should be noted that the words here are different from the word segmentation in the previous steps. Word segmentation refers to the smallest unit in each language, such as a single character in Chinese, while a word can be a single character or a word composed of multiple characters. For example, in the spoken text "I want to buy game character skins," the words contained are "I," "want," "buy," "game," "character," and "skin."

[0116] The controller 250 can perform part-of-speech tagging and entity recognition on spoken text, specifically using tools such as Stanza NLP (Stanford Natural Language Processing Toolkit) or other lexical analysis tools. After obtaining the part-of-speech tag for each word, the controller 250 can filter out words with named entity and noun parts of speech and treat them as text entities.

[0117] In some embodiments, after obtaining the text entity, the controller 250 can obtain the target entity label corresponding to each text entity.

[0118] The controller 250 can first query the entity tags associated with each text entity based on a preset multilingual knowledge graph. A knowledge graph refers to a knowledge base that describes various entities or concepts that exist in the real world and their relationships.

[0119] In this embodiment, a multilingual knowledge graph can be pre-generated, including the relationships between different entities in multiple languages. The multilingual knowledge graph can be the YAGO multilingual knowledge graph.

[0120] YAGO contains both entities (such as movies, people, cities, and countries) and relationships between them (who acted in which movie, which city is located in which country, etc.). YAGO entities include names and aliases in various languages. YAGO is stored in the standard Resource Description Framework (RDF), and its data consists of triples, each containing a subject, a predicate (also called a "relation" or "attribute"), and an object. YAGO categorizes these entities into different classes, such as people and cities. These classes have inclusion relationships; for example, the city class is a subclass of the place of residence class, and the place of residence class is a subclass of the place of location class. YAGO also defines relationships between entities; for example, there can be a birthplace relationship between the entities people and locations. Multilingual knowledge graphs can utilize existing technologies, which will not be described in detail here.

[0121] When acquiring entity tags, the controller 250 can perform entity link query processing on each text entity based on a preset multilingual knowledge graph to obtain the entity tag results for each text entity.

[0122] In this context, entity linking refers to retrieving multiple other entities associated with any given target entity from a pre-defined multilingual knowledge graph. Each other entity can serve as an entity tag for the target entity. These other entities can all be used to explain the target entity or can be entities that have a relationship with it. For example, for a person A, A is the target entity, and the retrieved entity tag could be "actor," representing A's profession, or it could be a media asset name, indicating media assets that A has participated in.

[0123] When performing entity link query processing, link results can be retrieved using rule-based methods such as minimum edit distance, intersection-union ratio, entity length, and entity popularity.

[0124] Therefore, after performing entity link query processing on each text entity, we can obtain the entity label results for each text entity. The entity label results include several entity labels corresponding to each entity, as well as the classification probability of each entity label (which can also be regarded as the entity label score). This classification probability can represent the degree of association between the entity label and the text entity. The higher the degree of association, the higher the score.

[0125] The controller 250 can select a preset number of entity tags with the highest classification probability from the entity tag results as the target entity tags for each text entity. For example, in the entity tag results for each text entity, the top 5 media asset tags with the highest classification probability can be selected as the target entity tags for each text entity. If fewer than 5 entity tags are found, they can be padded with zeros to obtain 5 entity tags.

[0126] The controller 250 can analyze the semantics of the speech text based on the speech text, text segmentation, and target entity labels.

[0127] In some embodiments, controller 250 can obtain user intent and domain type based on voice text and target entity labels. Controller 250 can first obtain text vectors based on voice text.

[0128] In this embodiment of the application, a multilingual text semantic understanding model can be generated in advance.

[0129] Multilingual text semantic understanding models can be obtained based on LaBSE (Language-agnostic BERT Sentence Embedding). Pre-trained LaBSE includes word vector alignment and sentence vector alignment. For word vector alignment, a multilingual BERT (pre-trained language representation model) is trained on corpora of multiple languages. Specifically, to map the encodings of different languages ​​to the same space and achieve word vector alignment across multiple languages, a hybrid pre-training method using MMLM (Multilingual Masked Language Model) and TLM (Translation Language Model) is employed to represent different language encodings in the same semantic space. For sentence alignment, a contrastive learning approach is used to further train LaBSE on parallel multilingual corpora, aligning sentence vectors from different languages ​​in a unified semantic space. During LaBSE training, dual encoders encode the source and target languages ​​separately. The two encoders share parameters and are initialized using the parameters of a BERT model pre-trained using MMLM and TLM methods.

[0130] In some embodiments, the controller 250 establishes a multilingual text semantic understanding model based on a pre-trained LaBSE model. The LaBSE model first encodes the input text, which is then fed into the LaBSE and processed through multiple transformer layers to output the encoded results. The encoding corresponding to [CLS] represents the encoding of the entire sentence. This result, along with a softmax layer, is used for intent classification, as shown in the following formula:

[0131] y i =softmax(W i h [CLS] +b i )

[0132] In the formula, h [CLS] W represents the embedding output corresponding to [CLS]. i This corresponds to the weight matrix of the linear layer, b i y is the bias vector of the linear layer. i This is the output vector.

[0133] The encoded output of the corresponding word is combined with a softmax layer for slot prediction, as shown in the following formula:

[0134]

[0135] In the formula, h [n]W represents the embedding output of the nth word in sentence s. s This corresponds to the weight matrix of the linear layer, b s The bias vector of the linear layer. The output vector corresponding to the nth word of sentence s.

[0136] In some embodiments, the loss function during LaBSE model fine-tuning comprises two parts: the cross-entropy loss for the intention classification prediction and the true classification, L. I The cross-entropy loss between the predicted and actual slot locations is L. s The total loss is L = L I +L s By fine-tuning the training, the total loss can be minimized.

[0137] In some embodiments, the controller 250 can encode the speech text based on a preset multilingual text semantic understanding model to obtain a text vector. This text vector is the vector corresponding to the entire speech text sentence and can represent the overall user intent of the speech text.

[0138] In some embodiments, after obtaining the target entity label for each text entity, the controller 250 can select a preset number of target entity labels from all the target entity labels, referred to as the first target entity labels in this embodiment. For example, five first target entity labels may be selected.

[0139] The controller 250 can select the five target entity labels with the highest classification probability from all target entity labels as the first target entity labels.

[0140] In some embodiments, considering the accuracy of semantic analysis and aiming to represent the user's intent across the entire speech text, as many text entities as possible are needed, each represented by its own target entity label. Therefore, the controller 250 can filter out the target entity labels with the highest classification probability from the target entity labels corresponding to each text entity. For example, if the speech text contains six text entities, the controller 250 can obtain the target entity labels with the highest classification probability for each of these six text entities. After obtaining these target entity labels, the controller can determine the relationship between the number of these target entity labels and a preset number.

[0141] If the number of target entity labels is less than the preset number, for example, if the speech text contains 4 text entities, resulting in 4 target entity labels, then the preset number of 5 has not yet been reached. Therefore, it is necessary to continue selecting target entity labels from the target entity labels other than these 4. This can be done by selecting the remaining target entity labels with the highest classification probability, until the preset number of target entity labels is obtained, and these are then used as the first target entity labels.

[0142] If the number of these target entity tags is exactly equal to the preset number, then these target entity tags are directly used as the first target entity tags.

[0143] If the number of these target entity labels exceeds a preset number, for example, if the speech text contains 6 text entities, resulting in 6 target entity labels, this exceeds the preset number of 5. The controller 250 can further filter these target entity labels, either by selecting the 5 target entity labels with the highest classification probability as the first target entity labels, or by directly using these 6 target entity labels as the first target entity labels.

[0144] In some embodiments, after obtaining the first target entity label, the controller 250 can obtain the first target entity label vector corresponding to the first target entity label.

[0145] The controller 250 can encode each first target entity label based on a preset multilingual text semantic understanding model to obtain the first target entity label vector corresponding to each first target entity label.

[0146] Alternatively, the controller 250 can perform one-hot encoding on the first target entity tag based on a preset entity tag type to obtain a first target entity tag vector. Considering that the number of entity tags involved in the control commands issued by the user when using the display device 200 is limited—for example, when a user searches for media assets, it may involve entity tags such as media asset name, actors, directors, and release dates—these entity tags are generally the most frequently used tags during user searches. Therefore, the entity tags that the voice commands may involve can be pre-calculated as preset entity tag types. For each obtained first target entity tag, the controller 250 can perform one-hot encoding, marking the correct entity tag type as 1 and the incorrect entity tag type as 0, thereby obtaining the first target entity tag vector corresponding to each first target entity tag.

[0147] In some embodiments, the controller 250 may also acquire the current device status of the display device 200. The device status may include whether the interface on the display is on the home screen, whether the display device 200 is playing music, whether the display device 200 is playing video, and whether a certain app is open, etc.

[0148] The controller 250 can statistically display the current device status information of the device 200. The controller 250 can further obtain a device status vector based on the device status.

[0149] The controller 250 can perform one-hot encoding on the device state based on a preset device state type to obtain a device state vector. The controller 250 can pre-set multiple possible device state types for the display device and perform one-hot encoding on the current device state of the display device 200, marking the current device state as 1 and the non-existent device state as 0.

[0150] In some embodiments, the controller 250 can obtain user intent and domain classification information based on text vectors, first target entity label vectors, and device state vectors, respectively.

[0151] The controller 250 can first concatenate the text vector, the first target entity label vector, and the device state vector to obtain the first concatenated vector.

[0152] The controller 250 can obtain the user intent probability distribution and the domain classification probability distribution based on the first concatenation vector.

[0153] Specifically, multilingual text semantic understanding models can include a softmax-based output layer. The softmax layer can be a softmax function, which normalizes a numerical vector into a probability distribution vector, where the sum of the probabilities is 1. Therefore, the softmax layer can be used for user intent and domain classification.

[0154] User intent can be one of several predefined intent types, representing user commands or actions, such as media asset search or volume adjustment. Domain refers to the business domain corresponding to the text; the domains supported by each display device may differ.

[0155] It is possible to predict in advance all user intentions and areas that user commands may involve.

[0156] The controller 250 can process the first concatenated vector according to the multilingual text semantic understanding model. It can use the Softmax layer to obtain the user intent probability distribution of the first concatenated vector in all user intents, as well as the domain classification probability distribution of the first concatenated vector in all domains.

[0157] The controller 250 can also use other probability distribution algorithms to process the first concatenated vector, such as using a logistic regression algorithm to process the first concatenated vector to obtain the probability distribution of user intent and domain classification.

[0158] The controller 250 can determine the user intent with the highest probability in the user intent probability distribution as the user intent of the speech text, and determine the domain type with the highest probability in the domain classification probability distribution as the domain type of the speech text.

[0159] In some embodiments, the controller 250 can obtain slot filling information based on text segmentation and target entity labels.

[0160] The controller 250 can obtain text segmentation vectors based on text segmentation and obtain target entity label vectors based on target entity labels.

[0161] The controller 250 can encode each text segment based on a preset multilingual text semantic understanding model to obtain the text segmentation vector corresponding to each text segment.

[0162] The steps for obtaining the target entity label vector based on the target entity label can be referred to above, and will not be repeated here.

[0163] It should be noted that each text segment is the smallest unit in a language, such as each character in Chinese. The target entity label, on the other hand, corresponds to each text entity, that is, each word. Therefore, we can first establish the relationship between text segments and target entity labels.

[0164] The controller 250 can determine the text entity to which each text segment belongs, thereby determining the target entity label corresponding to each text segment. The controller 250 can also calculate the target entity label vector for each text segment. For example, in the speech text "I want to buy game character skins," the text segments "I," "want," and "buy" do not have corresponding entities, and therefore no corresponding target entity labels, so zero padding is applied. The text segments "game" and "play" both correspond to the entity "game," so the target entity label for the entity "game" is matched. The text segments "character" and "color" both correspond to the entity "role," so the target entity label for the entity "role" is matched. The text segments "skin" and "skin" both correspond to the entity "skin," so the target entity label for the entity "skin" is matched.

[0165] For each text segment, the controller 250 can concatenate the text segment vector of the text segment and the target entity tag vector of its corresponding target entity tag to obtain the concatenated vector of each text segment, which is referred to as the second concatenated vector in this embodiment.

[0166] In some embodiments, the controller 250 can obtain slot filling information for each text segment based on the second concatenation vector.

[0167] The controller 250 can first obtain the slot parameter probability distribution for each text segment. The controller 250 can then process each second concatenation vector based on the Softmax layer of the multilingual text semantic understanding model to obtain the slot parameter probability distribution for each second concatenation vector, which is the slot parameter probability distribution for each text segment.

[0168] Based on user control commands to various display devices 200, all possible slot parameters can be pre-calculated. For example, for a television, slot parameters could include actor, media title, volume, etc. For an air conditioner, slot parameters could include temperature, mode, etc.

[0169] After obtaining the probability distribution of slot parameters for each text segment, the controller 250 can determine the slot parameter with the highest probability in the probability distribution as the slot filling information for each text segment.

[0170] In this embodiment of the application, three types are set for each slot: B, I, and O. Among them, slot B indicates that the slot parameter corresponding to the slot is the beginning of an entity, slot I indicates that the slot parameter corresponding to the slot is the subsequent part of an entity, and slot O indicates that the slot parameter corresponding to the slot is not an entity, but generally a verb or other part of speech. Figure 9 A schematic diagram of slot filling information is shown in some embodiments. For example... Figure 9 As shown, the audio text is "Search for Tom Hanks'spider man threeon youtube". There are a total of 9 text segments, represented by h1-h9 from beginning to end. h1 is the text segment "Search", corresponding to slot O, with no entity. h2 is the text segment "for", corresponding to slot O, with no entity. h3 is the text segment "Tom", corresponding to slot B, with the entity "Tom Hanks", and its slot parameter is "actor". h4 is the text segment "Hanks'", corresponding to slot I, with the entity "Tom Hanks", and its slot parameter is "actor". h5 is the text segment "spider", corresponding to slot B, with the entity "spider man", and its slot parameter is "title". h6 is the text segment "man", corresponding to slot I, with the entity "spider man", and its slot parameter is "title". h7 represents the text segment "three", corresponding to slot B, with the entity "three". The slot parameter for h7 is "season". h8 represents the text segment "on", corresponding to slot O, with no entity. h9 represents the text segment "youtube", corresponding to slot B, with the entity "youtube". The slot parameter for h9 is "search Target".

[0171] In some embodiments, after obtaining user intent, domain type, and slot filling information, the controller 250 can encapsulate these data into encapsulated information. This encapsulated information can be used to characterize the semantics of the speech text.

[0172] The controller 250 can directly generate control commands based on the encapsulation information and control the display device to execute the control commands in order to achieve the corresponding functions and meet user needs.

[0173] Alternatively, the controller 250 can send the encapsulation information to a server, which can then generate control commands based on the encapsulation information and send them to the display device. Upon receiving the control commands from the server, the controller 250 can control the display device to execute the commands, thereby meeting the user's needs.

[0174] In some embodiments, the step of semantic understanding of user-input voice commands may also be performed by the server. Figure 10 The diagram illustrates the interaction flowcharts between the display device and the server in some embodiments. For example... Figure 10 As shown, users can input voice commands into the display device. The display device can then directly send the voice commands to the server. The server can parse the voice commands to obtain voice text. The server can then obtain text segmentation and text entities based on the voice text. Text segmentation refers to all words obtained after segmenting the voice text, and text entities refer to all entities obtained after entity extraction from the voice text. The server can also obtain the target entity tags corresponding to the text entities and the device status of the display device. Based on the voice text, target entity tags, and device status, the server can obtain the user intent and domain type, and obtain slot filling information based on text segmentation and target entity tags. The server can encapsulate the user intent, domain type, and slot filling information into encapsulated parameters to obtain encapsulated information, which is used to characterize the semantics of the voice text. The server can then send the encapsulated information to the display device. The display device can generate and execute control commands based on the encapsulated information.

[0175] Figure 11 The illustrations show scenarios of voice interaction between users and display devices in some embodiments. For example... Figure 11 As shown, when a user inputs the voice command "Search for XXX movie season 3," the display device 200 sends the voice command to the controller 250. The controller 250 then sends a control command back to the display device 200, which is used to search for relevant media resources. The display device 200 can execute this control command and prompt the user via voice, "Videos about XXX have been recommended for you."

[0176] In some embodiments, the display device 200 may display a search interface for media assets that a user wants to search for. Figure 12 A schematic diagram of a display device showing a search interface is shown in some embodiments.

[0177] When a user selects a target media asset, the display device 200 can display the media asset details page of the target media asset. Figure 13 The following are schematic diagrams of media asset details pages in some embodiments, such as Figure 13 As shown, the media asset details page may include a video preview window to display the video footage of the target media asset; a media asset introduction, including media asset type and cast and crew information, etc.; a playlist to display the number of episodes of the media asset; and playback controls, i.e. Figure 13 The "Full Screen Playback" option; the related recommendations area, used to display other media assets. Users can touch the playback controls to control the display device 200 to display the target media asset in full screen.

[0178] If the display device 200 does not find any relevant media resources, it can also display a preset prompt message to inform the user that no relevant media resources were found. Figure 14 A schematic diagram showing a display device 200 displaying prompt information is shown in some embodiments.

[0179] This application also provides a semantic understanding method applied to a display device, such as... Figure 15 As shown, the method includes:

[0180] Step 1501: Obtain the voice text, which is obtained by parsing the received voice command.

[0181] Step 1502: Obtain text segmentation and text entities based on speech text. Text segmentation refers to all the words obtained after segmenting the speech text, and text entities refer to all the entities obtained after extracting entities from the speech text.

[0182] Step 1503: Obtain the target entity label corresponding to the text entity.

[0183] Step 1504: Obtain user intent and domain type based on voice text and target entity labels, and obtain slot filling information based on text segmentation and target entity labels.

[0184] Step 1505: Perform parameter encapsulation on user intent, domain type, and slot filling information to obtain encapsulated information, which is used to characterize the semantics of speech text.

[0185] In some embodiments, obtaining voice text includes:

[0186] The system controls a voice acquisition device to collect user-input voice commands; it then parses the voice commands to obtain speech-to-text. Alternatively, the voice commands can be sent to a server, which then generates speech-to-text and sends it back to the display device.

[0187] In some embodiments, obtaining the entity tag corresponding to the text entity includes:

[0188] Based on a pre-defined multilingual knowledge graph, entity link query processing is performed on text entities to obtain entity label results for the text entities. The entity label results include several entity labels and the classification probability of the entity labels. Among the entity label results, a pre-defined number of entity labels with the highest classification probability are selected as the target entity labels for the text entities.

[0189] In some embodiments, user intent and domain classification information are obtained based on voice text and target entity labels, including:

[0190] The text vector is obtained based on the voice text; a preset number of first target entity labels are selected from the target entity labels, and the first target entity label vector of the first target entity labels is obtained; the current device status of the display device is obtained, and the device status vector is obtained based on the device status; based on the text vector, the first target entity label vector, and the device status vector, the user intent and domain classification information are obtained.

[0191] In some embodiments, obtaining text vectors based on speech text includes:

[0192] Based on a pre-defined multilingual text semantic understanding model, speech text is encoded to obtain text vectors.

[0193] In some embodiments, obtaining the first target entity label vector of the first target entity label includes:

[0194] Based on a pre-defined multilingual text semantic understanding model, the first target entity label is encoded to obtain the first target entity label vector; or, based on a pre-defined entity label type, the first target entity label is one-hot encoded to obtain the first target entity label vector.

[0195] In some embodiments, obtaining a device state vector based on device state includes:

[0196] The device state is one-hot encoded based on the preset device state type to obtain the device state vector.

[0197] In some embodiments, obtaining user intent and domain type includes:

[0198] The text vector, the first target entity label vector, and the device state vector are concatenated to obtain the first concatenated vector; the user intent probability distribution and the domain classification probability distribution are obtained based on the first concatenated vector; the user intent with the highest probability in the user intent probability distribution is determined as the user intent of the speech text, and the domain type with the highest probability in the domain classification probability distribution is determined as the domain type of the speech text.

[0199] In some embodiments, slot filling information is obtained based on text segmentation and target entity tags, including:

[0200] Obtain text segmentation vectors based on text segmentation; obtain target entity label vectors based on target entity labels; count the target entity label vectors corresponding to the target entity labels of the text segmentation; concatenate the text segmentation vectors and the corresponding target entity label vectors to obtain the second concatenated vector of the text segmentation; obtain the slot filling information of the text segmentation based on the second concatenated vector.

[0201] In some embodiments, obtaining slot filling information for text segmentation includes:

[0202] Obtain the probability distribution of slot parameters in text segmentation; determine the slot parameter with the highest probability in the probability distribution as the slot filling information for text segmentation.

[0203] In some embodiments, it also includes:

[0204] Control commands are generated and executed based on the encapsulation information; alternatively, the encapsulation information is sent to the server so that the server generates control commands based on the encapsulation information and sends them back to the display device.

[0205] The same or similar parts among the various embodiments in this specification can be referred to mutually, and will not be repeated here.

[0206] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or certain parts of the embodiments of the present invention.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0208] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the embodiments and various different variations of embodiments suitable for specific application considerations.

Claims

1. A display device, characterized in that, include: monitor; A voice acquisition device is configured to capture voice commands input by the user. The controller is configured as follows: Get speech text; Control the sound acquisition device to collect the user's voice commands; The voice command is parsed to obtain voice text; based on the voice text, text segmentation and text entities are obtained, wherein the text segmentation is obtained by segmenting the voice text, and the text entities are obtained by extracting entities from the voice text. Obtain the target entity label corresponding to the text entity; User intent and domain type are obtained based on the voice text and the target entity tags; slot filling information is obtained based on the text segmentation and the target entity tags. The user intent, the domain type, and the slot filling information are encapsulated with parameters to obtain encapsulation information, which is used to characterize the semantics of the speech text. The controller is configured to perform actions based on the spoken text and the target entity tags to obtain user intent and domain classification information, and is also configured to: Obtain text vectors based on the spoken text; Select a preset number of first target entity tags from the target entity tags, and obtain the first target entity tag vector of the first target entity tags; Obtain the current device state of the display device, and obtain the device state vector based on the device state; Based on the text vector, the first target entity label vector, and the device state vector, user intent and domain classification information are obtained.

2. The display device according to claim 1, characterized in that, Also includes: A voice acquisition device is configured to capture voice commands input by the user. The controller is configured to acquire voice text and is also configured to: Control the sound acquisition device to collect the user's voice commands; Alternatively, the voice command can be sent to a server, which will then generate voice text based on the voice command and send it back to the display device.

3. The display device according to claim 1, characterized in that, The controller is configured to retrieve the entity tag corresponding to the text entity and is also configured to: Based on a pre-defined multilingual knowledge graph, entity link query processing is performed on the text entity to obtain the entity tag result of the text entity. The entity tag result includes several entity tags and the classification probability of the entity tags. In the entity label results, a preset number of entity labels with the highest classification probability are selected as the target entity labels of the text entity.

4. The display device according to claim 1, characterized in that, The controller is configured to perform text vector acquisition based on the spoken text and is also configured to: Based on a pre-defined multilingual text semantic understanding model, the speech text is encoded to obtain a text vector; The controller is configured to execute the acquisition of the first target entity tag vector of the first target entity tag, and is also configured to: Based on a preset multilingual text semantic understanding model, the first target entity label is encoded to obtain a first target entity label vector; or, based on a preset entity label type, the first target entity label is one-hot encoded to obtain a first target entity label vector. The controller is configured to obtain a device state vector based on the device state and is also configured to: The device state is one-hot encoded based on a preset device state type to obtain a device state vector.

5. The display device according to claim 1, characterized in that, The controller is configured to retrieve user intent and domain type, and is also configured to: The text vector, the first target entity label vector, and the device state vector are concatenated to obtain the first concatenated vector; Based on the first concatenated vector, obtain the user intent probability distribution and the domain classification probability distribution; The user intent with the highest probability in the user intent probability distribution is determined as the user intent of the voice text, and the domain type with the highest probability in the domain classification probability distribution is determined as the domain type of the voice text.

6. The display device according to claim 1, characterized in that, The controller is configured to obtain slot filling information based on the text segmentation and the target entity tags, and is also configured to: Obtain text segmentation vectors based on the text segmentation; obtain target entity tag vectors based on the target entity tags; and calculate the target entity tag vectors corresponding to the target entity tags in the text segmentation. The text segmentation vector and the corresponding target entity tag vector are concatenated to obtain the second concatenated vector of the text segmentation. Based on the second concatenation vector, the slot filling information of the text segmentation is obtained.

7. The display device according to claim 6, characterized in that, The controller is configured to retrieve the slot filling information for the text segmentation and is also configured to: Obtain the probability distribution of slot parameters for the text segmentation; The slot parameter with the highest probability in the probability distribution of the slot parameters is determined as the slot filling information for the text segmentation.

8. The display device according to claim 1, characterized in that, The controller is also configured to: Control instructions are generated based on the encapsulation information, and the control instructions are executed. Alternatively, the encapsulation information can be sent to a server, so that the server can generate control commands based on the encapsulation information and feed them back to the display device.

9. A semantic understanding method applied to a display device, characterized in that, The method includes: Get speech text; Based on the speech text, text segmentation and text entities are obtained. The text segmentation is the segmentation obtained after segmenting the speech text, and the text entities are the entities obtained after extracting entities from the speech text. Obtain the target entity label corresponding to the text entity; User intent and domain type are obtained based on the voice text and the target entity tags; slot filling information is obtained based on the text segmentation and the target entity tags. The user intent, the domain type, and the slot filling information are encapsulated with parameters to obtain encapsulation information, which is used to characterize the semantics of the speech text. Obtain text vectors based on the spoken text; Select a preset number of first target entity tags from the target entity tags, and obtain the first target entity tag vector of the first target entity tags; Obtain the current device state of the display device, and obtain the device state vector based on the device state; Based on the text vector, the first target entity label vector, and the device state vector, user intent and domain classification information are obtained.

Images