Dialogue state tracking method, intelligent device and service device

Abstract

This application discloses a conversation state tracking method, intelligent device, and service device. The intelligent device includes an input interface and a controller. The controller is configured to: send a current query statement input through the input interface to a service device, so that the service device determines whether to perform conversation segmentation on the current query statement and the previous query statement based on the time interval between the current query statement and the previous query statement, the similarity between the current query statement and the previous query statement, and the location domain and score of the current query statement; segment the user's human-computer interaction conversation based on the judgment result of the conversation segmentation to obtain at least one segmented conversation statement corresponding to the user; and process each conversation statement using a preset deep learning model to obtain conversation state information corresponding to each user conversation statement, thereby improving the accuracy of the output conversation state information and thereby improving the accuracy of the machine output content.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and more specifically, to a method for tracking conversation status, an intelligent device, and a service device. Background Art

[0002] With the rapid development of the information age, flexible and fast human-computer interaction has gradually entered people's lives. The task of dialog state tracking is an important part of the human-computer dialog system and the basis for the system to make decisions. Therefore, state tracking has important research significance.

[0003] Dialogue State Tracking (DST) is an important issue in the field of human-computer dialogue. Its purpose is to identify the user's dialogue state (intention and slot information) at the current moment and use the results as the basis for action generation in the dialogue system.

[0004] Currently, the existing dialogue state tracking process is usually performed on the results of spoken language comprehension, which has the problem of mistransmission, resulting in the mixing of user's previous and subsequent query information; furthermore, during the human-computer dialogue process, it is impossible to clearly define the user's conversation scope, resulting in the introduction of invalid content in the dialogue state tracking process, leading to dialogue state recognition errors and inaccurate machine predictions. Summary of the Invention

[0005] The embodiments of the present application provide a conversation state tracking method, an intelligent device, and a service device to overcome the problem that, during a human-computer conversation, the user's conversation scope cannot be clearly defined, invalid content is introduced during the conversation state tracking process, conversation state recognition errors occur, and the machine-predicted content is inaccurate.

[0006] In a first aspect, a smart device is provided, comprising: an input interface and a controller; the controller being configured to:

[0007] Sending the query statement of the current round input through the input interface to the service device, so that the service device segments the user's human-computer interaction conversation according to a preset rule, obtains at least one segmented conversation statement corresponding to the user, and processes each conversation statement to obtain dialogue state information corresponding to each conversation statement of the user;

[0008] The session state information is acquired from the service device and outputted.

[0009] In a second aspect, a service device is provided, comprising:

[0010] An acquisition unit, configured to acquire the time interval between the user's current query and the previous query, the similarity between the current query and the previous query, and the location domain of the current query and its score;

[0011] a determination unit, configured to determine whether to perform session segmentation on the query statements in the current round and the query statements in the previous round based on the time interval between the query statements in the current round and the query statements in the previous round, the similarity between the query statements in the current round and the query statements in the previous round, and the location domain of the query statements in the current round and the score thereof;

[0012] a segmentation unit, configured to segment the user's human-computer interaction session according to a determination result of the session segmentation, and obtain at least one segmented session sentence corresponding to the user;

[0013] The processing unit is used to process each conversation sentence using a preset deep learning model to obtain the dialogue state information corresponding to each conversation sentence of the above-mentioned user.

[0014] In a third aspect, a method for tracking a conversation state is provided, which is applied to a smart device and includes:

[0015] The current round query statement input through the above-mentioned input interface is sent to the service device, so that the service device obtains the time interval between the user's current round query statement and the previous round query statement, the similarity between the current round query statement and the previous round query statement, and the positioning domain and score of the current round query statement; based on the time interval between the current round query statement and the previous round query statement, the similarity between the current round query statement and the previous round query statement, and the positioning domain and score of the current round query statement, it is determined whether to perform session segmentation on the current round query statement and the previous round query statement; based on the judgment result of the session segmentation, the human-computer interaction session of the above-mentioned user is segmented to obtain at least one segmented session statement corresponding to the above-mentioned user; and using a preset deep learning model to process each session statement to obtain dialogue state information corresponding to each session statement of the above-mentioned user;

[0016] The session status information is obtained from the service device and output.

[0017] In a fourth aspect, a method for tracking a conversation state is provided, which is applied to a service device and includes:

[0018] Obtain the time interval between the user's current query and the previous query, the similarity between the current query and the previous query, and the positioning domain and score of the current query;

[0019] Determine whether to perform session segmentation on the query statements in this round and the query statements in the previous round based on the time interval between the query statements in this round and the query statements in the previous round, the similarity between the query statements in this round and the query statements in the previous round, and the positioning domain and score of the query statements in this round;

[0020] Segmenting the user's human-computer interaction session according to the session segmentation judgment result to obtain at least one segmented session sentence corresponding to the user;

[0021] Using a preset deep learning model, each conversation sentence is processed to obtain the dialogue state information corresponding to each conversation sentence of the above user.

[0022] In the above embodiments, the embodiments of the present application can effectively utilize the context information of user query statements, eliminate irrelevant query statements, and realize the division of user conversation statements. After the divided conversation statements are input into the preset deep learning model, the accuracy of the output intent and slot information can be improved, thereby improving the accuracy of the machine output content. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the implementation methods in the embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0024] Figure 1 is a schematic diagram exemplarily illustrating an operation scenario between a display device and a control apparatus according to some embodiments;

[0025] Figure 2 is a block diagram exemplarily illustrating a hardware configuration of a display device according to some embodiments;

[0026] Figure 3 is a block diagram exemplarily illustrating a hardware configuration of a control device according to some embodiments;

[0027] Figure 4 is a schematic diagram illustrating software configuration in a display device according to some embodiments;

[0028] Figure 5 This is a schematic diagram showing an icon control interface display of an application in a display device according to some embodiments;

[0029] Figure 6 This is a schematic diagram illustrating an exemplary application of a display device in a voice interaction scenario;

[0030] Figure 7 This is a flow chart showing an exemplary application of a display device in a voice interaction scenario;

[0031] Figure 8 This is a schematic diagram illustrating another application of a display device in a voice interaction scenario;

[0032] Figure 9This is another flowchart illustrating an exemplary application of a display device in a voice interaction scenario;

[0033] Figure 10 A schematic diagram of an exemplary identification model being issued by a supplier of the identification model;

[0034] Figure 11 A schematic diagram of a flow chart showing an exemplary process of obtaining a recognition model by a server;

[0035] Figure 12 A schematic diagram of a flow chart showing an exemplary process of updating a recognition model by a server;

[0036] Figure 13 A schematic diagram of a system architecture for illustrative purposes of a conversation state tracking embodiment of the present application;

[0037] Figure 14 The process diagram of the conversation state tracking method provided by the embodiment of the present application is shown as an example Figure 1 ;

[0038] Figure 15 The process diagram of the conversation state tracking method provided by the embodiment of the present application is shown as an example Figure 2 ;

[0039] Figure 16 A schematic diagram illustrating the principle of a preset deep learning model provided in an embodiment of the present application;

[0040] Figure 17 The process diagram of the conversation state tracking method provided by the embodiment of the present application is shown as an example Figure 3 ;

[0041] Figure 18 The figure is a schematic diagram of the structure of the service equipment provided by the embodiment of the present application for exemplary purposes. DETAILED DESCRIPTION

[0042] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0043] All other embodiments derived by persons of ordinary skill in the art based on the exemplary embodiments described herein without inventive effort are within the scope of protection of the claims appended hereto. Furthermore, although the disclosure herein is presented based on one or more exemplary embodiments, it should be understood that each aspect of the disclosure may constitute a complete embodiment on its own.

[0044] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0045] In the specification and claims of this application and the drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or precedence, unless otherwise indicated. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the drawings or descriptions of the embodiments of this application.

[0046] In addition, the terms "comprise" and "have" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0047] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0048] As used herein, the term "remote control" refers to a component of an electronic device (such as the display device disclosed herein) that is capable of wirelessly controlling the electronic device, typically over a short distance. It typically connects to the electronic device using infrared and / or radio frequency (RF) signals and / or Bluetooth, and may also include functional modules such as WiFi, wireless USB, Bluetooth, and motion sensors. For example, a handheld touch remote control replaces most of the physical built-in hard keys found in conventional remote controls with a touchscreen user interface.

[0049] The term "gesture" used in this application refers to a user's behavior of expressing an intended idea, action, purpose, or result through a change in hand shape or hand movement.

[0050] Figure 1 FIG2 is a schematic diagram showing an operation scenario between a display device and a control device according to an embodiment. Figure 1 As shown in FIG, a user can operate the display device 200 through a mobile terminal 1002 and a control apparatus 1001 .

[0051] In some embodiments, the control device 1001 may be a remote control. Communication between the remote control and the display device may include infrared protocol communication, Bluetooth protocol communication, or other short-range communication methods, thereby controlling the display device 200 wirelessly or through wired methods. A user may control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, and the like. For example, a user may input corresponding control commands through the volume up / down keys, channel control keys, up / down / left / right movement keys, voice input keys, menu keys, power on / off keys, and the like on the remote control to control the display device 200.

[0052] In some embodiments, a mobile terminal, tablet computer, computer, laptop computer, or other smart device can also be used to control the display device 200. For example, an application running on the smart device can be used to control the display device 200. The application can be configured to provide the user with various controls in an intuitive user interface (UI) on a screen associated with the smart device.

[0053] In some embodiments, mobile terminal 1002 can install software applications on display device 200 and establish communication via a network communication protocol, enabling one-to-one control and data communication. For example, a control command protocol can be established between mobile terminal 1002 and display device 200, allowing a remote control keyboard to be synchronized with mobile terminal 1002, and controlling the user interface on mobile terminal 1002 to control display device 200. Alternatively, audio and video content displayed on mobile terminal 1002 can be transmitted to display device 200 for synchronized display.

[0054] like Figure 1 It is also shown that the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN) and other networks. The server 400 may provide various content and interactions to the display device 200. For example, the display device 200 receives software program updates or accesses a remotely stored digital media library by sending and receiving information and interacting with an electronic program guide (EPG). The server 400 may be a cluster or multiple clusters, and may include one or more types of servers. Other network service content such as video on demand and advertising services is provided through the server 400.

[0055] The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. The specific display device type, size, and resolution are not limited. Those skilled in the art will appreciate that the display device 200 can be modified in terms of performance and configuration as needed.

[0056] In addition to providing broadcast reception television functions, the display device 200 may also provide intelligent network television functions with computer support functions, including but not limited to network television, smart TV, Internet Protocol television (IPTV), etc.

[0057] Figure 2 exemplarily shows a hardware configuration block diagram of the display device 200 according to an exemplary embodiment.

[0058] In some embodiments, the display device 200 includes at least one of a controller 250, a tuner and demodulator 210, a communicator 220, a detector 230, an input / output interface 255, a display 275, an audio output interface 285, a memory 260, a power supply 290, a user interface 265, and an external device interface 240.

[0059] In some embodiments, the display 275 is a component for receiving the image signal output from the first processor and displaying video content and images as well as a menu control interface.

[0060] In some embodiments, the display 275 includes a display screen component for presenting images and a driving component for driving image display.

[0061] In some embodiments, the displayed video content may be from broadcast television content, or various broadcast signals received via wired or wireless communication protocols, or various image content received from a network server via a network communication protocol.

[0062] In some embodiments, the display 275 is used to present a user manipulation UI interface generated in the display device 200 and used to control the display device 200 .

[0063] In some embodiments, depending on the type of display 275 , a driving component for driving the display is also included.

[0064] In some embodiments, the display 275 is a projection display and may further include a projection device and a projection screen.

[0065] In some embodiments, the communicator 220 is a component for communicating with external devices or external servers according to various communication protocols. For example, the communicator may include at least one of a Wi-Fi chip, a Bluetooth communication protocol chip, a wired Ethernet communication protocol chip, or other network communication protocol chip or a near-field communication protocol chip, as well as an infrared receiver.

[0066] In some embodiments, the display device 200 may establish transmission and reception of control signals and data signals with the external control device 1001 or the content providing device through the communicator 220 .

[0067] In some embodiments, the user interface 265 may be configured to receive infrared control signals from the control device 1001 (eg, an infrared remote controller, etc.).

[0068] In some embodiments, the detector 230 is a signal used by the display device 200 to collect external environment or interact with the outside.

[0069] In some embodiments, the detector 230 includes a light receiver, a sensor for collecting ambient light intensity, and can adaptively display parameter changes by collecting ambient light.

[0070] In some embodiments, the image collector 232 in the detector 230, such as a camera, a camera, etc., can be used to collect external environment scenes, as well as to collect user attributes or gestures for interacting with the user. It can adaptively change display parameters and recognize user gestures to realize the function of interacting with the user.

[0071] In some embodiments, the detector 230 may further include a temperature sensor, such as by sensing the ambient temperature.

[0072] In some embodiments, the display device 200 can adaptively adjust the color temperature of the image displayed. For example, when the temperature is high, the display device 200 can be adjusted to display a cooler color temperature image, or when the temperature is low, the display device 200 can be adjusted to display a warmer color temperature image.

[0073] In some embodiments, the detector 230 may further include a sound collector 231, such as a microphone, which can be used to collect voice data. When a user speaks a command via voice, the microphone can collect voice data including the user's command. For example, the sound collector 231 can collect voice signals including control commands for controlling the display device 200 by the user, or collect ambient sounds for identifying the type of ambient scene, so that the display device 200 can adaptively adapt to ambient noise.

[0074] In some embodiments, as Figure 2 As shown, the input / output interface 255 is configured to perform data transmission between the controller 250 and other external devices or other controllers 250, such as receiving video signal data and audio signal data, or command instruction data from external devices.

[0075] In some embodiments, the external device interface 240 may include, but is not limited to, any one or more interfaces such as a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface, a composite video input interface, a USB input interface, an RGB port, etc. Alternatively, multiple interfaces may be combined to form a composite input / output interface.

[0076] In some embodiments, as Figure 2 As shown, the tuner-demodulator 210 is configured to receive broadcast television signals through wired or wireless reception, and can perform modulation and demodulation processing such as amplification, mixing and resonance, and demodulate audio and video signals from multiple wireless or wired broadcast television signals. The audio and video signals may include television audio and video signals carried in the television channel frequency selected by the user, as well as EPG data signals.

[0077] In some embodiments, the frequency demodulated by the tuner-demodulator 210 is controlled by the controller 250, which can send a control signal based on the user's selection to enable the modem to respond to the TV signal frequency selected by the user and modulate and demodulate the TV signal carried by the frequency.

[0078] In some embodiments, broadcast television signals can be classified as terrestrial broadcast signals, cable broadcast signals, satellite broadcast signals, or Internet broadcast signals based on the broadcast format of the television signal. Alternatively, they can be classified as digital modulation signals, analog modulation signals, etc. based on the modulation type. Alternatively, they can be classified as digital signals, analog signals, etc. based on the type of signal.

[0079] In some embodiments, the controller 250 and the tuner / demodulator 210 may be located in different separate devices, that is, the tuner / demodulator 210 may be located in an external device of the main device where the controller 250 is located, such as an external set-top box. In this way, the set-top box modulates and demodulates the received broadcast television signal into television audio and video signals and outputs them to the main device, which then receives the audio and video signals via the first input / output interface.

[0080] In some embodiments, controller 250 controls the operation of the display device and responds to user operations via various software control programs stored in memory. Controller 250 can control the overall operation of display device 200. For example, in response to receiving a user command to select a UI object for display on display 275, controller 250 can perform operations related to the object selected by the user command.

[0081] In some embodiments, the object can be any one of selectable objects, such as a hyperlink or an icon. Operations related to the selected object, for example, displaying a page, document, image connected to the hyperlink, or executing a program corresponding to the icon. User commands for selecting UI objects can be commands input through various input devices connected to the display device 200 (e.g., a mouse, keyboard, touchpad, etc.) or voice commands corresponding to voice spoken by the user.

[0082] like Figure 2As shown, the controller 250 includes at least one of a random access memory 251 (RAM), a read-only memory 252 (ROM), a video processor 270, an audio processor 280, other processors 253 (e.g., a graphics processing unit (GPU)), a central processing unit (CPU) 254, a communication interface (Communication Interface), and a communication bus 256 (Bus). The communication bus connects the various components.

[0083] In some embodiments, RAM 251 is used to store temporary data for the operating system or other running programs.

[0084] In some embodiments, ROM 252 is used to store various system startup instructions.

[0085] In some embodiments, ROM 252 is used to store a basic input / output system, called a Basic Input / Output System (BIOS), which is used to perform a power-on self-test of the system, initialize various functional modules in the system, provide basic input / output drivers for the system, and boot the operating system.

[0086] In some embodiments, upon receiving a power-on signal, the display device 200 starts up, and the CPU executes system startup instructions in the ROM 252, copying temporary operating system data stored in the memory to the RAM 251 to facilitate booting or running the operating system. After the operating system boot is complete, the CPU copies temporary data for various application programs in the memory to the RAM 251 to facilitate booting or running the various application programs.

[0087] In some embodiments, the CPU processor 254 is configured to execute operating system and application instructions stored in the memory, and to execute various applications, data, and content based on various interactive instructions received from external input, so as to ultimately display and play various audio and video content.

[0088] In some exemplary embodiments, the CPU processor 254 may include multiple processors. The multiple processors may include a main processor and one or more sub-processors. The main processor is used to perform certain operations of the display device 200 in pre-power mode and / or display images in normal mode. The one or more sub-processors are used to perform certain operations in states such as standby mode.

[0089] In some embodiments, the graphics processor 253 is used to generate various graphical objects, such as icons, operation menus, and graphics displayed in response to user input commands. It includes an operator that performs operations based on various user interaction commands and displays various objects according to display attributes. It also includes a renderer that renders the various objects generated by the operator. These rendered objects are then displayed on the display.

[0090] In some embodiments, the video processor 270 is configured to receive an external video signal and perform video processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image synthesis, etc. according to the standard codec protocol of the input signal, and obtain a signal that can be directly displayed or played on the display device 200.

[0091] In some embodiments, the video processor 270 includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, and the like.

[0092] The demultiplexing module is used to demultiplex the input audio and video data streams. For example, if MPEG-2 is input, the demultiplexing module demultiplexes the data into video signals and audio signals.

[0093] The video decoding module is used to process the demultiplexed video signal, including decoding and scaling.

[0094] The image synthesis module, such as an image synthesizer, is used to superimpose and mix the GUI signal generated by the graphics generator according to user input or itself with the scaled video image to generate an image signal for display.

[0095] The frame rate conversion module is used to convert the input video frame rate, such as converting the 60Hz frame rate to 120Hz frame rate or 240Hz frame rate. The common format is implemented by interpolation.

[0096] The display formatting module is used to convert the received frame rate converted video output signal and change the signal to conform to the display format signal, such as outputting RGB data signal.

[0097] In some embodiments, the graphics processor 253 can be integrated with the video processor or can be separately configured. When integrated, it can perform processing of the graphics signals output to the display. When separately configured, it can perform different functions separately, such as a GPU+FRC (Frame Rate Conversion) architecture.

[0098] In some embodiments, the audio processor 280 is used to receive an external audio signal, and perform decompression and decoding, as well as noise reduction, digital-to-analog conversion, and amplification processing according to the standard codec protocol of the input signal to obtain a sound signal that can be played in a speaker.

[0099] In some embodiments, the video processor 270 may include one or more chips. The audio processor may also include one or more chips.

[0100] In some embodiments, the video processor 270 and the audio processor 280 may be separate chips, or may be integrated with the controller into one or more chips.

[0101] In some embodiments, the audio output receives the sound signal output by the audio processor 280 under the control of the controller 250, such as: the speaker 286, and in addition to the speaker carried by the display device 200 itself, can be output to the external audio output terminal of the generating device of the external device, such as: an external audio interface or a headphone interface, etc., and can also include a short-range communication module in the communication interface, for example: a Bluetooth module for Bluetooth speaker sound output.

[0102] The power supply 290, under the control of the controller 250, uses the power input from the external power source to provide power supply support for the display device 200. The power supply 290 can include a built-in power supply circuit installed inside the display device 200, or it can be installed externally to the display device 200 and provide a power interface for the display device 200 to connect to an external power source.

[0103] The user interface 265 is used to receive user input signals and then send the received user input signals to the controller 250. The user input signals may be remote control signals received by an infrared receiver or various user control signals received by a network communication module.

[0104] In some embodiments, the user inputs a user command through the control device 1001 or the mobile terminal 1002 , the user input interface is based on the user input, and the display device 200 responds to the user input through the controller 250 .

[0105] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 275, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific sound or gesture, and the user input interface may recognize the sound or gesture through a sensor to receive the user input command.

[0106] In some embodiments, a "user interface" is a medium interface for interaction and information exchange between an application or operating system and a user. It enables the conversion between the internal form of information and a form acceptable to the user. A common form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations that uses a graphical display. It can be an icon, window, control, or other interface element displayed on the display of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0107] The memory 260 stores various software modules for driving the display device 200. For example, the various software modules stored in the first memory include at least one of a basic module, a detection module, a communication module, a display control module, a browser module, and various service modules.

[0108] The basic module is a low-level software module used to communicate signals between various hardware components in the display device 200 and send processing and control signals to upper-level modules. The detection module is a management module used to collect various information from various sensors or user input interfaces, perform digital-to-analog conversion, and perform analysis and management.

[0109] For example, the speech recognition module includes a speech parsing module and a speech command database module. The display control module is used to control the display to display image content and can be used to play multimedia image content and UI interface information. The communication module is used to control and communicate data with external devices. The browser module is used to perform data communication between browsing servers. The service module is used to provide various services and various applications. Memory 260 is also used to store received external data and user data, images of various items in the user interface, and visual effects of focus objects.

[0110] Figure 3 FIG. 1 exemplarily shows a configuration block diagram of a control device 1001 according to an exemplary embodiment. Figure 3 As shown, the control device 1001 includes a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0111] The control device 1001 is configured to control the display device 200 and receive user input commands. It converts these commands into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200. For example, a user may press the channel up / down keys on the control device 1001, and the display device 200 may respond to these channel up / down operations.

[0112] In some embodiments, the control device 1001 may be a smart device. For example, the control device 1001 may be installed with various applications for controlling the display device 200 according to user needs.

[0113] In some embodiments, as Figure 1 As shown, after installing the application for controlling the display device 200, the mobile terminal 1002 or other smart electronic device can perform similar functions to controlling the device 1001. For example, the user can install the application and use various function keys or virtual buttons of the graphical user interface provided on the mobile terminal 1002 or other smart electronic device to realize the functions of the physical buttons of the control device 1001.

[0114] The controller 110 includes a processor 112, RAM 113, ROM 114, a communication interface 130, and a communication bus. The controller is used to control the operation and operation of the control device 1001, as well as the communication and cooperation between internal components and external and internal data processing functions.

[0115] Under the control of the controller 110, the communication interface 130 communicates control signals and data signals with the display device 200. For example, it transmits received user input signals to the display device 200. The communication interface 130 may include at least one of a WiFi chip 131, a Bluetooth module 132, an NFC module 133, or other near-field communication modules.

[0116] The user input / output interface 140 includes at least one of a microphone 141, a touchpad 142, a sensor 143, a keypad 144, and other input interfaces. For example, a user can input user commands through voice, touch, gestures, pressing, and other actions. The input interface converts received analog signals into digital signals, and then converts the digital signals into corresponding command signals, which are then sent to the display device 200.

[0117] The output interface includes an interface for transmitting received user commands to the display device 200. In some embodiments, this can be either an infrared interface or a radio frequency interface. For example, in an infrared signal interface, user input commands need to be converted into infrared control signals according to an infrared control protocol and then transmitted to the display device 200 via an infrared transmission module. Alternatively, in a radio frequency signal interface, user input commands need to be converted into digital signals, modulated according to a radio frequency control signal modulation protocol, and then transmitted to the display device 200 via a radio frequency transmission terminal.

[0118] In some embodiments, the control device 1001 includes at least one of a communication interface 130 and an input / output interface 140. The control device 1001 is configured with a communication interface 130, such as a WiFi, Bluetooth, or NFC module, to encode user input commands via the WiFi, Bluetooth, or NFC protocols and transmit them to the display device 200.

[0119] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 1001 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0120] The power supply 180 is used to provide operating power support for various components of the control device 1001 under the control of the controller. It can be a battery and related control circuits.

[0121] In some embodiments, the system may include a kernel, a command parser (shell), a file system, and applications. The kernel, shell, and file system together form the basic operating system structure, allowing users to manage files, run programs, and use the system. After power is applied, the kernel boots up, activates kernel space, abstracts hardware, initializes hardware parameters, and runs and maintains virtual memory, a scheduler, signals, and inter-process communication (IPC). After the kernel boots up, the shell and user applications are loaded. After booting up, the application is compiled into machine code to form a process.

[0122] Figure 4 For the software system diagram of the display device provided in this application, see Figure 4 In some embodiments, the system is divided into four layers, namely, from top to bottom, the application layer (referred to as "application layer"), the application framework layer (referred to as "framework layer"), the Android runtime and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0123] In some embodiments, at least one application runs in the application layer. These applications may be window programs, system settings programs, clock programs, camera applications, etc. that come with the operating system; or they may be applications developed by third-party developers, such as hi-view programs, karaoke programs, and magic mirror programs. In specific implementations, the application packages in the application layer are not limited to the above examples and may actually include other application packages, which are not limited in this embodiment of the present application.

[0124] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions taken by applications in the application layer. Applications can access system resources and obtain system services through the API interface during execution.

[0125] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager (ActivityManager) is used to interact with all activities running in the system; a location manager (Location Manager) is used to provide system services or applications with access to system location services; a package manager (Package Manager) is used to retrieve various information related to the application packages currently installed on the device; a notification manager (NotificationManager) is used to control the display and clearing of notification messages; a window manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0126] In some embodiments, the activity manager is used to: manage the life cycle of each application and the usual navigation back function, such as controlling the exit of the application (including switching the user interface currently displayed in the display window to the system desktop), opening, backing (including switching the user interface currently displayed in the display window to the upper-level user interface of the currently displayed user interface), etc.

[0127] In some embodiments, the window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, controlling display window changes (for example, shrinking the display window, shaking the display, distorting the display, etc.), etc.

[0128] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0129] In some embodiments, the kernel layer is a layer between hardware and software. Figure 4As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, touch sensor, pressure sensor, etc.), etc.

[0130] In some embodiments, the core layer further includes a power driver module for performing power management.

[0131] In some embodiments, Figure 4 The software programs and / or modules corresponding to the software architecture in Figure 2 or Figure 3 In the first memory or the second memory shown.

[0132] In some embodiments, taking the Magic Mirror application (photography application) as an example, when the remote control receiving device receives a remote control input operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the input operation into a raw input event (including the value of the input operation, the timestamp of the input operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer, identifies the control corresponding to the input event based on the current focus position, and determines that the input operation is a confirmation operation. The control corresponding to the confirmation operation is the control of the Magic Mirror application icon. The Magic Mirror application calls the interface of the application framework layer to start the Magic Mirror application, and then starts the camera driver by calling the kernel layer to capture still images or videos through the camera.

[0133] In some embodiments, for a display device with a touch function, taking a split-screen operation as an example, the display device receives an input operation (such as a split-screen operation) performed by a user on the display screen, and the kernel layer can generate a corresponding input event based on the input operation and report the event to the application framework layer. The activity manager of the application framework layer sets the window mode (such as a multi-window mode) and the window position and size corresponding to the input operation. The window management of the application framework layer draws the window according to the settings of the activity manager, and then sends the drawn window data to the display driver of the kernel layer, which displays the corresponding application interface in different display areas of the display screen.

[0134] In some embodiments, Figure 5 This is a schematic diagram of the application programs that the display device provided by this application can provide, such as Figure 5 As shown in , the application layer includes at least one application that can display corresponding icon controls on the display, such as: live TV application icon controls, video on demand application icon controls, media center application icon controls, application center icon controls, game application icon controls, etc.

[0135] In some embodiments, the live TV application can provide live TV from various signal sources. For example, the live TV application can provide a TV signal using an input from a cable TV service, over-the-air broadcast, satellite service, or other type of live TV service. Furthermore, the live TV application can display video from the live TV signal on the display device 200.

[0136] In some embodiments, a video-on-demand application can provide videos from various storage sources. Unlike a live TV application, a video-on-demand application provides video display from a storage source. For example, the video-on-demand application can be served from a cloud storage server or from a local hard drive containing stored video programs.

[0137] In some embodiments, a media center application may be an application that provides various multimedia content playback applications. For example, a media center may be a service that is different from live TV or video on demand, and a user may access various images or audio services through the media center application.

[0138] In some embodiments, an application center can provide storage for various applications. These applications can be games, apps, or other applications related to computer systems or other devices that can be run on a smart TV. The application center can obtain these applications from various sources, store them in local storage, and then run them on the display device 200.

[0139] More specifically, in some embodiments, any of the display devices 200 described above in this application may have a voice interaction function to improve the intelligence level of the display device 200 and enhance the user experience of the display device 200 .

[0140] In some embodiments, Figure 6 The present invention is a schematic diagram of an application of a display device in a voice interaction scenario, wherein user 1 can speak the instructions that he wants the display device 200 to execute through voice. The display device 200 can then collect voice data in real time, recognize the instructions of user 1 included in the voice data, and directly execute the instructions after recognizing the instructions of user 1. During the entire process, user 1 does not actually operate the display device 200 or other devices, but simply speaks the instructions.

[0141] In some embodiments, when Figure 2 The display device 200 shown is used in Figure 6 In the scenario shown, the display device 200 can collect voice data in real time through its sound collector 231. Subsequently, the sound collector 231 sends the collected voice data to the controller 250, and finally the controller 250 recognizes the instructions included in the voice data.

[0142] In some embodiments, Figure 7 This is a flow chart of the application of the display device in the voice interaction scenario, which can be as follows: Figure 6 The devices in the scenario shown execute, specifically, in S11, the sound collector 231 in the display device 200 collects voice data in the surrounding environment of the display device 200 in real time, and sends the collected voice data to the controller 250 for recognition.

[0143] In some embodiments, Figure 7 In S12, after receiving the voice data, the controller 250 recognizes the instructions included in the voice data. For example, if the voice data includes the instruction "increase brightness" issued by user 1, the controller 250, after recognizing the instruction included in the voice data, can execute the recognized instruction to control the display 275 to increase the brightness. It will be understood that in this case, the controller 250 recognizes each received voice data and may recognize that there is no instruction in the voice data.

[0144] In other embodiments, the model based on instruction recognition is large and the computational efficiency is low. It is also possible to stipulate that user 1 adds keywords before speaking the instruction, for example, "ABCD", then the user needs to speak the instruction "ABCD, increase brightness", so that in the example Figure 7 In S12 shown, after receiving the voice data, the controller 250 first identifies whether there is a keyword "ABCD" in each voice data. After identifying the keyword, it uses the instruction recognition model to identify the specific instruction corresponding to "increase brightness" in the voice data.

[0145] In some embodiments, after receiving the voice data, the controller 250 may further perform denoising on the voice data, including removing echoes and ambient noise, processing the voice data into clean voice data, and recognizing the processed voice data.

[0146] In some embodiments, Figure 7 This is another application diagram of a display device in a voice interaction scenario, wherein the display device 200 can be connected to the server 400 via the Internet. When the display device 200 collects voice data, it can send the voice data to the server 400 via the Internet. The server 400 recognizes the instructions included in the voice data and sends the recognized instructions back to the display device 200, so that the display device 200 can directly execute the received instructions. This scenario is similar to the following: Figure 6 Compared with the scenario shown, the requirement for the computing power of the display device 200 is reduced, and a larger recognition model can be set on the server 400 to further improve the accuracy of command recognition in voice data.

[0147] In some embodiments, when Figure 2 The display device 200 shown is used in Figure 6 In the scenario shown, the display device 200 can collect voice data in real time through its sound collector 231. Subsequently, the sound collector 231 sends the collected voice data to the controller 250. The controller 250 sends the voice data to the server 400 through the communicator 220. After the server 400 recognizes the instructions included in the voice data, the display device 200 receives the instructions sent by the server 400 through the communicator 220, and finally the controller 250 executes the received instructions.

[0148] In some embodiments, Figure 9 Another flowchart of the application of the display device in the voice interaction scenario can be as follows Figure 8 The device in the scenario shown is executed, wherein in S21, the sound collector 231 in the display device 200 collects voice data in the surrounding environment of the display device 200 in real time, and sends the collected voice data to the controller 250. The controller 250 further sends the voice data to the server 400 through the communicator 220 in S22, and the server recognizes the instructions included in the voice data in S23. Subsequently, the server 400 sends the recognized instructions back to the display device 200 in S24. Correspondingly, the display device 200 receives the instructions through the communicator 220 and sends them to the controller 250. Finally, the controller 250 can directly execute the received instructions.

[0149] In some embodiments, as Figure 7In S23 shown, after receiving the voice data, the server 400 recognizes the instruction included in the voice data. For example, the voice data includes the instruction "increase brightness" issued by user 1. However, since the model for instruction recognition is relatively large and the server 400 recognizes each received voice data, it may happen that there is no instruction in the recognized voice data. Therefore, in order to reduce the invalid recognition by the server 400 and reduce the amount of communication interaction data between the display device 200 and the server 400, in the specific implementation, it can also be stipulated that user 1 adds keywords before speaking the instructions. For example, "ABCD", the user needs to say the instruction "ABCD, increase brightness". Subsequently, the controller 250 of the display device 200 first uses a keyword recognition model with a smaller model and lower computational complexity in S22 to identify whether the keyword "ABCD" exists in the voice data. If the keyword is not recognized in the voice data currently being processed by the controller 250, the controller 250 will not send the voice data to the server 400; if the keyword is recognized in the voice data currently being processed by the controller 250, the controller 250 will then send all the voice data, or the part after the keyword in the voice data, to the server 400, and the server 400 will recognize the received voice data. Since the voice data received by the controller 250 at this time includes keywords, the voice data sent to the server 400 for recognition is more likely to include user instructions, thereby reducing invalid recognition calculations of the server 400 and reducing invalid communications between the display device 200 and the server 400.

[0150] In some embodiments, in order to enable the display device 200 to have Figure 6 In a specific scenario shown, the recognition function of the command in the voice data, or the display device 200 can have the following Figure 6 or Figure 8 In a specific scenario, for the recognition of keywords in voice data, the supplier of the voice interaction function of the display device 200 needs to develop machine learning models that can be used to recognize commands or keywords, such as deep learning models such as TextCNN and Transform. These models are then stored in the display device 200 and used by the display device 200 during recognition.

[0151] In some embodiments, Figure 10 Schematic diagram of the recognition model supplier issuing the recognition model, wherein the server 400 set by the supplier obtains the recognition model (which can be a command recognition model or a keyword recognition model) and then sends the recognition model to each display device 200. Figure 10The process shown can be performed when the display device 200 is produced, and the server 400 sends the recognition model to each display device 200; or, after the display device 200 starts to be used, the server 400 sends the recognition model to the display device 200 via the Internet.

[0152] In some embodiments, the server 400 may obtain a recognition model by collecting voice data and learning based on a machine learning model. For example, Figure 11 A flow chart of a process for obtaining a recognition model for the server 400, wherein, in S31, each display device (taking display device 1-display device N, a total of N as an example) collects voice data 1-N, and sends the collected voice data 1-N to the server 400 in S32. Subsequently, in S33, the supplier's staff can manually annotate each voice data and the instructions or keywords included in the voice data, and then send the voice data itself and the annotation information corresponding to the voice data as data into the machine learning model, which is learned by the server. When the learned recognition model is used subsequently, when a voice data to be recognized is input, the recognition model compares the voice data with the learned voice data and outputs the probability of each annotation information. The annotation information corresponding to the maximum probability can be used as the recognition result of the voice data to be recognized. In S34, the server 400 can send the calculated recognition model to each display device.

[0153] In some embodiments, the server 400 may not use Figure 11 In the embodiment shown, the recognition model is calculated based on the voice data actually collected by the display devices 1-N. Instead, the staff can directly input different voice data and the annotation information of each voice data, and send the calculated recognition model to each display device.

[0154] In some embodiments, as Figure 11 The process of collecting voice data and sending it to the server by display devices 1-N, and the process of the server calculating the recognition model and then sending it to display devices 1-N, can be two independent processes. That is, in step S32, the server receives voice data collected by N display devices, and in step S34, the server can send the trained recognition model to another N display devices. The N display devices in these two processes can be the same, different, or partially the same.

[0155] In some embodiments, since the number of samples used in obtaining the recognition model is limited, the recognition model set by the display device 200 cannot achieve completely 100% accurate recognition. Therefore, the supplier can also collect the voice data collected by each display device 200 during actual use at any time through the server 400, and update the recognition model that has been recognized based on the collected voice data to further improve the recognition accuracy of the recognition model.

[0156] For example, Figure 12 A flow chart of updating the recognition model by the server. It can be understood that when executing Figure 12 Before the embodiment shown, each display device is configured as follows Figure 10 The recognition model is set as shown in the following example. Figure 12 In S31 shown, each display device (taking display device 1 to display device N, a total of N as an example) collects voice data 1-N, and sends the collected voice data 1-N to the server 400 in S32. Subsequently, in S33, the supplier's staff can manually annotate each voice data and the instructions or keywords included in the voice data, and then send the voice data itself and the annotation information corresponding to the voice data as data into the machine learning model. The server updates the recognition model that has been calculated based on the new voice data received, and in S34, the server 400 can resend the updated recognition model to each display device 200, so that each display device 200 can be updated using the updated recognition model. Among them, for any one of the N display devices, since the new learning model uses the voice data collected by this display device 200, it can effectively improve the accuracy of subsequent recognition of the collected voice data by this display device 200.

[0157] In some embodiments, as Figure 12 Each display device shown can send the voice data to the server after receiving it, or send the voice data collected within this time period to the server after a fixed time period ends, or send a certain amount of voice data to the server after it is collected, or send the received voice to the server according to the instructions of the user of the display device or the instructions of the server staff.

[0158] In some embodiments, as Figure 12The N display devices shown can send voice data to the server at the same agreed time, and the server will update the recognition model based on the N voice data received; or, the N display devices can also send voice data to the server separately, and after the server receives more than N voice data, it can start updating the recognition model based on the received voice data.

[0159] To clearly illustrate the embodiments of the present application, Figure 13 A network architecture provided in an embodiment of the present application is described.

[0160] Figure 13 This is a diagram of the system architecture for tracking the conversation state provided in the embodiment of the present application. Figure 13 As shown, the system provided in this embodiment includes a smart device 601 and a service device 602. This embodiment does not impose any particular restrictions on the implementation of smart device 601, as long as smart device 601 can interact with the user through voice. Service device 602 can be a standalone device or an all-in-one device integrated with smart device 601.

[0161] The following describes a method for tracking a conversation state provided by an embodiment of the present application. In one embodiment, the method can be applied to smart devices such as smart TVs, smart speakers, and smart phones.

[0162] Figure 14 Schematic diagram of the process of the conversation state tracking method provided in the embodiment of the application Figure 1 , the execution subject of this embodiment can be Figure 13 The service equipment in the embodiment shown. Figure 14 As shown, the method includes:

[0163] S701: Obtain the time interval between the current query and the previous query, the similarity between the current query and the previous query, and the location domain and score of the current query.

[0164] In this embodiment, the time of the user's current query statement and the time of the previous query statement are determined based on the input time of the user's human-computer interaction, and the time interval between the user's current query statement and the previous query statement is determined based on the time of the user's current query statement and the time of the previous query statement.

[0165] In this embodiment, the similarity may be determined according to the ratio of identical elements in the first set of query statements in the current round and the second set of query statements in the previous round.

[0166] Specifically, the first set of query statements in this round is set as set A, and the second set of query statements in the previous round is set as set B. The similarity is the ratio of the number of identical elements in set A and set B to the sum of the number of all elements in set A and set B.

[0167] In this embodiment, the current query statement and the previous query statement are respectively input into a preset intention understanding module to obtain the fields and field scores corresponding to the current query statement and the fields and field scores corresponding to the previous query statement.

[0168] The domains and domain scores corresponding to the current query statement include multiple domains and their corresponding domain scores, with the domain with the highest domain score being the domain targeted by the current query statement. The domains and domain scores corresponding to the previous query statement include multiple domains and their corresponding domain scores, with the domain with the highest domain score being the domain targeted by the previous query statement.

[0169] S702: Determine whether to perform session segmentation on the query statement in the current round and the query statement in the previous round according to the time interval between the query statement in the current round and the query statement in the previous round, the similarity between the query statement in the current round and the query statement in the previous round, and the positioning domain and score of the query statement in the current round.

[0170] Specifically, if the time interval between the query statement in this round and the query statement in the previous round does not exceed the set time threshold; the similarity between the query statement in this round and the query statement in the previous round is greater than the set similarity threshold; the positioning field of the query statement in this round is an unknown field; the positioning field of the query statement in this round and the positioning field of the query statement in the previous round meet the preset field condition restrictions, then the query statement in this round and the query statement in the previous round will not be session split; otherwise, the query statement in this round and the query statement in the previous round will be session split.

[0171] In this embodiment, the time threshold may be set based on the domain targeted by the previous round of query statements. Different time thresholds may be set based on different domains targeted by the previous round of query statements.

[0172] In some embodiments, the time threshold is set to a value that is positively correlated with the activity time of the domain located by the previous query statement. For domains with longer activity time, the time threshold is set to a longer value; for domains with shorter activity time, the time threshold is set to a shorter value.

[0173] For example, for long-term activities such as watching videos, the time threshold is set to 5 minutes; for short-term activities such as chatting, answering questions, etc., the time threshold is set to 1 minute.

[0174] In this embodiment, the similarity threshold may be set to 0.3.

[0175] For example, if the similarity between the following query sentences "Youth Song à Youth Song, Sparrow à Sparrow Song, Fiery Red Ear Man à Fiery Red Sarlang" is greater than 0.3, the query sentences should not be segmented.

[0176] In this embodiment, the positioning domain of this round of query statements is an unknown domain, which may be the unknown domain (UNKNOWN) with the highest corresponding domain score output by a preset intention understanding module.

[0177] In one embodiment of the present application, the positioning domain of the current round of query statements and the positioning domain of the previous round of query statements meet the preset domain condition limitations, including:

[0178] The positioning domain score of the current round of query statements is greater than the first set score threshold, and the relevance with the positioning domain of the previous round of query statements exceeds the set relevance threshold; or,

[0179] When the positioning domain of the current query statement and the positioning domain of the previous query statement are easily confused, the scores of the positioning domain of the current query statement and the positioning domain of the previous query statement are both less than the second set score threshold, or the slot information of the previous query statement is empty; or

[0180] The positioning domain of the query statement in this round is the same as the positioning domain of the query statement in the previous round.

[0181] The first set score threshold may be 0.8. The set relevance threshold may be set as needed. For example, if the relevance between movie search and TV control, or between ticket booking and weather query does not exceed the set relevance threshold, no segmentation is performed.

[0182] The current query's location domain is easily confused with the previous query's location domain, meaning the domain contains more than a certain number of identical elements. For example, movie and music searches, or game and app searches, are easily confused.

[0183] The second set score threshold may be 0.7. The slot information of the previous round query statement being empty means that the previous round query statement has not been parsed to slot information.

[0184] Among them, the positioning field of the current query statement is the same as the positioning field of the previous query statement, which means that the field with the highest field score corresponding to the current query statement is the same as the field with the highest field score corresponding to the previous query statement.

[0185] S703: Segment the human-computer interaction session of the user according to the judgment result of session segmentation to obtain at least one segmented session sentence corresponding to the user.

[0186] In this embodiment, based on the current query statement and the previous query statement that have been determined to be segmented in step S702, all user human-computer interaction conversations are segmented at the current query statement and the previous query statement to obtain at least one segmented conversation statement. There are generally multiple segmented conversation statements.

[0187] S704: Using a preset deep learning model, each conversation sentence is processed to obtain dialogue state information corresponding to each conversation sentence of the user.

[0188] In this embodiment, after feature extraction, each conversation sentence is input into a preset deep learning model, and the intention and slot information corresponding to each conversation sentence is output, that is, the dialogue state information corresponding to each conversation sentence is obtained.

[0189] From the above description, it can be seen that based on the time interval between the query statement of this round and the query statement of the previous round, the similarity between the query statement of this round and the query statement of the previous round, and whether the positioning field and score of the query statement of this round meet the requirements of the rules; and according to the judgment result of whether the rules are met, all conversations in the user's human-computer interaction process are segmented to obtain individual conversation sentences; each conversation sentence is input into the preset deep learning model to obtain the intention and slot information corresponding to the user conversation sentence. Since the embodiment of the present application can effectively utilize the context information of the user query sentence, exclude irrelevant query sentences, and realize the segmentation of the user's conversation sentence, after the segmented conversation sentence is input into the preset deep learning model, it can improve the accuracy of the output intention and slot information, thereby improving the accuracy of the machine output content.

[0190] Figure 15 Schematic diagram of the process of the conversation state tracking method provided in the embodiment of the application Figure 2 , this embodiment Figure 14 Based on the embodiment, the specific implementation process of step S704 in the above embodiment is described in detail. Figure 15 As shown, the method includes:

[0191] S801: Extract features of the conversation sentence according to a preset language representation model to obtain a conversation feature vector corresponding to the conversation sentence.

[0192] In this embodiment, the pre-set language representation model can be a BERT model. The BERT model, represented by a bidirectional encoder using a Transformer, is designed to pre-train deep bidirectional representations by jointly adjusting context across all layers. Therefore, the pre-trained BERT representation can be fine-tuned with an additional output layer, making it suitable for building state-of-the-art models for a wide range of tasks, such as question answering and language reasoning, without requiring significant architecture modifications for specific tasks.

[0193] After being processed by the Bert model, the conversation sentence starts with [CLS] and uses [SEP] as a separator to represent the conversation feature vector matrix u.

[0194] S802: Input the conversation feature vector into the first module of the preset deep learning model to obtain the intent of the conversation sentence and the probability distribution of each intent.

[0195] In this embodiment, the first module that inputs the preset deep learning model can be a first intent classification neural network.

[0196] Specifically, refer to the attached Figure 16 ,like Figure 16 As shown on the right, the conversation feature vector u is input into the first intent classification neural network, and first passes through the self-attention mechanism (Self-Attention) layer to obtain the latent vector h′ t , after the first two-layer bidirectional LSTM (Long Short-term Memory) layer, the feature vector d′ is obtained t , and then go through the first regularization layer (normalization processing), and finally classify the user's intentions to obtain the intention of the conversation sentence and the probability distribution of each intention.

[0197] S803: Each domain word related to the conversation sentence and each intention are combined into a main idea sentence, and the language representation model is used to perform feature extraction on the main idea sentence to obtain a main idea vector.

[0198] In this embodiment, each domain word and each of the intentions are combined into a main image sentence, which also starts with [CLS] and ends with [SEP] segmentation. The Bert model is used for feature extraction to obtain the main image vector matrix q.

[0199] S804: Input the main intention vector and the conversation feature vector into the second module of the preset deep learning model to obtain the feature vector of the domain and intention corresponding to the conversation sentence.

[0200] In this embodiment, the second module of the preset deep learning model can be a second intent classification neural network.

[0201] Specifically, refer to the attached Figure 16 ,like Figure 16 As shown in the middle, the session feature vector matrix u and the main image vector matrix q are input into the second intent classification neural network sink. First, the Multi-Attention layer is input, where q is used as the Question in Multi-Attention, and u is used as K and V in Multi-Attention. The latent vector h is calculated.t ; Then the second double-layer bidirectional LSTM layer obtains the feature vector d t ; Then after the second regularization layer (normalization processing), the feature vectors of domain and intention are obtained

[0202] S805: For each slot information corresponding to the conversation sentence, use the language representation model to perform feature extraction on each slot information to obtain a slot vector corresponding to each slot information.

[0203] In this embodiment, refer to the attached Figure 16 ,like Figure 16 As shown on the left, for each possible slot information, the Bert model is also used for feature extraction to obtain the slot vector y corresponding to each slot information s .

[0204] S806: Determine the probability distribution of each slot information based on each slot vector and the feature vector of the domain and intent.

[0205] In this embodiment, each slot vector y is calculated s A feature vector with the fields and intents The cosine similarity of is used to determine the probability distribution of each slot information.

[0206] S807: Determine the intent and slot information corresponding to the conversation sentence according to the probability distribution of each intent and the probability distribution of each slot information.

[0207] In this embodiment, the intent with the highest probability distribution among the intents is the intent corresponding to the conversation sentence; the slot information with the highest probability distribution among the slot information is the slot information corresponding to the conversation sentence.

[0208] From the above description, it can be seen that after the user-computer interaction conversation is segmented based on the preset rules, the segmented conversation sentences are input into the preset deep learning model, and the rule segmentation of the conversation is combined with deep learning to further improve the accuracy of predicting the intent and slot information corresponding to the conversation sentences.

[0209] Figure 17 Schematic diagram of the process of the conversation state tracking method provided in the embodiment of the application Figure 3 .like Figure 17 As shown, the process is applied to the above-mentioned smart device and includes the following steps:

[0210] S1701: Send the query statement of this round input through the input interface to the service device, so that the service device segments the user's human-computer interaction conversation according to preset rules to obtain at least one segmented conversation statement corresponding to the user; and process each conversation statement to obtain dialogue state information corresponding to each conversation statement of the user.

[0211] Specifically, the current round query statement input through the input interface is sent to the service device, so that the service device obtains the time interval between the user's current round query statement and the previous round query statement, the similarity between the current round query statement and the previous round query statement, and the positioning field of the current round query statement and its score; based on the time interval between the current round query statement and the previous round query statement, the similarity between the current round query statement and the previous round query statement, and the positioning field of the current round query statement and its score, it is determined whether to perform conversation segmentation on the current round query statement and the previous round query statement; based on the judgment result of the conversation segmentation, the user's human-computer interaction conversation is segmented to obtain at least one segmented conversation statement corresponding to the user; and each conversation statement is processed using a preset deep learning model to obtain dialogue state information corresponding to each conversation statement of the user.

[0212] S1702: Acquire the conversation state information from the service device and output it.

[0213] From the above description, it can be seen that based on the time interval between the query statement of this round and the query statement of the previous round, the similarity between the query statement of this round and the query statement of the previous round, and whether the positioning field and score of the query statement of this round meet the requirements of the rules; and according to the judgment result of whether the rules are met, all conversations in the user's human-computer interaction process are segmented to obtain individual conversation sentences; each conversation sentence is input into the preset deep learning model to obtain the intention and slot information corresponding to the user conversation sentence. Since the embodiment of the present application can effectively utilize the context information of the user query sentence, exclude irrelevant query sentences, and realize the segmentation of the user's conversation sentence, after the segmented conversation sentence is input into the preset deep learning model, it can improve the accuracy of the output intention and slot information, thereby improving the accuracy of the machine output content.

[0214] Figure 18 This is a schematic diagram of the structure of the service equipment provided in the embodiment of the present application. Figure 18 As shown, the service device 110 includes: an acquisition unit 1101 , a judgment unit 1102 , a segmentation unit 1103 and a processing unit 1104 .

[0215] The acquisition unit 1101 is used to obtain the time interval between the user's current query and the previous query, the similarity between the current query and the previous query, and the positioning field of the current query and its score;

[0216] A determination unit 1102 is configured to determine whether to perform session segmentation on the current query and the previous query based on the time interval between the current query and the previous query, the similarity between the current query and the previous query, and the location domain and score of the current query.

[0217] The segmentation unit 1103 is configured to segment the user's human-computer interaction session according to the judgment result of the session segmentation, and obtain at least one segmented session sentence corresponding to the user;

[0218] The processing unit 1104 is used to process each conversation sentence using a preset deep learning model to obtain the dialogue state information corresponding to each conversation sentence of the user.

[0219] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0220] In one possible design, the judgment unit 1102 is specifically configured to not perform session segmentation on the query statement in the current round and the query statement in the previous round if the time interval between the query statement in the current round and the query statement in the previous round does not exceed a set time threshold; the similarity between the query statement in the current round and the query statement in the previous round is greater than a set similarity threshold; the positioning domain of the query statement in the current round is an unknown domain; and the positioning domain of the query statement in the current round and the positioning domain of the query statement in the previous round meet a preset domain condition limitation;

[0221] Otherwise, the current round query statement and the previous round query statement are divided into sessions.

[0222] In one possible design, the positioning domain of the current round of query statements and the positioning domain of the previous round of query statements meet the preset domain condition restrictions, including:

[0223] The positioning domain score of the current round of query statements is greater than the first set score threshold, and the relevance with the positioning domain of the previous round of query statements exceeds the set relevance threshold; or,

[0224] The positioning domain of the current query statement and the positioning domain of the previous query statement are not easily confused, the scores of the positioning domain of the current query statement and the positioning domain of the previous query statement are both less than the second set score threshold, or the slot information of the previous query statement is empty; or,

[0225] The positioning domain of the query statement in this round is the same as the positioning domain of the query statement in the previous round.

[0226] In one possible design, the processing unit 1104 is specifically configured to extract features from the conversational sentence according to a preset language representation model to obtain a conversational feature vector corresponding to the conversational sentence;

[0227] Inputting the conversation feature vector into the first module of the preset deep learning model to obtain the intent of the conversation sentence and the probability distribution of each intent;

[0228] Combining each domain word related to the conversation sentence with each intention into a main idea sentence, and performing feature extraction on the main idea sentence using the language representation model to obtain a main idea vector;

[0229] Inputting the main intention vector and the conversation feature vector into a second module of a preset deep learning model to obtain a feature vector of the domain and intention corresponding to the conversation sentence;

[0230] For each slot information corresponding to the conversation sentence, extract features of each slot information using the language representation model to obtain a slot vector corresponding to each slot information;

[0231] Determine the probability distribution of each slot information based on each slot vector and the feature vector of the domain and intention;

[0232] Determine the intent and slot information corresponding to the conversation sentence based on the probability distribution of each intent and the probability distribution of each slot information.

[0233] In one possible design, the judgment unit 1102 is specifically configured to not perform session segmentation on the query statement in the current round and the query statement in the previous round if the time interval between the query statement in the current round and the query statement in the previous round does not exceed a set time threshold; the similarity between the query statement in the current round and the query statement in the previous round is greater than a set similarity threshold; the positioning domain of the query statement in the current round is an unknown domain; and the positioning domain of the query statement in the current round and the positioning domain of the query statement in the previous round meet a preset domain condition limitation;

[0234] Otherwise, the current round query statement and the previous round query statement are divided into sessions.

[0235] In one possible design, the positioning domain of the current round of query statements and the positioning domain of the previous round of query statements meet the preset domain condition restrictions, including:

[0236] The positioning domain score of the current round of query statements is greater than the first set score threshold, and the relevance with the positioning domain of the previous round of query statements exceeds the set relevance threshold; or,

[0237] The positioning domain of the current query statement and the positioning domain of the previous query statement are not easily confused, the scores of the positioning domain of the current query statement and the positioning domain of the previous query statement are both less than the second set score threshold, or the slot information of the previous query statement is empty; or,

[0238] The positioning domain of the query statement in this round is the same as the positioning domain of the query statement in the previous round.

[0239] In one possible design, the processing unit 1104 is specifically configured to extract features from the conversational sentence according to a preset language representation model to obtain a conversational feature vector corresponding to the conversational sentence;

[0240] Inputting the conversation feature vector into the first module of the preset deep learning model to obtain the intent of the conversation sentence and the probability distribution of each intent;

[0241] Combining each domain word related to the conversation sentence with each intention into a main idea sentence, and performing feature extraction on the main idea sentence using the language representation model to obtain a main idea vector;

[0242] Inputting the main intention vector and the conversation feature vector into a second module of a preset deep learning model to obtain a feature vector of the domain and intention corresponding to the conversation sentence;

[0243] For each slot information corresponding to the conversation sentence, extract features of each slot information using the language representation model to obtain a slot vector corresponding to each slot information;

[0244] Determine the probability distribution of each slot information based on each slot vector and the feature vector of the domain and intention;

[0245] Determine the intent and slot information corresponding to the conversation sentence based on the probability distribution of each intent and the probability distribution of each slot information.

[0246] In a possible design, the service device further includes a linear velocity determination unit configured to determine the similarity based on a ratio of identical elements in texts in a first set of query statements in the current round and a second set of query statements in the previous round.

[0247] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0248] As described in the above embodiment, the display device can send the received query statement to the service device, so that the service device can determine whether the query statement meets the rule requirements based on the time interval between the current query statement and the previous query statement, the similarity between the current query statement and the previous query statement, and the location domain and score of the current query statement. Based on the judgment result of whether the rule is met, all conversations during the user's human-computer interaction process are segmented to obtain individual conversation statements; each conversation statement is input into a preset deep learning model to obtain the intent and slot information corresponding to the user's conversation statement. This method can effectively utilize the contextual information of the user's query statement, eliminate irrelevant query statements, and achieve user conversation statement segmentation. After the segmented conversation statements are input into the preset deep learning model, the accuracy of the output intent and slot information can be improved, thereby improving the accuracy of the machine output content.

[0249] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

[0250] For ease of explanation, the above description has been made with reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations are possible. The above embodiments are selected and described to better explain the principles and practical applications, so that those skilled in the art can better utilize the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A smart device, characterized in that: include: Input interfaces and controllers; The controller is configured to: sending the query statement of the current round input through the input interface to the service device, so that the service device segments the user's human-computer interaction conversation according to a preset rule to obtain at least one segmented conversation statement corresponding to the user; and processing each conversation statement to obtain dialogue state information corresponding to each conversation statement of the user; Acquire the session state information from the service device and output it; The controller is specifically configured to: send the current round query statement input through the input interface to the service device, so that the service device obtains the time interval between the current round query statement and the previous round query statement, the similarity between the current round query statement and the previous round query statement, and the positioning field of the current round query statement and its score; Determine whether to perform session segmentation on the query statements in the current round and the query statements in the previous round based on the time interval between the query statements in the current round and the query statements in the previous round, the similarity between the query statements in the current round and the query statements in the previous round, and the positioning domain and score of the query statements in the current round; Segmenting the user's human-computer interaction conversation according to the judgment result of the conversation segmentation to obtain at least one segmented conversation sentence corresponding to the user; performing feature extraction on the conversation sentence according to a preset language representation model to obtain a conversation feature vector corresponding to the conversation sentence; Inputting the conversation feature vector into the first module of a preset deep learning model to obtain the intent of the conversation sentence and the probability distribution of each intent; combining each domain word related to the conversation sentence with each intent to form a main idea sentence, and using the language representation model to extract features from the main idea sentence to obtain a main idea vector; The main idea vector and the conversation feature vector are input into the second module of the preset deep learning model to obtain the feature vector of the domain and intention corresponding to the conversation sentence; for each slot information corresponding to the conversation sentence, the language representation model is used to extract the features of each slot information to obtain the slot vector corresponding to each slot information; the probability distribution of each slot information is determined based on each slot vector and the feature vector of the domain and intention; the intent and slot information corresponding to the conversation sentence are determined based on the probability distribution of each intention and the probability distribution of each slot information, the highest intent in the probability distribution of each intention is the intent corresponding to the conversation sentence, and the conversation state information corresponding to each conversation sentence of the user is obtained; the conversation state information is obtained from the service device and output.

2. A service device, characterized in that: include: An acquisition unit, configured to acquire the time interval between the user's current query and the previous query, the similarity between the current query and the previous query, and the location domain of the current query and its score; a judgment unit, configured to judge whether to perform session segmentation on the query statements of the current round and the query statements of the previous round based on the time interval between the query statements of the current round and the query statements of the previous round, the similarity between the query statements of the current round and the query statements of the previous round, and the positioning domain of the query statements of the current round and the score thereof; a segmentation unit, configured to segment the human-computer interaction session of the user according to a determination result of the session segmentation, and obtain at least one segmented session statement corresponding to the user; a processing unit, configured to process each conversation statement using a preset deep learning model to obtain dialogue state information corresponding to each conversation statement of the user; The processing unit is specifically configured to extract features from the conversational sentence according to a preset language representation model to obtain a conversational feature vector corresponding to the conversational sentence; input the conversational feature vector into the first module of the preset deep learning model to obtain the intent of the conversational sentence and the probability distribution of each intent; form a main idea sentence with each domain word related to the conversational sentence and each of the intents, and extract features from the main idea sentence using the language representation model to obtain a main idea vector; The main idea vector and the conversation feature vector are input into the second module of the preset deep learning model to obtain the feature vector of the domain and intent corresponding to the conversation sentence; for each slot information corresponding to the conversation sentence, the language representation model is used to extract features of each slot information to obtain the slot vector corresponding to each slot information; based on each slot vector and the feature vector of the domain and intent, the probability distribution of each slot information is determined; based on the probability distribution of each intent and the probability distribution of each slot information, the intent and slot information corresponding to the conversation sentence are determined, and the intent with the highest probability in the probability distribution of each intent is the intent corresponding to the conversation sentence.

3. The service device according to claim 2, characterized in that The judgment unit is specifically configured to not perform session segmentation on the query statement in the current round and the query statement in the previous round if the time interval between the query statement in the current round and the query statement in the previous round does not exceed a set time threshold; the similarity between the query statement in the current round and the query statement in the previous round is greater than a set similarity threshold; the positioning domain of the query statement in the current round is an unknown domain; and the positioning domain of the query statement in the current round and the positioning domain of the query statement in the previous round meet a preset domain condition limitation; Otherwise, the current round query statement and the previous round query statement are divided into sessions.

4. The service device according to claim 2, characterized in that The positioning domain of the current round of query statements and the positioning domain of the previous round of query statements meet the preset domain condition restrictions, including: The positioning domain score of the current round of query statements is greater than the first set score threshold, and the relevance with the positioning domain of the previous round of query statements exceeds the set relevance threshold; or, The positioning domain of the current query statement and the positioning domain of the previous query statement are not easily confused, the scores of the positioning domain of the current query statement and the positioning domain of the previous query statement are both less than the second set score threshold, or the slot information of the previous query statement is empty; or, The positioning domain of the query statement in this round is the same as the positioning domain of the query statement in the previous round.

5. The service device according to any one of claims 2 to 3, characterized in that: It also includes a linear speed determination unit, which is used to determine the similarity according to the proportion of the same elements in the text of the first set of query statements in this round and the second set of query statements in the previous round.

6. A method for tracking a conversation state, characterized in that: Applied to smart devices, including: The current round of query statements input through the input interface is sent to the service device, so that the service device obtains the time interval between the current round of query statements and the previous round of query statements, the similarity between the current round of query statements and the previous round of query statements, and the positioning field and score of the current round of query statements; according to the time interval between the current round of query statements and the previous round of query statements, the similarity between the current round of query statements and the previous round of query statements, and the positioning field and score of the current round of query statements, it is judged whether to perform conversation segmentation on the current round of query statements and the previous round of query statements; according to the judgment result of the conversation segmentation, the human-computer interaction conversation of the user is segmented to obtain at least one segmented conversation statement corresponding to the user; according to the preset language representation model, the feature extraction of the conversation statement is performed to obtain the conversation feature vector corresponding to the conversation statement; the conversation feature vector is input into the first module of the preset deep learning model to obtain the intention of the conversation statement and each intention concept rate distribution; each domain word related to the conversation sentence and each of the intentions form a main intention sentence, and use the language representation model to extract features of the main intention sentence to obtain a main intention vector; the main intention vector and the conversation feature vector are input into the second module of the preset deep learning model to obtain the feature vector of the domain and intention corresponding to the conversation sentence; for each slot information corresponding to the conversation sentence, the language representation model is used to extract features of each slot information to obtain the slot vector corresponding to each slot information; based on each slot vector and the feature vector of the domain and intention, the probability distribution of each slot information is determined; based on the probability distribution of each intention and the probability distribution of each slot information, the intent and slot information corresponding to the conversation sentence are determined, the highest intent in the probability distribution of each intent is the intent corresponding to the conversation sentence, and the dialogue state information corresponding to each conversation sentence of the user is obtained; The session state information is acquired from the service device and outputted.

7. A method for tracking a conversation state, characterized in that: Applied to service equipment, including: Obtain the time interval between the user's current query and the previous query, the similarity between the current query and the previous query, and the positioning domain and score of the current query; Determine whether to perform session segmentation on the query statements in the current round and the query statements in the previous round based on the time interval between the query statements in the current round and the query statements in the previous round, the similarity between the query statements in the current round and the query statements in the previous round, and the positioning domain and score of the query statements in the current round; Segmenting the user's human-computer interaction session according to the judgment result of the session segmentation to obtain at least one segmented session sentence corresponding to the user; The method further comprises: extracting features of the conversation sentence according to a preset language representation model to obtain a conversation feature vector corresponding to the conversation sentence; inputting the conversation feature vector into a first module of a preset deep learning model to obtain the intent of the conversation sentence and the probability distribution of each intent; forming a main idea sentence with each domain word related to the conversation sentence and each intent, and extracting features of the main idea sentence using the language representation model to obtain a main idea vector; inputting the main idea vector and the conversation feature vector into a second module of the preset deep learning model to obtain a feature vector of the domain and intent corresponding to the conversation sentence; extracting features of each slot information corresponding to the conversation sentence using the language representation model to obtain a slot vector corresponding to each slot information; determining the probability distribution of each slot information based on each slot vector and the feature vector of the domain and intent; determining the intent and slot information corresponding to the conversation sentence based on the probability distribution of each intent and the probability distribution of each slot information; the highest intent in the probability distribution of each intent is the intent corresponding to the conversation sentence, and obtaining the dialogue state information corresponding to each conversation sentence of the user.

8. The method according to claim 7, characterized in that The determining whether to perform session segmentation on the query statements in the current round and the query statements in the previous round according to the time interval between the query statements in the current round and the query statements in the previous round, the similarity between the query statements in the current round and the query statements in the previous round, and the positioning domain and score of the query statements in the current round includes: If the time interval between the current query statement and the previous query statement does not exceed the set time threshold; the similarity between the current query statement and the previous query statement is greater than the set similarity threshold; the positioning domain of the current query statement is an unknown domain; the positioning domain of the current query statement and the positioning domain of the previous query statement meet the preset domain condition, then the current query statement and the previous query statement will not be session split; Otherwise, the current round query statement and the previous round query statement are divided into sessions.

Images