Question and answer method and device and electronic equipment

Abstract

The invention relates to the technical field of electronic equipment, and discloses a question and answer method and device and electronic equipment, which are used for providing a question and answer accompanying service for a user. The method comprises the steps that the terminal equipment displays a question and answer dialogue window and a display window, and display content is displayed in the display window. The terminal device can display first dialogue information in a question-answer dialogue window in response to a question-answer input operation of a user, and the first dialogue information is used for asking a question for display content displayed in a display window. And the terminal equipment sends the first dialogue information and timestamp information to a server, wherein the timestamp information is used for representing a timestamp of a key frame of which the display content is related to the question content of the first dialogue information. And the server determines answer information corresponding to the first dialogue information according to the received timestamp information and the first dialogue information, and returns the answer information to the terminal equipment for display. In this way, the question and answer service can be provided for the user in the display content display process, and therefore the use experience of the user can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a question-answering method, device, and electronic device. Background Art

[0002] In the current era of mobile Internet and big data, video data on the Internet is growing explosively. Existing applications or websites can provide users with video content viewing services. However, as users watch more, they need to ask questions and get explanations about certain objects / images / plots in the picture in certain specific areas, such as watching the World Cup live and wanting to discuss the game trends, watching a drama and needing to discuss the plot, etc. Therefore, there is an urgent need for a question-and-answer method that provides users with a viewing companion service. Summary of the invention

[0003] The embodiments of the present application provide a question-and-answer method, device, and electronic device for providing users with a question-and-answer service, thereby improving the user experience.

[0004] In a first aspect, an embodiment of the present application provides a question-answering method, which is applied to a terminal device. The method comprises:

[0005] The terminal device displays a question-and-answer dialogue window and a display window, wherein the display window displays display content. The terminal device may display first dialogue information in the question-and-answer dialogue window in response to the user's question-and-answer input operation, wherein the first dialogue information is used to ask questions about the display content displayed in the display window. The terminal device may determine the timestamp information corresponding to the first dialogue information, and send the timestamp information and the first dialogue information to the server. The timestamp information is used to characterize the timestamp of a key frame in the display content related to the question content of the first dialogue information. After receiving the answer information returned by the server, the terminal device displays the answer information in the question-and-answer dialogue window.

[0006] In this method, the terminal device can perform question-and-answer interaction with the user during the display of the content, and provide the user with the information he needs. In addition, after the user inputs the first dialogue information, the terminal device can send the first dialogue information and the timestamp information corresponding to the first dialogue information to the server, so that the server can quickly and accurately output the answer information according to the user's question content and the key frames related to the user's question content, thereby improving the user's viewing experience.

[0007] In an optional design, in response to the user's question-and-answer input operation, before the question-and-answer dialogue window displays the first dialogue information, the terminal device may also display the question-and-answer dialogue window in response to the user's dialogue operation when displaying the display window. The terminal device may also obtain source information of the content displayed in the display window, classify the source information, and determine type information corresponding to the displayed content. The terminal device may also send the source information and type information to the server.

[0008] With this design, after displaying the question-and-answer dialogue window, the terminal device sends the source information and type information of the display content displayed in the display window to the server, so that the server can obtain the display data of the display content according to the source information and type information, and pre-process the display data. In this way, when the user subsequently inputs the first dialogue information, after the terminal device sends the first dialogue information and the timestamp information corresponding to the first dialogue information to the server, the server can quickly and accurately determine the answer information corresponding to the first dialogue information based on the pre-processed display data, thereby ensuring the real-time nature of the answer to the first dialogue information.

[0009] In an optional design, the first dialogue message is a text message or a voice message.

[0010] In an optional design, the terminal device may also determine the first timestamp of the first dialogue information input by the user, and determine the time period during which the first display content where the key frame is located is displayed in the display window according to the first timestamp. The terminal device may extract the first display content from the display content according to the time period. The terminal device may also identify the question content corresponding to the first dialogue information, extract the key frame from the first display content according to the question content, and use the timestamp information corresponding to the key frame as the timestamp information corresponding to the first dialogue information.

[0011] Through this design, the terminal device can determine the key frames related to the question content from the displayed content, so that the server can directly understand the key frames in the displayed content without having to understand the overall content of the displayed content, thereby improving the rate of intent understanding and dialogue generation.

[0012] In an optional design, the terminal device may also obtain context information of the first dialogue information, and record the first dialogue information and the context information in an array in chronological order to obtain a question-answer sequence. The terminal device may send the question-answer sequence and timestamp information to the server.

[0013] Through this design, the terminal device can send a question-answer sequence and timestamp information to the server, so that the server can determine the answer information based on the key frames corresponding to the context information and timestamp information, thereby providing users with contextual dialogue functions and customized viewing intention understanding, and further enabling long-term dialogue and discussion with users based on a certain content, thereby improving the user experience.

[0014] In an optional design, the terminal device may also record the answer information into the question and answer sequence.

[0015] Through this design, the integrity of contextual information in the question-answer sequence can be guaranteed.

[0016] In a second aspect, the present application provides a question-answering method, which is applied to a server. The method comprises:

[0017] The server may receive the timestamp information and the first dialogue information sent by the terminal device, the first dialogue information being used to ask questions about the display content displayed in the display window of the terminal device, and the timestamp information being used to represent the timestamp of a key frame in the display content related to the question content of the first dialogue information. The server may also determine the answer information corresponding to the first dialogue information based on the timestamp information and the first dialogue information, and send the answer information to the terminal device.

[0018] Through this method, the server can analyze the key frames related to the question content of the first conversation information based on the first conversation information, quickly understand the intention and generate the conversation, thereby ensuring the real-time nature of the conversation and improving the user's viewing experience.

[0019] In an optional design, the server may also receive timestamp information and a question-and-answer sequence sent by the terminal device, where the question-and-answer training includes the first dialogue information and context information of the first dialogue information.

[0020] Through this design, the server can understand the context of the displayed content according to the context information of the first conversation information, so that the user can obtain the required information more conveniently and comprehensively.

[0021] In an optional design, the server may also encode the key frame and question-answer sequence corresponding to the timestamp information to obtain a coding vector. The server may input the coding vector into the question-answer generation model, analyze the coding vector based on the question-answer generation model, and obtain the answer information.

[0022] Through this design, the server can output accurate answer information to users through the question-answer generation model.

[0023] In an optional design, the server may also determine the display vector corresponding to the display content, and extract the display vector corresponding to the key frame from the display vector according to the timestamp information. The server may encode the display vector corresponding to the key frame and the question-answer sequence to obtain an encoding vector.

[0024] Through this design, when the server determines the answer information corresponding to the first dialogue information, it can determine the display vector corresponding to the key frame related to the question content of the first dialogue information based on the display vector obtained by pre-data processing, thereby reducing the data processing time and quickly generating dialogues, thereby ensuring the real-time nature of the dialogue and improving the user's viewing experience.

[0025] In an optional design, the server may also obtain a question and answer sample from a database of the server, encode the question and answer sample, and obtain a question and answer vector. The server may input the question and answer vector and the encoding vector into a question and answer generation model, analyze the encoding vector and the question and answer vector based on the question and answer generation model, and obtain answer information corresponding to the first dialogue information.

[0026] Through this design, the server can also input the question-answer vector and the encoding vector into the question-answer generation model, so that the question-answer generation model can perform a more comprehensive analysis of the question and give accurate answer information.

[0027] In an optional design, before receiving the timestamp information and the first conversation information sent by the terminal device, the server may also receive source information of the display content and type information corresponding to the display content from the terminal device. The server may obtain display data corresponding to the display content according to the source information and type information, and vectorize the display data to obtain a display vector.

[0028] Through this design, the server can pre-acquire the display data corresponding to the display content according to the source information and type information, providing a data basis for subsequent question and answer analysis, thereby reducing the time to determine the key frame corresponding to the timestamp information, and further increasing the conversation generation rate.

[0029] In an optional design, the server may also determine whether the display data is included in the database of the server according to the type information. If the display data is included in the database, the server may obtain the display data from the database based on the source information. Alternatively, if the database does not contain the display data, the server may also generate a data address of the display data based on the source information, and obtain the display data according to the data address.

[0030] Through this design, the server can obtain the display data corresponding to the display content in advance for subsequent processing.

[0031] In a third aspect, the present application provides a question-answering device, applied to a terminal device, the question-answering device comprising:

[0032] A display screen, used for displaying a user interface;

[0033] A communication unit for receiving and sending data;

[0034] A processing unit, used to display a question-and-answer dialogue window and a display window on the display screen; display content is displayed in the display window; in response to a user's question-and-answer input operation, a first dialogue message is displayed in the question-and-answer dialogue window; the first dialogue message is used to ask questions about the display content displayed in the display window; determine the timestamp information corresponding to the first dialogue message; the timestamp information is used to represent the timestamp of a key frame in the display content related to the question content of the first dialogue message; send the timestamp information and the first dialogue message to a server; receive the answer information returned by the server, and display the answer information in the question-and-answer dialogue window.

[0035] In a fourth aspect, the present application provides a question-answering device, applied to a server, the question-answering device comprising:

[0036] A communication unit for receiving and sending data;

[0037] A processing unit is used to receive timestamp information and first dialogue information sent by a terminal device; the first dialogue information is used to ask questions about display content displayed in a display window of the terminal device; the timestamp information is used to represent the timestamp of a key frame in the display content related to the question content of the first dialogue information; based on the timestamp information and the first dialogue information, determine answer information corresponding to the first dialogue information; and send the answer information to the terminal device.

[0038] In a fifth aspect, the present application provides an electronic device, comprising a display screen, one or more processors, and one or more memories. The display screen is used to display a user interface; the one or more memories are used to store one or more computer programs and data information; the one or more processors are used to execute the computer programs stored in the one or more memories, so that the electronic device executes the method described in any aspect of the first aspect. The electronic device may be the terminal device of the first aspect.

[0039] In a sixth aspect, the present application provides an electronic device, comprising one or more processors and one or more memories. The one or more memories are used to store one or more computer programs and data information, and the one or more processors are used to execute the computer programs stored in the one or more memories, so that the electronic device performs the method described in any aspect of the second aspect. The electronic device may be a server in the second aspect.

[0040] In the seventh aspect, the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed by a computing device, the computing device executes the method in any one of the above-mentioned first aspect or second aspect, and any possible implementation of any aspect.

[0041] In an eighth aspect, an embodiment of the present application provides a chip, which is coupled to a memory in an electronic device so that the chip calls a computer program stored in the memory during operation to implement a method that may be designed in any of the first or second aspects of the embodiment of the present application.

[0042] In a ninth aspect, the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a computing device, the computing device executes the method in any one of the above-mentioned first aspect or second aspect, and any possible implementation of any aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The following is a schematic diagram of the architecture of a question-answering system;

[0044] Figure 2 A schematic diagram of a possible hardware structure of an electronic device provided as an example for this application;

[0045] Figure 3 A flowchart of a question-and-answer method provided for this application;

[0046] Figure 4 A schematic diagram of a display interface provided for this application;

[0047] Figure 5 Another display interface schematic diagram provided for this application;

[0048] Figure 6 A schematic diagram of a question-answer input process provided for this application;

[0049] Figure 7 A training diagram of a question-answer generation model provided in this application;

[0050] Figure 8A schematic diagram of generating answer information by a question-answer generation model provided in this application;

[0051] Fig. 9 A schematic diagram of another display interface provided by this application;

[0052] Fig.10 A schematic diagram of another display interface provided by this application;

[0053] Fig.11 A schematic diagram of another display interface provided by this application;

[0054] Fig.12 A schematic diagram of the application architecture of a terminal device provided for this application;

[0055] Fig.13 A schematic diagram of the application architecture of a server provided for this application;

[0056] Fig.14 A schematic diagram of the architecture of a question-answering system provided for this application;

[0057] Fig.15 A schematic diagram of a question-and-answer use case provided for this application;

[0058] Fig.16 A complete flowchart of a question-and-answer method provided for this application;

[0059] Fig.17 A schematic diagram of the structure of a question-and-answer device provided in this application;

[0060] Fig.18 A schematic diagram of the structure of another question-and-answer device provided in this application. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the following embodiments of the present application.

[0062] First, concepts related to the embodiments of the present application are explained.

[0063] (1) Artificial intelligence generated content (AIGC) refers to the production, manipulation and modification of data or media through artificial intelligence algorithms, that is, content creation based on generative models (such as GPT, T5, etc.).

[0064] (2) Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It integrates linguistics, computer science, and mathematics, and studies various theories and methods that can enable effective communication between humans and computers using natural language.

[0065] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0066] Unless otherwise specified, ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the size, content, order, timing, priority or importance of multiple objects. For example, the first file and the second file are only used to distinguish different files, and do not indicate the difference in size, content, priority or importance of the two files.

[0067] In the current era of mobile Internet and big data, video data on the Internet is growing explosively. Existing applications or websites can provide users with video content viewing services. However, as users watch more, they need to ask questions and get explanations about certain objects / images / plots in the picture in certain specific areas, such as watching the World Cup live and wanting to discuss the game trends, watching a drama and needing to discuss the plot, etc. Therefore, there is an urgent need for a question-and-answer method that provides users with a viewing companion service.

[0068] Based on the above problems, the embodiments of the present application provide a question-and-answer method, device, and electronic device to provide users with a question-and-answer service. In this method, the terminal device can display a question-and-answer dialogue window and a display window, and the display window displays display content. The terminal device can display a first dialogue message in the question-and-answer dialogue window in response to the user's question-and-answer input operation, and the first dialogue message is used to ask questions about the display content displayed in the display window. The terminal device determines the timestamp information corresponding to the first dialogue message, and sends the timestamp information and the first dialogue message to the server. The server determines the answer information corresponding to the first dialogue message based on the timestamp information and the first dialogue message, and sends the answer information to the terminal device. The terminal device can display the received answer information in the question-and-answer dialogue window, perform question-and-answer interaction with the user, and provide the user with the information he needs, thereby improving the user's viewing experience.

[0069] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0070] Figure 1 A schematic diagram of the architecture of a question-answering system applicable to the method provided in the embodiment of the present application. Specifically, the question-answering system includes a terminal device 10 and a server 20. The terminal device 10 and the server 20 can communicate with each other through a communication network.

[0071] The communication network may be a local area network or a wide area network transferred by a relay device. When the communication network is a local area network, exemplarily, the communication network may be a short-distance communication network such as a Wi-Fi hotspot network, a Wi-Fi direct network, a Bluetooth network, a Zigbee network, or a near field communication (NFC) network. When the communication network is a wide area network, exemplarily, the communication network may be a third-generation wireless telephone technology (3G) network, a fourth-generation mobile communication technology (4G) network, a fifth-generation mobile communication technology (5G) network, a future-evolved public land mobile network (PLMN) or the Internet.

[0072] In some embodiments, the terminal device 10 is used to play display content, such as videos, games, live broadcasts, other files, images, audio, documents, etc., and interact with users. The server is used to generate dialogues. Exemplarily, the terminal device 10 can be a mobile phone, a tablet computer, a laptop computer, etc. The terminal device 10 can display a first dialogue message in a question-and-answer dialogue window in response to a user's question-and-answer input operation, and the first dialogue message is used to ask questions about the display content displayed in the display window. The terminal device 10 extracts core information from the display content and determines the timestamp information of the key frame related to the question content of the first dialogue message. The terminal device 10 sends the timestamp information and the first dialogue message to the server 20. The server 20 determines the answer information corresponding to the first dialogue message based on the timestamp information and the first dialogue message. The server 20 sends the answer information to the terminal device 10, and the terminal device 10 displays the answer information in the question-and-answer dialogue window.

[0073] Figure 2 A possible hardware structure diagram of an electronic device is shown. The electronic device 100 may be Figure 1 The terminal device 10 in the embodiment may also be Figure 1 Server 20 in the Figure 2 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0074] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a microcontroller unit (MCU), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors. Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. A memory may also be set in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a high-speed cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated access is avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.

[0075] In the embodiment of the present application, the processor 110 may store one or more applications, and in response to the user's question and answer input operation, display the first dialogue information on the display screen 194, and extract the timestamp information of the key frame related to the question content of the first dialogue information from the display content displayed on the display screen 194. In addition, the processor 110 may also determine the answer information corresponding to the first dialogue information based on the first dialogue information and the timestamp information.

[0076] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and peripheral devices. The charging management module 140 is used to receive charging input from the charger. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160.

[0077] The wireless communication function of the electronic device 100 can be implemented by antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modulation and demodulation processor and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antenna. For example, antenna 1 can be reused as a diversity antenna of a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0078] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0079] The wireless communication module 160 can provide wireless communication solutions including wireless local area network (WLAN) (such as wireless fidelity (Wi-Fi) network), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0080] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-CDMA), long term evolution (LTE), the fifth generation (5G) mobile communication system, future communication systems such as the sixth generation (6G) system, BT, GNSS, WLAN, NFC, FM and / or IR technology, etc. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0081] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1. In an embodiment of the present application, the display screen 194 can be used to display a main interface, an application interface, a granularity adjustment control, etc.

[0082] The camera 193 is used to capture still images or videos. The camera 193 may include a front camera and a rear camera.

[0083] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, and software code of at least one application (such as Huawei Video, Changlian, etc.). The data storage area can store data (such as images, videos, etc.) generated during the use of the electronic device 100. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0084] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing files such as pictures and videos in the external memory card.

[0085] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0086] Understandably, Figure 2 The components shown do not constitute a specific limitation on the electronic device, and the electronic device may also include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. Figure 2 The electronic device 100 shown is taken as an example for introduction.

[0087] In some scenarios, the electronic device can be a server. In this scenario, the server obtains data by communicating with the user's terminal device through a built-in application. The server can be deployed in the cloud or locally.

[0088] In one case of this scenario, the electronic device may be a terminal device with a display function, and the terminal device provides an interactive interface to the user, and the terminal device obtains data through the interactive interface. For example, the terminal device obtains data in response to the user's input operation on the interactive interface.

[0089] It should be noted that the terminal device involved in the question-answering method provided in the embodiment of the present application may be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, a drone, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a smart TV, and other display-supporting devices. The specific form of the terminal device is not particularly limited in the embodiment of the present application. For example, the terminal device may be a mobile phone or a tablet computer.

[0090] The following specifically describes the question-answering method used in the question-answering system composed of a terminal device and a server according to a specific embodiment.

[0091] In order to facilitate understanding of a question-and-answer method provided by this application, the following Figures 3 to 18 The content shown in introduces the implementation process of the method provided by this application.

[0092] Figure 3 A flowchart of a question-and-answer method provided in an embodiment of the present application. The process includes:

[0093] S301: The terminal device displays a question-and-answer dialog window and a display window, wherein the display window displays display content.

[0094] In some embodiments, the terminal device may display display content in the display window in response to the user's operation. Exemplarily, the display content may be video, game, live broadcast, image, document, etc. It should be noted that the terminal device may display dynamically changing content according to the type of display content (video, game, live broadcast), or may display dynamically changing content based on the user's control operation (e.g., adjusting the display content of the document, adjusting the image).

[0095] As an example, a user may trigger a play operation on a terminal device, and the terminal device displays the video content in a display window in response to the user's play operation. For example, a user may click an application loaded on the terminal device to play the displayed content. Figure 4 As shown, the user can click on the video in the browser to play the video content. For another example, the user can click on the video in the video application to play the video content. In addition, the user can also click on the video on the negative first screen of the terminal device to play the video content. Among them, the negative first screen is the interface presented by the intelligent assistant function of the terminal device.

[0096] When displaying the display window, the terminal device may display a question-and-answer dialog window in response to the user's dialog operation. Figure 4 For example, the user can click the watch with you button to trigger the conversation operation. Figure 5 As shown, the terminal device can display a question-and-answer dialogue window in response to the user's dialogue operation. The user can communicate with the companion robot plug-in or the dialogue robot in the question-and-answer dialogue window. As an example, when the user uses a browser or a negative screen to watch the displayed content, the question-and-answer dialogue window can be displayed below the display window, and the user can communicate with the dialogue robot in the question-and-answer dialogue window. As another example, when the user uses a video application to watch the displayed content, the question-and-answer dialogue window can be displayed as a dialogue floating window on the video application, and the user can talk to the robot in the dialogue floating window.

[0097] After displaying the question-and-answer dialogue window, the terminal device can obtain the source information of the content displayed in the display window. Exemplarily, the source information includes but is not limited to: title, source, id. After obtaining the source information of the displayed content, the terminal device can classify the source information and determine the type information corresponding to the displayed content. For example, when the displayed content is live content, the source information obtained by the terminal device can be as follows:

[0098] Live / video title (video_title): "2022 CCTV World Cup live broadcast France-Argentina";

[0099] Live broadcast / video source (source): "first provider";

[0100] Live broadcast / video id (video_id): 12345678.

[0101] The terminal device classifies the above source information to obtain type information corresponding to the displayed content. The type information is as follows:

[0102] Live / Video Type (video_type): "Live";

[0103] Live broadcast / video first-level category (video_cat1): "football";

[0104] Live broadcast / video secondary category (video_cat2): "World Cup".

[0105] The terminal device can also assemble the obtained source information and type information into JSON format, as shown below:

[0106]

[0107] After obtaining the source information and type information, the terminal device may also send the source information and type information to the server. After receiving the source information and type information sent by the terminal device, the server may obtain display data corresponding to the display content according to the source information and type information.

[0108] As an example, the server can determine whether the display data corresponding to the display content is contained in the server's database based on the type information. Exemplarily, the server can determine whether the display content is provided by a contracted content provider based on the type information. Among them, the contracted content provider will upload or synchronize the display data to the server's database in advance. For example, based on the "source" field, the server can determine whether the first provider is a contracted content provider that has signed a contract with the server to provide display data for it. When the first provider is a contracted content provider, the server can also determine whether the display data is contained in the server's database based on the type information. For another example, when the first provider is an uncontracted content provider, the server can determine that the display data is not contained in the server's database.

[0109] Optionally, when the server determines that the database contains display data, the display data is obtained from the database based on the source information. Exemplarily, when the server determines that the first provider hosts the display data corresponding to the type of information in the server's data, the display data can be obtained from the database based on the source information. For example, the server can directly pull the display data of "video_id": 12345678 from the database.

[0110] Optionally, when the server determines that the database does not contain the display data, it can generate a data address for the display data based on the source information. For example, when the server determines that the database does not contain the display data, it can generate a data address URL for the display data based on the "video_id": 12345678 field. Exemplarily, the URL can be: http: / / douyin.com / video_id=12345678 / . After generating the data address, the server can obtain the display data based on the data address. Exemplarily, the server can call a third-party data interface to pull the corresponding display data based on the data address.

[0111] In some embodiments, after acquiring the display data of the display content, the server may perform vectorization processing on the display data to obtain a display vector, wherein the display vector is used to represent the display content.

[0112] S302: The terminal device displays first dialogue information in the question-and-answer dialogue window in response to the user's question-and-answer input operation.

[0113] The first dialogue information is used to ask questions about the display content displayed in the display window.

[0114] The user can trigger the question-and-answer input operation in the question-and-answer dialog window on the terminal device, and input the first dialog information to ask questions about the display content displayed in the display window. Exemplarily, the first dialog information input by the user can be text information or voice information. Figure 6 For example, the user can click the "Q" button in the question-and-answer dialogue window to trigger the question-and-answer input operation. The user can choose to input text information or voice information.

[0115] The terminal device may display the first dialogue information in the question-and-answer dialogue window in response to the user's question-and-answer input operation. Figure 6 For example, the user clicks the "Q" button in the question-and-answer dialogue window to trigger the question-and-answer input operation and inputs the first dialogue information of "Who is this person?" The terminal device responds to the user's question-and-answer input operation and displays "Who is this person" in the question-and-answer dialogue window.

[0116] In some embodiments, after the terminal device displays the first dialogue information in the question-and-answer dialogue window, it can also obtain the context information of the first dialogue information, and record the first dialogue information and the context information in an array in chronological order to obtain a question-and-answer sequence. For example, after the terminal device displays "Who is this person" in the question-and-answer dialogue window, it is determined that the first dialogue information of "Who is this person" has no context information, and the first dialogue information is recorded in an array to obtain a question-and-answer sequence. Among them, the question-and-answer sequence can be {"Who is this person"}.

[0117] S303: The terminal device determines the timestamp information corresponding to the first dialogue information, where the timestamp information is used to represent the timestamp of a key frame in the displayed content that is related to the question content of the first dialogue information.

[0118] In some embodiments, after displaying the first dialogue information, the terminal device may extract key frames related to the question content from the displayed content based on the first dialogue information.

[0119] Optionally, the terminal device may determine a first timestamp when the user inputs the first dialogue information, and determine, based on the first timestamp, a time period during which the first display content where the key frame is located is displayed in the display window. For example, the first timestamp may be the moment when the user inputs the first dialogue information. For example, the terminal device may use the display content within a period of time before and after the first timestamp as the first display content. The period of time before and after the first timestamp may be preset. For example, the terminal device may use the display content displayed in the display window 3 seconds before and after the first timestamp as the first display content.

[0120] The terminal device may extract the first display content from the display content according to the determined time period. The terminal device may identify the question content of the first dialogue information and extract the key frame from the first display content according to the question content.

[0121] Exemplarily, the terminal device may use a pre-trained small model to identify the question content of the first dialogue information and extract key frames from the first display content. Specifically, the terminal device may input the first dialogue information and the first display content into the pre-trained small model, identify the question content of the first dialogue information based on the pre-trained small model, and extract key frames from the first display content according to the identified question content. The pre-trained small model may be a model that is integrated on the terminal device and has been trained offline.

[0122] After determining the key frame, the terminal device uses the timestamp information corresponding to the key frame as the timestamp information corresponding to the first dialogue information. The terminal device can also map the timestamp information to a corresponding time format according to the type of displayed content. For example, live broadcast is mapped to timestamp [1683603779175,1683603789180], and video is mapped to seconds [1654,1743].

[0123] S304: The terminal device sends the timestamp information and the first conversation information to the server. The server receives the first conversation information and the timestamp information.

[0124] The terminal device sends the timestamp information corresponding to the first dialogue information to the server instead of sending the key frames in the display content related to the question content of the first dialogue information to the server, which can reduce data transmission overhead and improve transmission efficiency.

[0125] In some embodiments, the terminal device may assemble the timestamp information and the first conversation information into a json format to reduce the amount of data transmitted. The terminal device sends the timestamp information and the first conversation information assembled into a json format to the server, as shown below:

[0126]

[0127] In other embodiments, after the terminal device determines the question-answer sequence, the terminal device may also send timestamp information and the question-answer sequence to the server. The server receives the timestamp information and the question-answer sequence sent by the terminal device. Since the question-answer sequence contains the context information of the first dialogue information, the server can determine the answer information of the first dialogue information in combination with the context information, so that the answer information determined by the server is more accurate. Exemplarily, the terminal device may assemble the timestamp information and the question-answer sequence into a json format and send it to the server, as shown below:

[0128]

[0129] S305: The server determines the answer information corresponding to the first dialogue information according to the timestamp information and the first dialogue information.

[0130] In some embodiments, the server may encode the key frame and the question-and-answer sequence corresponding to the timestamp information to obtain a coding vector. The question-and-answer sequence includes the first dialogue information and the context information of the first dialogue information. Exemplarily, after receiving the timestamp information and the first dialogue information, the server may obtain the pre-extracted display vector from the database. The server may extract the display vector corresponding to the key frame from the display vector according to the timestamp information, and encode the display vector corresponding to the key frame and the question-and-answer sequence to obtain a coding vector.

[0131] The server may also input the encoding vector into a question-answer generation model, analyze the encoding vector based on the question-answer generation model, and obtain answer information.

[0132] The question-answer generation model may be a trained AIGC model. The AIGC model can combine multimodal data and can generate and convert between multiple modalities such as text, images, and audio. For example, in the offline stage, the embodiment of the present application may train the AIGC model and integrate the trained AIGC model into the server. Figure 7As shown, in the offline stage, the embodiment of the present application can encode the text / audio / picture / video data through Multi-Encoder, send the encoded vector to the model for training, and obtain the model parameters. In addition, the embodiment of the present application can also store the text / audio / picture / video data used and generated in the training process as question and answer samples in the database of the server. In this way, when the online content is analyzed later, the question and answer samples can be used to perform multi-dimensional analysis of the online content to improve the accuracy of the output answer results.

[0133] As an example, the server may also obtain a question and answer sample from a database of the server, and encode the question and answer sample to obtain a question and answer vector. The server may input the question and answer vector and the encoding vector into a question and answer generation model, analyze the encoding vector and the question and answer vector based on the question and answer generation model, and obtain answer information corresponding to the first dialogue information.

[0134] For example, Figure 8 As shown, the server can encode the question and answer sample to obtain a question and answer vector. The server can determine the display vector corresponding to the key frame from the display vector according to the timestamp information. The server can also encode the display vector and the question and answer sequence corresponding to the key frame to obtain a coding vector. The server inputs the coding vector and the question and answer vector into the AIGC model, analyzes the coding vector and the question and answer vector based on the AIGC model, and outputs the answer information.

[0135] S306: The server sends the answer information to the terminal device. The terminal device receives the answer information sent by the server.

[0136] S307: The terminal device displays the answer information in the question-and-answer dialogue window.

[0137] For example, Fig. 9 As shown, the first dialogue information input by the user in the question-and-answer dialogue window is "Who is this person?", and the user conducts a dialogue with the robot on the terminal device. The terminal device sends the first dialogue information and the timestamp information corresponding to the first dialogue information to the server providing services for the robot. The server determines the answer information "Xiao A" corresponding to the first dialogue information based on the first dialogue information and the timestamp information, and returns the answer information to the terminal device. The terminal device displays the answer information in the question-and-answer dialogue window, and the robot completes the answer to the user's question.

[0138] For example, Fig.10As shown, the user can ask a series of questions based on the display content of the first question. The terminal device can obtain the context information of the first dialogue information and generate a question-and-answer sequence. Among them, the question-and-answer sequence is {question 1, answer 1, question 2}, and question 2 is the first dialogue information. Question 1 is "Who is this person", answer 1 is "this little A", and question 2 is "What song is he singing". The terminal device can send the question-and-answer sequence and the timestamp information corresponding to the first dialogue information to the server, and the server implements an accurate answer to the first dialogue information based on the question-and-answer sequence and the timestamp information. The server sends the generated answer information to the terminal device, and after the terminal device receives the answer information, it displays the answer information in the question-and-answer dialogue window. The answer information is "The Hat Song".

[0139] For example, Fig.11 As shown, the question-answer sequence {question 1, answer 1, question 2} sent by the terminal device to the server can be specifically: question 1 is "Is he XX?", answer 1 is "The person in the picture is Xiaoqiang, yes", and question 2 is "I don't understand how he changed." The server generates the answer information corresponding to question 2 based on the question-answer sequence and timestamp information, "The reasons for Xiaoqiang's change are very complicated, with the influence of his younger brother and the pressure of the external environment. Click to jump "Xiao Qiang's Story" ". Among them, the answer information in the " Xiao Qiang remember" " is a jump link. The server sends the answer information to the terminal device. After receiving the answer information, the terminal device displays the answer information in the question-and-answer dialogue window.

[0140] In some embodiments, the terminal device may also record the answer information into the question and answer sequence.

[0141] In an embodiment of the present application, the terminal device can provide the user with a question-and-answer service. The user can enter a first dialogue message for asking questions in the question-and-answer dialogue window. The terminal device can identify the first dialogue message, extract key frames related to the question content from the displayed content, and realize the user's intention to ask questions about specific details in the displayed content. The terminal device sends the first dialogue message and the timestamp information of the key frame to the server, which can reduce the time and power consumption of the server to generate answer information. The server can quickly understand the intention and generate dialogue based on the timestamp information and the first dialogue message, and output accurate answers, which can ensure the user's dialogue experience when watching in both effect and performance dimensions. In this way, the terminal device can interact with the user in real time regarding the displayed content, enhance the user's viewing experience, and enrich the content of the terminal device.

[0142] The content executed by the terminal device shown in the above embodiment may be executed by an application in the terminal device. Fig.12This is an application architecture diagram of a terminal device provided in an embodiment of the present application. The terminal device includes a data intelligent analysis module and a question-and-answer dialogue window.

[0143] Among them, the question-and-answer dialogue window is used for users to input voice / text and display the robot's answer information, thereby enabling users to interact with the robot.

[0144] The data intelligent analysis module is in the function package plug-in mode, which can modularize the functions to make them more flexible and scalable. The data intelligent analysis module includes a pre-trained model integrated on the terminal device side. The data intelligent analysis module is used to intelligently identify the displayed content, extract key frames and perform timestamp mapping, and contextualize the question and answer content.

[0145] like Fig.12 As shown, the data intelligent analysis module may include a speech recognition module, a video tag extraction module, a conversation content analysis module, a key frame extraction module, a timestamp recognition module and a data generation module.

[0146] Among them, the voice recognition module is used to recognize the voice information input by the user.

[0147] The video tag extraction module is used to obtain the source information of the displayed content, classify the source information, and determine the type information of the displayed content.

[0148] The dialogue content analysis module is used to identify the dialogue information input by the user in the question-and-answer dialogue window and the answer information input by the robot plug-in of the terminal device.

[0149] The data generation module is used to generate a question-answer sequence based on the context information of the dialogue information input by the user.

[0150] The key frame extraction module is used to input the first dialogue information and the first display content related to the user's question content into the pre-trained small model, and extract the key frame from the first display content based on the pre-trained small module.

[0151] The timestamp identification module is used to determine the timestamp of the first dialogue information input by the user, and determine the time period for displaying the first display content in the display window according to the timestamp.

[0152] The content executed by the server shown in the above embodiment can be executed by an application in the server. Fig.13 An application architecture diagram of a server provided in an embodiment of the present application. The server includes an AIGC generation model and a data processing module.

[0153] Among them, the data processing module is used to receive the timestamp information and question-answer sequence on the terminal device side, pull the corresponding display data from the server side based on the analyzed source information and type information, and perform vectorization and input standardization.

[0154] The AIGC generation model is a question-answer generation model deployed on the server side. It supports multimodal processing and can respond to user input in any form (text / image / video).

[0155] like Fig.13 As shown, the data processing module may include a video vectorization module and a model input generation module.

[0156] Among them, the video vectorization module is used to receive source information and type information on the terminal device side, obtain display data corresponding to the display content according to the source information and type information, and perform vectorization processing on the display data.

[0157] The model input generation module is used to receive the question and answer sequence and timestamp information on the terminal device side, and encode the key frames corresponding to the question and answer sequence and timestamp information to obtain a coding vector that conforms to the AIGC generation model input standard format.

[0158] Based on the contents described in the above embodiments, the embodiments of the present application provide a possible question-answering system. Fig.14 The following is a schematic diagram of the architecture of a question-answering system provided in an embodiment of the present application. The question-answering system includes a terminal device and a server.

[0159] The terminal device includes a display window for displaying content and a question-and-answer dialogue window, as well as a data intelligent analysis module. The server includes an AIGC generation module, a data processing module, a server database, and a third-party data interface.

[0160] The user can select the display content to be played and displayed on the terminal device to trigger the play operation. The terminal device responds to the user's operation and displays the display content in the display window. The terminal device can also determine whether the user has a dialogue requirement during the display of the display content.

[0161] When the user displays the content in the display window, the dialogue operation can be triggered. The terminal device responds to the user's dialogue operation and displays the question-and-answer dialogue window. The terminal device determines that the user has a dialogue demand and notifies the data intelligent analysis module of the terminal device to process.

[0162] The data intelligent analysis module of the terminal device can collect the source information of the displayed content, and classify the source information to obtain the type information of the displayed content. For example, the data intelligent analysis module can collect the source information of the live broadcast / video, intelligently classify and label the source information, and obtain the type information. Exemplarily, the information obtained after the data intelligent analysis module classifies and intelligently processes the source information can be as follows:

[0163] Title: 2022 CCTV World Cup Live France-Argentina, Type: Live, First Category: Football, Second Category: World Cup, Source: First Provider, Live ID: 12345678,

[0164] Title: Storm, Type: Video, First-level category: Suspense, Second-level category: None, Source: Second provider, Video id: 12345679.

[0165] The data intelligent analysis module of the terminal device can send the above analysis results to the data processing module of the server. The data processing module of the server can obtain the display data corresponding to the display content from the database / third-party data interface of the server according to the above analysis results. The data processing module of the server can also perform vector processing on the display data.

[0166] The user can also initiate a question-and-answer input operation in the question-and-answer dialogue window of the terminal device to ask questions about the displayed content. In response to the user's question-and-answer input operation, the terminal device displays the first dialogue information in the question-and-answer dialogue window. The data intelligent analysis module of the terminal device obtains the context information of the first dialogue information and generates a question-and-answer sequence. The data intelligent analysis module of the terminal device can also extract key frames from the displayed content based on the first dialogue information, and map the timestamps of the key frames to corresponding timestamps. For example, when the displayed content is live, the timestamp can be mapped to time_window: [1683603779175, 1683603789180]. For another example, when the displayed content is a video, the timestamp can be mapped to: time_window: [1654, 1743].

[0167] The data intelligent analysis module of the terminal device can also upload the timestamps of the question-answer sequence and the key frames to the data processing module of the server. The data processing module of the server integrates and encodes the question-answer sequence, the timestamp and the pre-extracted display vector to obtain a coding vector. The data processing module of the server inputs the coding vector into the AIGC intelligent question-answer model, and the AIGC intelligent question-answer model outputs the answer information. The data processing module of the server sends the answer information to the terminal device. The terminal device displays the answer information in the question-answer display window.

[0168] Fig.15An example of a question-answering method provided in an embodiment of the present application is provided. Taking the display content of the terminal device as video content as an example, the example includes the following steps.

[0169] S1501: The user selects video content to be played in the display interface of the terminal device and initiates a playback operation.

[0170] S1502: The application of the terminal device displays the video content in a display window in response to the user's play operation.

[0171] S1503: The user initiates a dialogue operation in the display window.

[0172] S1504: The application of the terminal device displays a question-and-answer dialogue window in response to the user's dialogue operation.

[0173] S1505: The application of the terminal device notifies the data intelligent analysis module to collect source information of the video content.

[0174] S1506: The data intelligent analysis module of the terminal device classifies the source information to obtain type information of the video content.

[0175] S1507: The data intelligent analysis module of the terminal device sends the type information and source information to the data processing module of the server.

[0176] S1508: The data processing module of the server requests display data of the video content from the database / third-party data interface of the server according to the source information and type information.

[0177] S1509: The data processing module of the server performs vectorization processing on the display data to obtain a display vector.

[0178] S1510: The user can initiate a question-and-answer input operation in the question-and-answer dialogue window to ask questions about the video content.

[0179] S1511: The application of the terminal device displays the first dialogue information in the question and answer dialogue window in response to the user's question and answer input operation.

[0180] S1512: The application program of the terminal device notifies the data intelligent analysis module to collect context information of the first conversation information.

[0181] S1513: The data intelligent analysis module of the terminal device generates a question and answer sequence according to the first dialogue information and context information.

[0182] S1514: The data intelligent analysis module of the terminal device extracts key frames from the video content according to the question content of the first dialogue information, wherein the key frames are video content related to the question content of the first dialogue information.

[0183] S1515: The data intelligent analysis module of the terminal device determines the timestamp information of the key frame.

[0184] S1516: The data intelligent analysis module of the terminal device sends the timestamp information and the question and answer sequence to the data processing module of the server.

[0185] S1517: The data processing module of the server integrates the display vector, the question-answer sequence and the timestamp information according to a preset format to obtain a coding vector.

[0186] S1518: The data processing module of the server inputs the coding vector into the AIGC model to obtain the answer information.

[0187] In some embodiments, the AIGC model of the server analyzes the coding vector and outputs answer information to the data processing module of the server.

[0188] S1518: The data processing module of the server sends a reply message to the application of the terminal device.

[0189] S1519: The application of the terminal device displays the answer information in the question-and-answer dialogue window.

[0190] based on Fig.15 As shown in the content, the embodiment of the present application can provide users with a new way of interaction when watching live broadcasts / videos. Based on AIGC technology, combined with the data intelligent analysis module on the terminal device side, it can realize the recognition of users' intention to ask questions about specific details in the video content and generate multi-modal contextual answer results, so that users can obtain the required information more conveniently and comprehensively.

[0191] Fig.16 A complete flow chart of a question-and-answer method is provided for this application. Fig.16 As shown, the method comprises the following steps:

[0192] S1601: The terminal device displays content in the display window.

[0193] S1602: The user initiates a dialogue operation in the display window.

[0194] S1603: The terminal device displays a question-and-answer dialogue window in response to the user's dialogue operation.

[0195] S1604: The terminal device obtains source information of the content displayed in the display window.

[0196] S1605: The terminal device classifies the source information and determines the type information corresponding to the displayed content.

[0197] S1606: The terminal device sends the source information and type information to the server. The server receives the source information and type information from the terminal device.

[0198] S1607: The server obtains display data corresponding to the display content according to the source information and type information.

[0199] In some embodiments, the server may determine whether a database of the server contains display data based on the type information.

[0200] When the database contains display data, the server may obtain the display data from the database based on the source information; or,

[0201] When the database does not contain the display data, the server may generate a data address of the display data based on the source information, and obtain the display data based on the data address.

[0202] S1608: The server performs vectorization processing on the display data to obtain a display vector.

[0203] S1609: The user may initiate a question-and-answer input operation in the question-and-answer dialog window, wherein S1609 and S1607-1608 are executed in no particular order.

[0204] S1610: The terminal device displays a first dialogue message in the question-and-answer dialogue window in response to the user's question-and-answer input operation. The first dialogue message is used to ask questions about the display content displayed in the display window. The first dialogue message is voice information or text information.

[0205] S1611: The terminal device may determine a first timestamp when the user inputs the first conversation information.

[0206] S1612: The terminal device determines the time period during which the first display content where the key frame is located is displayed in the display window according to the first timestamp, wherein the key frame is the display content related to the question content of the first dialogue information.

[0207] S1613: The terminal device extracts the first display content from the display content according to the time period.

[0208] S1614: The terminal device identifies the question content of the first dialogue information, and extracts key frames from the first display content according to the question content.

[0209] S1615: The terminal device uses the timestamp information corresponding to the key frame as the timestamp information corresponding to the first conversation information.

[0210] S1616: The terminal device obtains context information of the first conversation information.

[0211] S1617: The terminal device records the first dialogue information and the context information into an array in chronological order to obtain a question-answer sequence.

[0212] S1618: The terminal device sends the timestamp information and the question-answer sequence to the server. The server receives the timestamp information and the question-answer sequence from the terminal device.

[0213] S1619: The server extracts the display vector corresponding to the key frame from the display vector according to the timestamp information.

[0214] S1620: The server encodes the display vector and question-answer sequence corresponding to the key frame to obtain an encoding vector.

[0215] S1621: The server obtains a question and answer sample from a database of the server, encodes the question and answer sample, and obtains a question and answer vector.

[0216] S1622: The server inputs the question-answer vector and the encoding vector into the question-answer generation model, analyzes the encoding vector and the question-answer vector based on the question-answer generation model, and obtains answer information corresponding to the first dialogue information.

[0217] S1623: The server sends the answer information to the terminal device. The terminal device receives the answer information sent by the server.

[0218] S1624: The terminal device displays the answer information in the question-and-answer dialogue window.

[0219] S1625: The terminal device records the answer information into the question and answer sequence.

[0220] based on Fig.16 The user can ask questions about specific details such as objects / images / plots while watching the displayed content. The terminal device extracts key frames related to the question content and sends them to the server, so that the server can extract the key frames for question-related analysis and obtain accurate context-related answers, thereby improving the user's viewing experience and enriching the content of the terminal device.

[0221] Based on the above content and the same idea, Fig.17 A schematic diagram of the structure of a possible question-answering device 1700 provided in the present application, wherein the question-answering device 1700 is applied in a terminal device.

[0222] In some embodiments, the terminal device includes the question-answering device 1700, or the terminal device is the question-answering device 1700. The question-answering device 1700 can be used to implement the functions of the terminal device in the above method embodiment, and thus can also achieve the beneficial effects possessed by the above method embodiment.

[0223] Display screen 1701, used to display a user interface;

[0224] Communication unit 1702, used for receiving and sending data;

[0225] The processing unit 1703 is used to display a question-and-answer dialogue window and a display window on the display screen 1701; display content is displayed in the display window; in response to a user's question-and-answer input operation, a first dialogue message is displayed in the question-and-answer dialogue window; the first dialogue message is used to ask questions about the display content displayed in the display window; the timestamp information corresponding to the first dialogue message is determined; the timestamp information is used to represent the timestamp of a key frame in the display content related to the question content of the first dialogue message; the timestamp information and the first dialogue message are sent to a server; the answer information returned by the server is received, and the answer information is displayed in the question-and-answer dialogue window.

[0226] In an optional implementation manner, in response to the user's question and answer input operation, before the question and answer dialogue window displays the first dialogue information, the processing unit 1703 is further configured to:

[0227] When the display window is displayed, in response to the user's dialogue operation, displaying the question-and-answer dialogue window;

[0228] Obtaining source information of content displayed in the display window;

[0229] Classifying the source information to determine type information corresponding to the displayed content;

[0230] The source information and the type information are sent to the server.

[0231] An optional implementation manner is that the first dialogue information is a text message or a voice message.

[0232] In an optional implementation manner, the processing unit 1703 is specifically configured to:

[0233] Determine a first timestamp of the user inputting the first conversation information;

[0234] Determine, according to the first timestamp, a time period during which the first display content where the key frame is located is displayed in the display window;

[0235] extracting the first display content from the display content according to the time period;

[0236] identifying the question content of the first dialogue information, and extracting the key frame from the first display content according to the question content;

[0237] The timestamp information corresponding to the key frame is used as the timestamp information corresponding to the first dialogue information.

[0238] In an optional implementation manner, the processing unit 1703 is specifically configured to:

[0239] Acquire context dialogue information of the first dialogue information;

[0240] Recording the first dialogue information and the context information into an array in chronological order to obtain a question-answer sequence;

[0241] The timestamp information and the question-answer sequence are sent to the server.

[0242] In an optional implementation manner, the processing unit 1703 is further configured to:

[0243] The answer information is recorded in the question and answer sequence.

[0244] Based on the above content and the same idea, Fig.18 A schematic diagram of the structure of a possible question-answering device 1800 provided in the present application, wherein the question-answering device 1800 is applied in a server.

[0245] In some embodiments, the server includes the question-answering device 1800, or the server is the question-answering device 1800. The question-answering device 1800 can be used to implement the functions of the server in the above method embodiment, and thus can also achieve the beneficial effects possessed by the above method embodiment.

[0246] Communication unit 1801, used for receiving and sending data;

[0247] Processing unit 1802 is used to receive timestamp information and first dialogue information sent by a terminal device; the first dialogue information is used to ask questions about the display content displayed in the display window of the terminal device; the timestamp information is used to represent the timestamp of a key frame in the display content related to the question content of the first dialogue information; based on the timestamp information and the first dialogue information, determine the answer information corresponding to the first dialogue information; and send the answer information to the terminal device.

[0248] In an optional implementation manner, the processing unit 1802 is specifically configured to:

[0249] The timestamp information and the question-and-answer sequence sent by the terminal device are received; the question-and-answer sequence includes the first dialogue information and context dialogue information of the first dialogue information.

[0250] In an optional implementation manner, the processing unit 1802 is specifically configured to:

[0251] Encoding the key frame corresponding to the timestamp information and the question-answer sequence to obtain a coding vector;

[0252] The encoding vector is input into a question-answer generation model, and the encoding vector is analyzed based on the question-answer generation model to obtain the answer information.

[0253] In an optional implementation manner, the processing unit 1802 is specifically configured to:

[0254] Determining a display vector corresponding to the display content;

[0255] extracting a display vector corresponding to the key frame from the display vector according to the timestamp information;

[0256] The display vector corresponding to the key frame and the question-answer sequence are encoded to obtain the encoding vector.

[0257] In an optional implementation manner, the processing unit 1802 is specifically configured to:

[0258] Obtaining a question and answer sample from a database of the server, encoding the question and answer sample, and obtaining a question and answer vector;

[0259] Inputting the question-answer vector and the encoding vector into the question-answer generation model;

[0260] The encoding vector and the question-answer vector are analyzed based on the question-answer generation model to obtain answer information corresponding to the first dialogue information.

[0261] In an optional implementation manner, before receiving the timestamp information and the first conversation information sent by the terminal device, the processing unit 1802 is further configured to:

[0262] Receiving source information of the display content and type information corresponding to the display content from the terminal device;

[0263] Acquire display data corresponding to the display content according to the source information and the type information;

[0264] Vectorization processing is performed on the display data to obtain the display vector.

[0265] In an optional implementation manner, the processing unit 1802 determines whether the database of the server contains the display data according to the type information;

[0266] If the database contains the display data, acquiring the display data from the database based on the source information; or,

[0267] If the database does not contain the display data, a data address of the display data is generated based on the source information, and the display data is acquired based on the data address.

[0268] Based on the above content and the same concept, the present application provides an electronic device, including a display screen, a processor and a memory, the display screen is used to display a user interface, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the steps performed by the terminal device in the above method embodiment.

[0269] Based on the above content and the same concept, the present application provides an electronic device, including a processor and a memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the steps executed by the server in the above method embodiment.

[0270] Based on the above content and the same concept, the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed by a computing device, the computing device executes the steps performed by the terminal device or server in the above method embodiment.

[0271] Based on the above content and the same concept, the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a computing device, the computing device executes the steps executed by the terminal device or server in the above method embodiment.

[0272] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0273] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0274] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0275] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0276] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A question-answering method, characterized in that: Applied to a terminal device, the method comprises: Displaying a question-and-answer dialogue window and a display window; wherein display content is displayed in the display window; In response to a question-and-answer input operation by a user, displaying first dialogue information in the question-and-answer dialogue window; the first dialogue information is used to ask questions about the display content displayed in the display window; Determine timestamp information corresponding to the first dialogue information; the timestamp information is used to represent the timestamp of a key frame in the displayed content that is related to the question content of the first dialogue information; Sending the timestamp information and the first conversation information to a server; Receive the answer information returned by the server, and display the answer information in the question-and-answer dialogue window.

2. The method according to claim 1, characterized in that In response to the user's question-and-answer input operation, before the question-and-answer dialogue window displays the first dialogue information, the method further includes: When the display window is displayed, in response to the user's dialogue operation, displaying the question-and-answer dialogue window; Obtaining source information of content displayed in the display window; Classifying the source information to determine type information corresponding to the displayed content; The source information and the type information are sent to the server.

3. The method according to claim 1 or 2, characterized in that: The first dialogue information is text information or voice information.

4. The method according to any one of claims 1 to 3, characterized in that: The determining the timestamp information corresponding to the first conversation information includes: Determine a first timestamp of the user inputting the first conversation information; Determine, according to the first timestamp, a time period during which the first display content where the key frame is located is displayed in the display window; extracting the first display content from the display content according to the time period; identifying the question content of the first dialogue information, and extracting the key frame from the first display content according to the question content; The timestamp information corresponding to the key frame is used as the timestamp information corresponding to the first dialogue information.

5. The method according to any one of claims 1 to 4, characterized in that: The sending the timestamp information and the first conversation information to the server includes: Acquire context dialogue information of the first dialogue information; Recording the first dialogue information and the context information into an array in chronological order to obtain a question-answer sequence; The timestamp information and the question-answer sequence are sent to the server.

6. The method according to claim 5, characterized in that The method further comprises: The answer information is recorded in the question and answer sequence.

7. A question-answering method, characterized in that: Applied to a server, the method comprises: Receiving timestamp information and first dialogue information sent by a terminal device; the first dialogue information is used to ask questions about display content displayed in a display window of the terminal device; the timestamp information is used to represent the timestamp of a key frame in the display content related to the question content of the first dialogue information; determining answer information corresponding to the first dialogue information according to the timestamp information and the first dialogue information; The answer information is sent to the terminal device.

8. The method according to claim 7, characterized in that The receiving the timestamp information and the first conversation information sent by the terminal device includes: The timestamp information and the question-and-answer sequence sent by the terminal device are received; the question-and-answer sequence includes the first dialogue information and context dialogue information of the first dialogue information.

9. The method according to claim 7 or 8, characterized in that: The determining, according to the timestamp information and the first dialogue information, answer information corresponding to the first dialogue information includes: Encoding the key frame corresponding to the timestamp information and the question-answer sequence to obtain a coding vector; The encoding vector is input into a question-answer generation model, and the encoding vector is analyzed based on the question-answer generation model to obtain the answer information.

10. The method according to claim 9, characterized in that The step of encoding the key frame corresponding to the timestamp information and the question-answer sequence to obtain a coding vector includes: Determining a display vector corresponding to the display content; extracting a display vector corresponding to the key frame from the display vector according to the timestamp information; The display vector corresponding to the key frame and the question-answer sequence are encoded to obtain the encoding vector.

11. The method according to claim 10, characterized in that The step of inputting the encoding vector into a question-answer generation model and analyzing the encoding vector based on the question-answer generation model to obtain the answer information includes: Obtaining a question and answer sample from a database of the server, encoding the question and answer sample, and obtaining a question and answer vector; Inputting the question-answer vector and the encoding vector into the question-answer generation model; The encoding vector and the question-answer vector are analyzed based on the question-answer generation model to obtain answer information corresponding to the first dialogue information.

12. The method according to any one of claims 7 to 11, characterized in that: Before receiving the timestamp information and the first conversation information sent by the terminal device, the method further includes: Receiving source information of the display content and type information corresponding to the display content from the terminal device; Acquire display data corresponding to the display content according to the source information and the type information; Vectorization processing is performed on the display data to obtain the display vector.

13. The method according to claim 12, characterized in that The acquiring, according to the source information and the type information, display data corresponding to the display content includes: determining, according to the type information, whether the database of the server contains the display data; If the database contains the display data, acquiring the display data from the database based on the source information; or, If the database does not contain the display data, a data address of the display data is generated based on the source information, and the display data is acquired based on the data address.

14. A question-answering device, applied to a terminal device, characterized in that: The device comprises: A display screen, used for displaying a user interface; A communication unit for receiving and sending data; A processing unit, used to display a question-and-answer dialogue window and a display window on the display screen; display content is displayed in the display window; in response to a user's question-and-answer input operation, a first dialogue message is displayed in the question-and-answer dialogue window; the first dialogue message is used to ask questions about the display content displayed in the display window; determine the timestamp information corresponding to the first dialogue message; the timestamp information is used to represent the timestamp of a key frame in the display content related to the question content of the first dialogue message; send the timestamp information and the first dialogue message to a server; receive the answer information returned by the server, and display the answer information in the question-and-answer dialogue window.

15. A question-answering device, characterized in that: Applied to a server, the device comprises: A communication unit for receiving and sending data; A processing unit is used to receive timestamp information and first dialogue information sent by a terminal device; the first dialogue information is used to ask questions about display content displayed in a display window of the terminal device; the timestamp information is used to represent the timestamp of a key frame in the display content related to the question content of the first dialogue information; based on the timestamp information and the first dialogue information, determine answer information corresponding to the first dialogue information; and send the answer information to the terminal device.

16. An electronic device, characterized in that: include: Display screen; one or more processors; one or more memories; The display screen is used to display a user interface; The one or more memories are used to store one or more computer programs and data information; wherein the one or more computer programs include instructions; When the instructions are executed by the one or more processors, the electronic device executes the method according to any one of claims 1 to 6.

17. An electronic device, characterized in that: include: one or more processors; one or more memories; The one or more memories are used to store one or more computer programs and data information; wherein the one or more computer programs include instructions; When the instructions are executed by the one or more processors, the electronic device executes the method as claimed in any one of claims 7 to 13.

18. A chip, characterized in that: The chip is coupled to a memory in an electronic device so that the chip calls a computer program stored in the memory when running to implement the method according to any one of claims 1 to 13.

19. A computer storage medium, characterized in that: The computer-readable storage medium comprises a computer program, and when the computer program is executed on the electronic device, the electronic device causes the method according to any one of claims 1 to 13 to be performed.

20. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 13.

Images