Query method and device, equipment, storage medium and program product

Abstract

The invention provides a query method and device, equipment, a storage medium and a program product. In one method, first media data is acquired by a first acquisition device of a terminal device. And in response to determining that second media data needs to be acquired based on the first media data and the query request, acquiring the second media data by a second acquisition device of the terminal device. And providing a response to the query request based on the first media data and the second media data. Therefore, rich query modes can be supported, the input efficiency of the user is improved, and the user is supported to express own demands in a more accurate mode; and on the other hand, more accurate responses can be provided based on more accurate query requirements.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, apparatuses, electronic devices, computer-readable storage media, and computer program products for querying. Background Art

[0002] With the development of information technology, various terminal devices can provide various services to people in work and life. For example, applications that provide services can be deployed in terminal devices. Terminal devices or applications can provide users with digital assistant functions to assist users in using terminal devices or applications. Users can query the desired information through various interactions with digital assistants. Summary of the invention

[0003] In a first aspect of the present disclosure, a query method is provided. The method includes: a first acquisition device of a terminal device acquires first media data. In response to determining that second media data needs to be acquired based on the first media data and a query request, a second acquisition device of the terminal device acquires the second media data. Based on the first media data and the second media data, a response to the query request is provided.

[0004] In a second aspect of the present disclosure, a device for querying is provided. The device comprises: a first acquisition module configured to acquire first media data by a first acquisition device of a terminal device; a second acquisition module configured to acquire second media data by a second acquisition device of the terminal device in response to determining that the second media data needs to be acquired based on the first media data and the query request; and a providing module configured to provide a response to the query request based on the first media data and the second media data.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the electronic device executes the method according to the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the method according to the first aspect of the present disclosure.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer executable instructions, which, when executed by a device, cause the device to perform the method of the first aspect.

[0008] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1A A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 1B A schematic diagram showing a process for querying according to some embodiments of the present disclosure is shown;

[0012] Figure 1C A schematic diagram for collecting media data according to some embodiments of the present disclosure is shown;

[0013] Figure 1D A schematic diagram for collecting media data from an application running on a terminal device according to some embodiments of the present disclosure is shown;

[0014] FIG. 2A to FIG. 2H shows an example interface according to some embodiments of the present disclosure;

[0015] Figure 3 Examples of some embodiments according to the present disclosure are shown;

[0016] Figure 4 A flowchart of a method for querying according to some embodiments of the present disclosure is shown;

[0017] Figure 5 An exemplary structural block diagram of an apparatus for querying according to some embodiments of the present disclosure is shown; and

[0018] Figure 6 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0021] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0022] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0023] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0024] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0025] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0026] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0027] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0028] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0029] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. The testing stage can sometimes be integrated into the training stage. In the application or inference stage, the trained model can be used to process the actual model input based on the parameter values ​​obtained through training to determine the corresponding model output.

[0030] Figure 1A A schematic diagram of an example environment 100A in which embodiments of the present disclosure can be implemented is shown. In the example environment 100A, an application 112 and a digital assistant 114 are installed in a terminal device 110. A user 140 can interact with the application 112 via the terminal device 110 and / or an attached device of the terminal device 110. In some implementations, the application 112 can be authorized to collect voice via a microphone of the terminal device 110, collect images via a camera of the terminal device 110, and so on.

[0031] In some embodiments, the application 112 and the digital assistant 114 can be downloaded and installed on the terminal device 110. In some embodiments, the application 112 and the digital assistant 114 can also be accessed through other means, such as through a web page. Although a single application 112 and a single digital assistant 114 are shown respectively, the digital assistant 114 can be integrated into the application 112. Alternatively and / or additionally, the digital assistant 114 can be called by multiple applications.

[0032] In the embodiments of the present disclosure, the application 112 may be any appropriate application with a response function, which may include but is not limited to one or more of the following: a chat application component (also known as an instant messaging application component), a browser application component, a planning application component, a document application component, an audio and video conference application component, an email application component, a task application component, a calendar application component, an objectives and key results (OKR) application component, etc. It can be understood that although Figure 1A A single application is shown in the figure, but in fact, multiple applications can be installed on the terminal device 110. In some embodiments, the application 112 may include a multi-functional collaboration platform, such as an office collaboration platform (also called an office suite) that can provide integration of various types of business components to facilitate people to carry out office work, communication and other activities. In the multi-functional collaboration platform, people can start different business components as needed to complete corresponding information processing, sharing, communication, etc.

[0033] In some embodiments, the digital assistant 114 may be provided by a separate application or may be integrated into an application 112 that can provide a content entity. The application service component for providing a client interface of the digital assistant may correspond to a single-function application service component or a multi-function collaboration platform, such as an office suite or other collaboration platform that can integrate multiple components. It is understood that similar to the application, although Figure 1A A single digital assistant is shown in the figure, but there may actually be multiple digital assistants.

[0034] In some embodiments, the digital assistant 114 supports the use of plug-ins. Each plug-in can provide one or more functions of the application. Such plug-ins include, but are not limited to, one or more of the following: a search plug-in, a contact plug-in, a message plug-in, a document plug-in, a table plug-in, a mail plug-in, a calendar plug-in, a schedule plug-in, a task plug-in, and the like.

[0035] The digital assistant 114 is an intelligent assistant for the user, which has intelligent dialogue and information processing capabilities. In an embodiment of the present disclosure, the digital assistant 114 is used to interact with the user 140 to assist the user 140 in using the terminal device 110 or the application 112. In some embodiments, multiple interaction modes between the user 140 and the digital assistant 114 can be provided, and flexible switching between multiple interaction modes can be achieved. When a certain interaction mode is triggered, the corresponding interaction area is presented to facilitate the interaction between the user 140 and the digital assistant 114. The interaction mode between the user 140 and the digital assistant 114 is different in different interaction modes, so that it can be flexibly adapted to the interaction needs in different application scenarios. Alternatively and / or additionally, the digital assistant can be installed in the application as a plug-in and called by the application.

[0036] In the environment 100A, in response to the application 112 being started, the terminal device 110 may present an interface 150 of the application 112 and / or the digital assistant 114. The interface 150 may include, for example, an interactive interface of the application 112 and the digital assistant 114. In some embodiments, an interactive window between the user 140 and the digital assistant 114 may be presented in the interface 150. In the interactive window, the user 140 can communicate with the digital assistant 114 by inputting natural language, pictures, audio files, video files, web page files, etc., to instruct the digital assistant to assist in completing various tasks.

[0037] The interactive window between the digital assistant 114 and the user 140 may include a conversation window, such as a conversation window in an instant messaging application or an instant messaging module of a specific application. In the conversation window, the interaction between the digital assistant 114 and the user 140 may be presented in the form of a conversation message. Alternatively or additionally, the interactive window between the digital assistant 114 and the user 140 may also include other types of windows, such as a window in a floating window mode, in which the user 140 may trigger the digital assistant 114 to perform a corresponding operation by inputting instructions, selecting shortcut instructions, etc.

[0038] In some embodiments, the digital assistant 114 may support an interactive mode of a conversation window, also referred to as a conversation mode. In this interactive mode, a conversation window between the user 140 and the digital assistant 114 is presented, in which the user 140 and the digital assistant 114 interact through conversation messages. In conversation mode, the digital assistant 114 may perform tasks based on conversation messages in the conversation window. In the interactive window, the user 140 inputs an interactive message, and the digital assistant 114 provides a reply message in response to the user input. By selecting the digital assistant 114, a conversation window with the digital assistant 114 may be opened. The conversation window may include interface elements for information interaction, such as an input box, a message list, a message bubble, and the like.

[0039] In some embodiments, a communication connection is established between the terminal device 110 and the server 120. The communication connection can be established in a wired manner or a wireless manner. The communication connection may include, but is not limited to, a Bluetooth connection, a mobile network connection, a Universal Serial Bus (USB) connection, a Wireless Fidelity (WiFi) connection, etc., and the embodiments of the present disclosure are not limited in this respect. In an embodiment of the present disclosure, the terminal device 110 and the server 120 can implement signaling interaction through the communication connection between the two to implement the supply of services for the application 112 and / or the digital assistant 114.

[0040] like Figure 1A As shown, the server 120 can call the machine learning model 130 to support the answering function of the application 112 based on the output of the machine learning model 130. The machine learning model 130 can be based on any appropriate model structure, including but not limited to a Transformer model, a convolutional neural network (CNN), a recurrent neural network (RNN), a deep neural network (DNN), and the like. In some embodiments, the machine learning model 130 can be based on a language model (LM). The language model can have question-answering capabilities by learning from a large amount of corpus. The machine learning model 130 can also be based on other appropriate models.

[0041] The machine learning model 130 can be deployed on the server 120, or on other devices, for example, on the terminal device 110. The machine learning model 130 may include one or more machine learning models. It should be noted that if the machine learning model 130 includes multiple machine learning models, these multiple machine learning models may have the same or different structures, uses, and functions, and the present disclosure does not limit this. It should be noted that the terminal device 110 may also be locally deployed with a machine learning model (not shown in the figure), and the terminal device 110 may also directly call the local machine learning model to support the response function of the application 112 based on the output of the local machine learning model.

[0042] The terminal device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the terminal device 110 can also support any type of interface for the user (such as "wearable" circuits, etc.).

[0043] The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The server 120 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0044] It should be understood that the structure and function of each element in the environment 100A are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0045] As mentioned above, users can complete a variety of operations through various interactions with digital assistants. Users can provide at least one modal user input to the digital assistant, and the digital assistant can then determine the corresponding response based on the received user input. Traditionally, in the multimodal interaction process between the user and the digital assistant, the user can send a query request and media data that has been pre-stored locally to the digital assistant. The digital assistant can then provide a response to the query request based on the media data. The user needs to first obtain the media data, and then provide the pre-acquired media data to the digital assistant, and the real-time performance of the media data is poor. In addition, the digital assistant cannot obtain the media data collected by the user's client device in real time, which may affect the efficiency of the query and response.

[0046] In view of this, according to an embodiment of the present disclosure, an improved query solution is provided. Figure 1B Describe in more detail the Figure 1B A schematic diagram 100B is shown of a process for querying according to some embodiments of the present disclosure. Figure 1BAs shown, according to the solution of the embodiment of the present disclosure, the first acquisition device 152 of the terminal device acquires the first media data 154. It can be determined whether the second media data 158 needs to be acquired based on the first media data 154 and the query request 142. In response to determining that the second media data 158 needs to be acquired based on the first media data 154 and the query request 142, the second acquisition device 156 of the terminal device acquires the second media data 158. Based on the first media data 154 and the second media data 158, a response 132 to the query request 142 is provided.

[0047] Here, the terminal device may include multiple acquisition devices. For example, the first acquisition device 152 may include an image acquisition device, a sound acquisition device, etc. for acquiring data from the outside of the terminal device; the second acquisition device 156 may include an image acquisition device, a sound acquisition device, a text acquisition device, etc. for acquiring data from the inside of the terminal device. In this way, in the multimodal query process, media data and query requests can be acquired in real time, which can not only support a rich query method but also improve the user's input efficiency. Furthermore, it can support users to express their needs in a more accurate way, and can support more accurate responses based on more accurate query needs. In this way, it can support the acquisition of different media data through different acquisition devices, thereby supporting user queries in a more comprehensive way.

[0048] In some embodiments, the first media data comes from an external environment of the terminal device, and the second media data comes from an application running in the terminal device. Figure 1C Describe in more detail the Figure 1C A schematic diagram 100C for collecting media data according to some embodiments of the present disclosure is shown. Figure 1C As shown, the terminal device 110 may be located in the physical environment 122. For example, the user may hold the terminal device 110 and collect video data, image data, and sound data of the physical environment 122. Alternatively and / or additionally, text data may be extracted from the collected data, for example, text may be recognized from video, image, and sound data, etc. At this time, the first media data 154 may have different modalities, such as video, image, sound, text, and the like.

[0049] Alternatively and / or additionally, one or more applications may be run in the terminal device 110, for example, application 112, etc. The second media data 158 may be collected from at least one application. Here, the first media data and the second media data may have the same modality. Alternatively and / or additionally, the first media data and the second media data may have different modalities. In this way, multimodal data processing may be supported in a richer manner. For example, the first media data may be a video of the physical environment 122 captured by a camera. Assuming the query request is "where is here", it can be determined that the second media data does not need to be collected. For another example, assuming the query request is "Is this the museum in the travel note I just read", the second media data may be collected from the application. Specifically, relevant video, text, image and other data may be collected from the travel application previously used by the user, and then it may be determined whether the content in the first media data is consistent with the museum in the travel note, and then a response may be provided.

[0050] In some embodiments, the first media data may be continuously collected, for example, the user may move within the physical environment 122 and capture more videos of the physical environment 122 in real time. In this way, richer media data may be provided for the query and the accuracy of the query may be improved.

[0051] In some embodiments, in the process of providing a response to a query request, a prompt word for processing the query request can be generated based on the first media data, the second media data and the query request; and then a response is provided based on the answer to the prompt word from the machine learning model. In the above example, the name of the museum "XXX" can be determined from the second media data. At this time, an example of the prompt word can be expressed as, for example: "Please determine whether the building in the first media data is the XXX museum." Alternatively and / or additionally, an image (or video) of the museum can be extracted from the second media data. At this time, an example of the prompt word can be expressed as, for example: "Please determine whether the building in the first media data is consistent with the museum in the image (or video)", etc. Alternatively and / or additionally, a description of the museum can be extracted from the second media data, for example, "five floors, marble exterior walls,..." At this time, an example of the prompt word can be expressed as, for example: "Please determine whether the building in the first media data is consistent with the following description", etc.

[0052] It should be understood that although the query process is described above using video data as an example of the first media data, alternatively and / or additionally, the first media data may include audio data. For example, the sound collection device in the terminal device may record the song played in the physical environment 122. If the query request is "Is this the singer introduced in the news?", the name of the singer may be found in the news application in the terminal device, and further processing may be performed. For another example, if the query request is "Have I downloaded this song?", the song that has been downloaded may be found in the music application in the terminal device, and further processing may be performed.

[0053] In some embodiments, the method of the present disclosure is executed in a first application running in a terminal device, and obtaining the second media data by a second acquisition device of the terminal device includes: determining an application from at least one second application running at a client device; and acquiring the second media data from the application. Specifically, one or more applications may be running in the terminal device, and the data source of the second media data 158 may be determined in a variety of ways. For example, the method of the present disclosure may be executed in a digital assistant 114 (e.g., also referred to as a first application), and the second media data may be acquired from an application 112 (e.g., also referred to as a second application) different from the digital assistant 114.

[0054] See also Figure 1D Describe in more detail the Figure 1D A schematic diagram 100D is shown for collecting media data from an application running at a terminal device according to some embodiments of the present disclosure. Figure 1D As shown, multiple applications may be run in the terminal device 110, for example, applications 112, 116, 118, etc. An application matching the first media data and the query request may be selected from the multiple applications based on multiple methods, and then the second media data may be collected from the selected application.

[0055] In some embodiments, in the process of determining an application from at least one second application running at the client device, at least one application identifier of at least one second application may be provided; and in response to receiving a selection request for an application identifier in the at least one application identifier, the application may be determined based on the selection request. The application identifier of the running application may be displayed at the terminal device unit 110. Figure 1DAs shown, multiple application identifiers can be displayed in different styles. For example, multiple applications can be displayed in a stacked manner, and the user can use a sliding operation to view each application one by one in order to select the desired application. Alternatively and / or additionally, multiple application identifiers can be tiled in a grid manner in a multi-row and multi-column manner so that the user can select the desired application. The first media data is a video of a physical environment, and the query request is "Is this the museum I just saw?" Assuming that the user has browsed museums in different applications (queried the introduction of Museum A in a travel application, queried the address of Museum B in a map application, and so on), multiple application identifiers can be provided at this time so that the user can select the desired application. In this way, the user's intention can be understood in a more accurate manner, thereby improving the accuracy of the query.

[0056] It should be understood that the second application here can be an application started at the terminal device (for example, being used in the foreground or running in the background), alternatively and / or additionally, the second application can be an application installed at the terminal device. The most recently used application can be presented preferentially according to the time sequence in which the applications are used.

[0057] In some embodiments, in the process of determining an application from at least one second application running at a client device, a machine learning model can be used to select a desired application from multiple second applications. Specifically, a prompt word for determining an application can be generated based on the first media data and the query request; and the application can be determined based on the answer to the prompt word from the machine learning model. For example, examples of prompt words may include: Please select an application that matches the first media data and the query request from the following multiple applications, please select the application specified by the query request, and so on. In this way, the powerful processing power of the machine learning model can be used to automatically determine an application that meets the user's needs.

[0058] In some embodiments, in the process of determining an application from at least one second application running on a client device, the relevance between at least one of the first media data and the query request and the at least one second application may be determined; and based on the relevance, the application may be determined from the at least one second application. Specifically, content information may be extracted from the first media data, and the relevance of each second application to the content information may be determined respectively. For example, assuming that the content of the first media data is an image, the second media data may be preferentially collected from an image application; assuming that the content of the first media data is a song, the second media data may be preferentially collected from a music application.

[0059] Alternatively and / or additionally, content information can be extracted from the query request, and the relevance of each second application to the content information can be determined separately. For example, assuming that the query request specifies "compare whether the first media data is consistent with the museum in the travel note", the second media data can be preferentially collected from the travel application, and so on. Alternatively and / or additionally, an application with a higher relevance can be selected from multiple second applications based on both the first media data and the query request. In this way, the user's intention can be understood in a more accurate manner, thereby improving the accuracy of processing the query request.

[0060] It should be understood that although the above describes an example of selecting an application from multiple second applications and collecting second media data from the selected application, alternatively and / or additionally, multiple applications can be selected and the second media data can be collected from the multiple applications respectively. Assume that the query request is: "Is this a museum that has been searched for?" Assume that the museum has been searched for in the map application and the travel application respectively, the second media data can be collected from the two applications respectively.

[0061] The process for querying requests has been described, and some example embodiments of the present disclosure will be described below with reference to the accompanying drawings. The query method involved in the present disclosure can be implemented in the terminal device 110, and the processing capabilities of the server 120 are called. For the convenience of description, the query method is implemented at the terminal device 110 as an example for explanation. It should be noted that the operations performed by the terminal device 110 can specifically be performed by related applications and / or digital assistants installed on the terminal device 110. It should be noted that if the query method is implemented at the server 120 as an example, the server 120 needs to receive and send data from the terminal device 110 via the communication connection between the terminal device 110. Similarly, it can be understood that some operations described with reference to the terminal device 110 may require the assistance of the server 120 to complete.

[0062] FIG. 2A to FIG. 2H Example interfaces 200A to 200H (also referred to as examples 200A to 200H) according to some embodiments of the present disclosure are shown. The interfaces shown in examples 200A to 200H may be presented at the terminal device 110. For ease of discussion, reference will be made to Figure 1A Examples 200A to 200H are described with reference to environment 100A. It should be understood that the interfaces shown in the drawings are merely examples, and various interface designs may actually exist. The various graphical elements in the interface may have different arrangements and different visual representations, one or more of which may be omitted or replaced, and one or more other elements may also exist. The embodiments of the present disclosure are not limited in this respect.

[0063] A digital assistant (e.g., digital assistant 114) may be running in the terminal device 110. The method of the present disclosure may be performed by the terminal device 110. Specifically, in the digital assistant, in response to receiving an access request for a media control, media data corresponding to the access request is obtained. The media data may include image data and / or video data. The terminal device 110 may also receive a query request from a user.

[0064] For example, the terminal device 110 can provide an interactive interface between the user (eg, user 140) and the digital assistant during the interaction between the user and the digital assistant. Figure 2A , Example 200A shows an example of an interactive interface. Example 200A presents an identifier of the digital assistant currently interacting with the user, which includes the name of the digital assistant (e.g., "digital assistant A" in the figure), the avatar of the digital assistant, etc. Example 200A may also present multiple media controls in the digital assistant. Multiple media controls may include a cancel control 204. The terminal device 110 may end the user's interaction with the digital assistant in response to receiving a trigger for the cancel control 204.

[0065] The plurality of media controls may further include a video acquisition control 203. The terminal device 110 may, for example, determine that an access request for the video acquisition control 203 is received in response to receiving a trigger for the video acquisition control 203. The terminal device 110 may, in response to receiving an access request for the video acquisition control 203, receive media data from a video acquisition unit (e.g., a camera) of the terminal device 110. The terminal device 110 may, for example, start the camera and switch to presenting the interactive interface shown in the example 200B in response to receiving an access request for the video acquisition control 203 in the example 200A.

[0066] The interactive interface shown in Example 200B presents media data that the terminal device 110 can collect via a camera. When the media data is not received, the terminal device 110 can present the video acquisition control 203 in a first visual style, and in the process of receiving the media data via the camera, the terminal device 110 can present the video acquisition control 203 in a different second visual style. The present disclosure does not limit the specific visual style. As an example only, the terminal device 110 can present the video acquisition control 203 in the visual style of a regular border in Example 200A, and present the video acquisition control 203 in the visual style of a bold border in Example 200B. Alternatively and / or additionally, the enabled camera can be represented in other highlighting ways. Thus, the terminal device 110 can capture the video / image of the environment in which the terminal device 110 is located via the camera, which is convenient for obtaining information that is not easy to describe in words, and helps to improve the accuracy of the answer.

[0067] In some embodiments, when multiple cameras are included, the interactive interface shown in example 200B may further include a lens conversion control 206. The terminal device 110 may switch the camera currently being used in response to receiving a trigger on the lens conversion control 206. As an example, if the terminal device 110 includes a front camera and a rear camera, and the rear camera is currently being used (that is, the media data currently presented is the media data obtained by means of the rear camera), the terminal device 110 may switch to using the front camera in response to receiving a trigger on the lens conversion control 206, and present the media data obtained via the front camera.

[0068] Alternatively and / or additionally, the multiple media controls in example 200A may include an audio control 201. For example, the terminal device 110 may determine that an interaction request for the audio control 201 is received in response to receiving a trigger for the audio control 201. The terminal device 110 may receive audio data via an audio acquisition unit (e.g., a microphone) of the terminal device in response to receiving an interaction request for the audio control 201. In some embodiments, during the interaction between the user and the digital assistant, the audio control 201 may be in a triggered state by default. That is, the terminal device 110 may maintain an audio receiving state during the interaction between the user and the digital assistant, and always receive audio data via a microphone. In the case of being in an audio receiving state (i.e., during the reception of audio data), the terminal device 110 may exit the audio receiving state in response to receiving a trigger for the audio control 201, and no longer receive audio data. In some embodiments, the terminal device 110 may directly determine the audio data as a query request. For example, the received audio data may be converted to text data so as to be used as a query request.

[0069] In some embodiments, during the period of receiving audio data via the microphone, the terminal device 110 may also present an audio reception indication. The audio reception indication may prompt the user that the terminal device 110 is currently receiving audio data. Figure 2C During the reception of audio data, the terminal device 110 may present an audio reception indication 240 in the example 200C. The audio reception indication 240 may indicate that audio data is currently being received.

[0070] Alternatively and / or additionally, the multiple media controls in example 200A may further include a screen sharing control 202. For example, the terminal device 110 may determine that an access request for the screen sharing control 202 is received in response to receiving a trigger for the screen sharing control 202. In some embodiments, the terminal device 110 may present the interactive interface shown in example 200D in response to receiving an access request for the screen sharing control 202 in example 200A. Example 200D includes a confirmation window 250, which is used to prompt the user to confirm whether to share the screen (for example, the promotion text "Sharing the screen with digital assistant A After sharing the screen... answering questions, etc." shown in the figure), and includes a cancel control 252 and a confirmation control 254. The terminal device 110 may return to presenting the interactive interface shown in example 200A in response to receiving a trigger for the cancel control 252, and at this time, the receiving of media data based on the screen display interface of the terminal device 110 is prohibited.

[0071] The terminal device 110 may determine to receive media data based on the screen display interface of the terminal device 110 in response to receiving a trigger on the confirmation control 254. For example, the terminal device 110 may switch to the interactive interface shown in the presentation example 200F in response to receiving a trigger on the confirmation control 254. The terminal device 110 may also switch to the interactive interface shown in the presentation example 200E in response to receiving a trigger on the confirmation control 254. Example 200E includes a permission acquisition window 260. The permission acquisition window 260 may include a prompt text (e.g., "recruitment screen") for prompting the user to authorize the digital assistant, prompt information of the current digital assistant (e.g., "digital assistant A" in the figure), and a confirmation control 262. The terminal device 110 may determine that the digital assistant A has obtained the permission for the admission screen in response to receiving a trigger on the confirmation control 262. The terminal device 110 may then switch to the interactive interface shown in the presentation example 200F after the digital assistant A obtains the permission.

[0072] In some embodiments, the permission acquisition window 260 may also include a cancel control (not shown in the figure). The terminal device 110 may determine, in response to the received trigger of the cancel control, that the digital assistant A has not obtained permission to access the recording screen and cannot receive media data based on the screen display interface of the terminal device 110. The terminal device 110 may, for example, return to present example 200A.

[0073] Although the above describes the collection of media data from an application running in the foreground, alternatively and / or additionally, media data can be collected from other applications running at the terminal device. For example, media data can be automatically collected from one or more applications running in the terminal device. Specifically, during the startup of the digital assistant, a confirmation window 250 can be automatically presented to allow the collection of media data from one or more applications running in the terminal device. Further, a list of installed applications can be provided, and the user can select one or more applications from the list that allow the collection of media data; alternatively and / or additionally, the user can specify that media data is only allowed to be collected from applications running in the foreground, can specify that media data is allowed to be collected from applications running in the background, and so on. In this way, while improving data security, users can be supported to express their needs in a more accurate manner.

[0074] In some embodiments, the Figure 2D and Figure 2E Specifically, the terminal device 110 may respond to receiving the interactive interface for Figure 2A The terminal device 110 receives the media data based on the screen display interface of the terminal device 110 in response to the access request to the screen sharing control 202 received in example 200A. For example, the terminal device 110 can directly switch to presenting example 200F in response to receiving the access request to the screen sharing control 202 in example 200A. Example 200F shows an example of the screen display interface of the terminal device 110. The content presented in the screen display interface shown in example 200F can be, for example, the content of another application other than the digital assistant. Switching the screen display interface from example 200A to example 200F is called switching from the interactive interface of the digital assistant to the interface of another application different from the digital assistant.

[0075] Example 200F may include prompt text (e.g., the text “The screen includes… cancel live screen recording” shown in the figure), which may be used to prompt the user to cancel the authorization of the digital assistant A by triggering other areas of the permission acquisition window 260 in example 200F. That is, the terminal device 110 may determine that the digital assistant A has not obtained the permission to record the screen in response to receiving the trigger of other areas of the permission acquisition window 260 in example 200F, and cannot receive media data based on the screen display interface of the terminal device 110.

[0076] In some embodiments, when the screen display interface is an interface of another application different from the digital assistant (for example, in the case of presenting example 200F), the terminal device 110 may overlay a screen sharing representation on the screen display interface, and the screen sharing representation may, for example, indicate that the screen display interface is shared with the digital assistant. Figure 2F, example 200F may include a screen sharing representation 274, and the screen sharing representation 272 may inform the user that the interface is currently being shared with the digital assistant. In some embodiments, example 200F may also include element 272, on which the identification of the application currently active in the terminal device 110 may be presented. As an example, since the screen display interface is currently being shared with digital assistant A, digital assistant A is active, and the identification of digital assistant A may be presented in element 272. Thus, the user can provide media data to the digital assistant by sharing the screen with the digital assistant, which enriches the way of providing media data, can improve the richness of the information provided to the digital assistant, and help improve the accuracy of the answer. The user can know that the digital assistant is currently receiving media data via the shared representation of the superimposed display, thereby avoiding providing erroneous screen data to the digital assistant.

[0077] In some embodiments, during the acquisition of media data, the terminal device 110 may present the media data being acquired on the screen display interface. In this case, the terminal device 110 may also overlay a thumbnail representation of the digital assistant on the screen display interface. Figure 2B , Figure 2C and Figure 2F , Example 200B, Example 200C, and Example 200F may also include a thumbnail representation 230. Thumbnail representation 230 may present an identifier of the digital assistant being interacted with (e.g., the avatar of digital assistant A). On the one hand, thumbnail representation 230 may prompt the user that the user is currently in the process of interacting with the digital assistant, and that the digital assistant is acquiring media data. On the other hand, terminal device 110 may also determine that an interaction request for thumbnail representation 230 has been received in response to receiving a trigger for thumbnail representation 230.

[0078] The terminal device 110 may switch the screen display interface of the terminal device 110 to the digital assistant in response to receiving an interaction request for the thumbnail representation 230. That is, the terminal device 110 may switch back to presenting the interaction interface with the digital assistant in response to receiving an interaction request for the thumbnail representation 230. For example, the terminal device 110 may determine to stop receiving media data via the screen display interface of the terminal device 110 (that is, the digital assistant A no longer has sharing authority) in response to receiving an interaction request for the thumbnail representation 230 in example 200F. In this case, the terminal device 110 may switch back to presenting example 200A.

[0079] For example, the terminal device 110 may also return to presenting the interactive interface with the digital assistant in response to receiving an interaction request for the thumbnail representation 230, but maintain the screen sharing permission of the digital assistant A (that is, the digital assistant A can still receive media data via the screen display interface). In this case, the terminal device 110 can switch back to presenting example 200G. After the terminal device 110 switches to presenting example 200G, the reception of the media data shown in example 200F can be suspended, and the terminal device 110 can continue to obtain media data in response to switching back to presenting another application. The terminal device 110 no longer needs user confirmation at this time, that is, there is no need to present example 200D and example 200E.

[0080] It should be noted that, in some embodiments, the terminal device 110 can obtain media data via the camera and the screen display interface respectively. Alternatively and / or additionally, the terminal device 110 can also obtain media data via the camera and the screen display interface in parallel. In this case, the two channels of media data obtained by the two methods can be combined and provided to the digital assistant. For example, a video frame of screen recording can be provided in the gap between the video frames of the video captured by the camera.

[0081] In some embodiments, the terminal device 110 may perform speech recognition (ASR) on the received audio data to determine the text data corresponding to the audio data. The terminal device 110 may, for example, use a trained ASR model to perform speech recognition on the audio data to determine the text data. The ASR model may be a machine learning model local to the terminal device 110, or a machine learning model deployed in other devices (e.g., remote devices). The terminal device 110 may determine a query request based on the text data corresponding to the audio data. For example, the terminal device 110 may directly determine the text data as a query request. Thus, the text may be determined based on the collected audio data without manual input by the user, which may improve the efficiency of obtaining the query request.

[0082] In some embodiments, in the process of determining whether it is necessary to obtain the second media data, in response to determining that the difference between the acquisition time of the first media data and the acquisition time of the query request meets the predetermined time range, the correlation between the query request and the first media data can be determined. In response to determining that the correlation meets the predetermined threshold condition, it is determined that the second media data needs to be obtained. Specifically, the terminal device 110 can determine the correlation between the query request and the media data in response to determining that the query request is received within the predetermined time range. The predetermined time range can be any appropriate time range. As an example, the terminal device can determine that the query request is received within the predetermined time range in response to the difference between the time when the query request is started to be received and the time when the media data is received being less than a predetermined duration. For example, the terminal device 110 can determine that the query request is received within the predetermined time range when the query request is received when receiving the media data (for example, the user speaks the query request while shooting a video), the query request is received within 1 minute before the media data is received, the query request is received within 1 minute after the media data is received, and the like.

[0083] The terminal device 110 can use a machine learning model to determine the text description of the media data, and then determine the relevance based on the query request and the text description. The machine learning model can also be a local machine learning model of the terminal device 110, or a machine learning model deployed in other devices (such as remote devices). For example, the terminal device 110 can determine the text similarity between the text description and the query request, and determine the text similarity as the relevance between the query request and the media data. It can be understood that the more identical or similar words are included between the text description and the query request, the higher the text similarity between the two. In some embodiments, the terminal device 110 can also directly provide the media data and the query request to the machine learning model to directly determine the relevance between the two with the help of the machine learning model. For example, a numerical value between 0 and 1 can be used to represent the relevance, and the present disclosure does not limit the specific method for determining the relevance.

[0084] The terminal device 110 may also obtain a predetermined threshold condition and determine whether the correlation satisfies the predetermined threshold condition. The predetermined threshold condition may, for example, indicate a correlation threshold, for example, the value of the correlation is higher than a threshold value of 0.7 (or other values). The terminal device 110 may determine that the correlation satisfies the predetermined threshold condition in response to the correlation reaching the correlation threshold. The terminal device 110 may determine that the correlation does not satisfy the predetermined threshold condition in response to the correlation not reaching the correlation threshold.

[0085] If it is determined that the relevance meets the predetermined threshold condition, the terminal device 110 can obtain the second media data and provide a response to the query request based on the first media data and the second media data. For example, the terminal device 110 can use a trained response model to determine the response to the query request based at least on the first and second media data and the query request. The response model can be a machine learning model local to the terminal device 110, or it can be a machine learning model deployed in other devices (such as remote devices). As an example, if the media data is a video of a certain scenic spot, and the query request is "Is this a scenic spot in the travel notes? Please provide introduction information of the current scenic spot", at this time, the relevance between the query request and the media data meets the predetermined condition, and the relevant media data of the scenic spot in the travel notes can be determined from the travel application, and then the response can be provided. At this time, the response can include, for example: This is a scenic spot in the travel notes, and further provides introduction information of the scenic spot. The terminal device 110 can provide the response to the user in any appropriate manner. For example, the terminal device 110 can present the text corresponding to the response via its own screen display interface, and can play the audio corresponding to the response via a speaker, etc.

[0086] If it is determined that the relevance does not meet the predetermined threshold condition, the terminal device 110 can obtain context data associated with the query request based on the interaction history in the digital assistant. The interaction history is the interaction history between the user who provides the query request and the terminal device. The context data associated with the query request may include historical interaction information between the user and the digital assistant, and the historical interaction information may include the user's historical query requests, historical responses to historical query requests, historically acquired media data, etc. In some embodiments, the context data associated with the query request may also include any appropriate data such as system information of the terminal device 110, environmental information of the physical environment in which the terminal device 110 is located, user attribute information of the user who sends the query request, etc.

[0087] The terminal device 110 can provide a response to the query request based on the context data associated with the query request. For example, the terminal device 110 can use a trained response model to determine the response to the query request based on the context data associated with the query request and the query request. Similarly, the response model can be a machine learning model local to the terminal device 110, or a machine learning model deployed in other devices (such as remote devices). The terminal device 110 can provide the response to the user. Determining the response based on the context data can ensure the accuracy of the response provided.

[0088] In some embodiments, the terminal device 110 may also provide feedback information to obtain another query request for the media data. The feedback information may be used to prompt the user that the correlation between the query request and the currently acquired media data is poor, prompting the user to provide a query request with a higher correlation with the media data. As an example, if the media data is a video of a certain scenic spot, and the query request is "what day is today", the terminal device 110 may determine that the correlation between the media data and the query request does not meet the predetermined threshold condition, obtain the context data associated with the query request, and provide a response based on the morning and afternoon data. The response may be, for example, "Today's date is XX year XX month XX day". The terminal device 110 may further provide feedback information "Do you want to ask questions associated with the video". The terminal device 110 may provide feedback information in any appropriate manner. For example, the terminal device 110 may present the text corresponding to the feedback information via its own screen display interface, and may play the audio corresponding to the feedback information via a speaker, and so on.

[0089] The terminal device 110 may receive another query request and determine another correlation between the other query request and the media data. The terminal device 110 may determine whether the other correlation satisfies a predetermined threshold condition. In response to determining that the other correlation satisfies the predetermined threshold condition, the terminal device 110 may provide another response to the other query request based on the media data. In response to determining that the other correlation does not satisfy the predetermined threshold condition, the terminal device 110 may first provide a response to the other query request, and then provide feedback information to the user to obtain other query requests for the media data. That is, the terminal device 110 may first provide a response to the query request and then obtain other query requests in response to the query request being irrelevant to the media data, and may repeatedly determine the correlation between the newly obtained query request and the media data.

[0090] In some embodiments, when providing an answer, the terminal device 110 may overlay an answer provision representation on the screen display interface. The answer provision representation may be used to prompt the user that an answer is currently being provided to the user. For example, if the terminal device 110 only obtains the query request but does not obtain the media data, the terminal device 110 may present the interactive interface shown in example 200A. If the terminal device 110 provides an answer via a speaker (i.e., plays an answer voice), in the process of providing an answer, example 200A may also present an answer provision representation 210, which is used to indicate that an answer voice is currently being played.

[0091] For another example, if the terminal device 110 acquires media data via a camera, the terminal device 110 may present the interactive interface shown in example 200B. If the terminal device 110 provides a response (i.e., plays a response voice) via a speaker, during the process of providing a response, the terminal device 110 may overlay a response providing representation 232 on the thumbnail representation 230 of the digital assistant, and the response providing representation 232 is used to indicate that the response voice is currently being played.

[0092] For another example, if the terminal device 110 obtains media data via the screen display interface, the terminal device 110 may switch back to presenting example 200G in response to an interactive request for a thumbnail representation. Afterwards, if the terminal device 110 provides a response (i.e., plays a response voice) via a speaker, during the process of providing the response, the terminal device 110 may superimpose a response providing representation 280 at the identifier of the digital assistant A (e.g., the avatar of the digital assistant A) in example 200G, and the response providing representation 280 is used to indicate that the response voice is currently being played.

[0093] In some embodiments, the user's interactive interface with the digital assistant may further include a subtitle control. For example, example 200A and example 200G may include a subtitle control 205. The terminal device 110 may switch to presenting the subtitle control 205 in response to receiving a trigger for the subtitle control 205. Figure 2H Example 200H is shown. Example 200H may present text corresponding to the response. For example, the terminal device 110 may present the text in streaming in example 200H, that is, as the number of responses provided increases, the number of texts presented in example 200H also increases.

[0094] In some embodiments, the terminal device 110 may also present an interruption control in the screen display interface during the provision of an answer. For example, the terminal device 110 may also provide an interruption control 220 in Example 200B, Example 200G, and Example 200H. The terminal device 110 may stop providing an answer to the user in response to receiving a trigger for the interruption control 220. Even if only a portion of the answer is provided, the terminal device 110 will no longer provide the user with the remaining answer. Thus, if the currently provided answer does not meet the user's expectations or the user is not interested in it, the user can instruct the terminal device 110 to stop providing the answer by triggering the interruption control 220, and the user can re-provide a query request and / or instruct the terminal device 110 to recapture the media data, which helps to improve the efficiency of the answer.

[0095] In some embodiments, assuming that the second media data is collected from a second application (for example, a travel application), a thumbnail representation of the second application can be superimposed and displayed at the terminal device, that is, a thumbnail representation of the application that is the data source of the second media data can be superimposed and displayed. In this way, the user can be supported to understand that media data is currently being collected from the second application. In response to receiving an interaction with the thumbnail representation, a control for stopping the collection of media data can be further provided, thereby improving data security. Alternatively and / or additionally, a control for switching to the second application can be further provided, thereby quickly switching to the second application. It should be understood that the second media data collected here may include: data in the page currently being presented in the second application, data in the previously visited page in the second application, and so on. In this way, with the permission of the user, one's own needs can be described in a more accurate manner.

[0096] In some embodiments, the terminal device 110 may also obtain context data associated with the media data based on the interaction history in the digital assistant in response to determining that no query request has been received within a predetermined time range. The context data associated with the media data may include historical interaction information between the user and the digital assistant, and the historical interaction information may include the user's historical query requests, historical responses to historical query requests, historical media data obtained, etc. In some embodiments, the context data associated with the media data may also include any appropriate data such as system information of the terminal device 110, environmental information of the physical environment in which the terminal device 110 is located, and user attribute information of the user who sent the query request.

[0097] The terminal device 110 may determine the query intent based on the context data and the query request associated with the media data, and provide a response to the media data based on the query intent. As an example, if the media data is a video of a certain scenic spot, and the user's previous query request is "coffee recommendation", the terminal device 110 may determine that the query intent is to find a coffee shop near the scenic spot corresponding to the media data. In this case, the terminal device 110 may also provide feedback information to obtain the query request for the media data. For example, the terminal device 110 may provide feedback information such as "Do you want to find a coffee shop near scenic spot XX?", "What do you want to ask about the video?", etc.

[0098] In some embodiments, in response to determining that the difference between the acquisition time of the first media data and the acquisition time of the query request does not meet the predetermined time range, context data associated with the query request is acquired based on the interaction history. The interaction history is the interaction history between the user who provides the query request and the terminal device. Further, the query intent can be determined based on the context data and the query request; and a response to the query request can be provided based on the query intent. Specifically, if the interval between the first media data and the query request is too long, for example, more than 3 minutes or other time intervals, it means that the query request may be irrelevant to the first media data. At this time, context data related to the query request can be acquired, for example, other questions asked by the user to the digital assistant in a certain period of time before and after receiving the query request, interactions with the answers provided by the digital assistant, interactions with other applications, and so on. At this time, the context data can reflect the user's latest query intent, and thus a response that better matches the user's expectations can be provided based on the latest query intent.

[0099] Figure 3 An example 300 according to some embodiments of the present disclosure is shown. The dashed arrows in the example 300 indicate the transmission of uplink data, and the solid arrows indicate the transmission of downlink data. Figure 3 As shown, the terminal device 110 can collect query requests (such as voice 301) and media data (such as video 302) during the interaction between the user 140 and the digital assistant. The terminal device 110 can present the collected video 302 via the screen 303. The terminal device 110 can provide the collected voice 301 and video 302 to the real-time communication (RTC) gateway 315 of the RTC service 310.

[0100] In some embodiments, during the acquisition of the video 302, the RTC gateway 315 may provide the acquired video 302 to the model engineering module 340. The model engineering module 340 may perform calculations of text, pre-calculated frames, contextual data, and the like. For example, the model engineering module 340 may process the video frames to obtain pre-calculated frames, and may determine responses based on the text and pre-calculated frames. The model engineering module 340 involves a model application framework RTC gateway 342, a model application framework 344, and a model service 346. The model application framework RTC gateway 342 is used to receive the voice 301 and video 302 sent by the RTC gateway 315. The model engineering module 340 obtains multiple video frames in the video 302, and may call a machine learning model (e.g., a visual language model) to perform pre-coding on the multiple video frames to obtain multiple pre-calculated frames.

[0101] After the video 302 and the voice 301 are collected, the RTC gateway 315 can provide the voice 301 to the voice gateway 332 of the voice service 330. The voice gateway 332 can provide the voice 301 to the ASR service 334. The ASR service 334 can perform ASR processing on the voice 301 to determine the text corresponding to the voice 301. The ASR service 334 can determine the text corresponding to the voice 301 by means of a trained ASR model, for example. The ASR service 334 can provide the determined text to the voice interaction interface (VUI) engine 325 in the answering service 320.

[0102] The VUI engine 325 can provide the session creation request corresponding to the text and the interaction process to the model engineering module 340. The model engineering module 340 can call the machine learning model to determine the response to the text based on the text and the pre-calculated frame, and provide the response to the VUI engine 325. The model engineering module 340 can directly use the multiple pre-calculated frames determined by the video acquisition process to determine the model output for the text. In this case, the model engineering module 340 does not need to encode the video frame again, and can directly call the pre-encoded result, which can improve the efficiency of the model calculation and thus improve the efficiency of the response. Alternatively and / or additionally, one or more videos can be cached at the model service 346 (or other locations accessible to the model 350). In the process of determining the query context, the video that best matches the query request can be determined from the cached one or more videos to be used as the query context.

[0103] The response determined by the model engineering module 340 may be, for example, a response text, and the VUI engine 325 may provide the response text to a text-to-speech (TTS) service 336 in the voice service 330. The TTS service 336 may perform TTS on the response text to determine the response voice corresponding to the response text. The TTS service 336 may determine the response voice, for example, with the aid of a trained TTS model. The VUI engine 325 may provide the response voice to the terminal device 110 via the RTC gateway 315. The terminal device 110 may play the response voice via the speaker 304.

[0104] In one example, a prompt word may be determined based on at least one of a query request, media data, and intermediate processing results for the query request and the media data. The prompt word may be input to the model 350, and a response from the model may be received. For example, assuming that the media data is a video of a certain scenic spot, and the query request is "Where is this?" The prompt word may include, for example, "Please confirm the name and related introduction of the scenic spot in the video", etc. For another example, assuming that the media data is a shared page including a travel note, the prompt word may include, for example, "Please provide a travel guide for the scenic spot in the video", etc.

[0105] In summary, according to the embodiments of the present disclosure, in a multimodal query process, media data and query requests can be obtained in real time, and in response to the strong correlation between the query request and the media data, a response to the query request is determined based on the media data. This can not only enrich the query method, but also help improve the efficiency and quality of the response.

[0106] Figure 4 400 according to some embodiments of the present disclosure. The method 400 may be implemented in Figure 1A The terminal device 110 will refer to Figure 1A The method 400 is described with reference to the environment 100A.

[0107] At block 410, the first media data is acquired by a first acquisition device of the terminal device. At block 420, in response to determining that the second media data needs to be acquired based on the first media data and the query request, the second media data is acquired by a second acquisition device of the terminal device. At block 430, a response to the query request is provided based on the first media data and the second media data.

[0108] In some embodiments, providing a response to a query request includes: generating a prompt word for processing the query request based on the first media data, the second media data, and the query request; and providing a response based on an answer to the prompt word from a machine learning model.

[0109] In some embodiments, the first media data comes from an external environment of the terminal device, and the second media data comes from an application running in the terminal device.

[0110] In some embodiments, method 400 is executed in a first application running in a terminal device, and obtaining second media data by a second acquisition device of the terminal device includes: determining an application from at least one second application running in a client device; and acquiring the second media data from the application.

[0111] In some embodiments, determining an application from at least one second application running at a client device includes: providing at least one application identifier of the at least one second application; and in response to receiving a selection request for an application identifier in the at least one application identifier, determining the application based on the selection request.

[0112] In some embodiments, determining an application from at least one second application running at the client device includes: generating a prompt word for determining the application based on the first media data and the query request; and determining the application based on an answer to the prompt word from a machine learning model.

[0113] In some embodiments, determining an application from at least one second application running at the client device includes: determining a correlation between at least any one of the first media data and the query request and the at least one second application; and determining an application from the at least one second application based on the correlation.

[0114] In some embodiments, the method 400 further includes: in response to determining that the screen display interface of the terminal device is switched to the second application, superimposing a screen sharing representation on the screen display interface, the screen sharing representation indicating that the screen display interface is shared with the first application; and superimposing a thumbnail representation of the first application on the screen display interface.

[0115] In some embodiments, the method 400 further includes: in response to receiving an interaction request for the thumbnail representation, switching the screen display interface of the terminal device to the first application.

[0116] In some embodiments, determining the need to acquire the second media data further includes: in response to determining that the difference between the acquisition time of the first media data and the acquisition time of the query request satisfies a predetermined time range, determining the correlation between the query request and the first media data; and in response to determining that the correlation satisfies a predetermined threshold condition, determining the need to acquire the second media data.

[0117] In some embodiments, the method 400 further includes: in response to determining that the relevance does not meet a predetermined threshold condition, obtaining context data associated with the query request based on the interaction history, the interaction history being the interaction history between the user providing the query request and the terminal device; and providing a response to the query request based on the context data.

[0118] In some embodiments, the method 400 further includes: in response to determining that the difference between the acquisition time of the first media data and the acquisition time of the query request does not meet the predetermined time range, based on the interaction history, obtaining context data associated with the query request, the interaction history being the interaction history between the user who provides the query request and the terminal device; based on the context data and the query request, determining the query intent; and providing a response to the query request based on the query intent.

[0119] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 5 An exemplary structural block diagram of an apparatus 500 for querying according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in a terminal device 110. Each module / component in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0120] like Figure 5As shown, the apparatus 500 includes: a first acquisition module 510, configured to acquire first media data by a first acquisition device of a terminal device; a second acquisition module 520, configured to acquire second media data by a second acquisition device of the terminal device in response to determining that the second media data needs to be acquired based on the first media data and a query request; and a providing module 530, configured to provide a response to the query request based on the first media data and the second media data.

[0121] In some embodiments, the providing module 530 is further configured to: generate a prompt word for processing the query request based on the first media data, the second media data and the query request; and provide a response based on an answer to the prompt word from the machine learning model.

[0122] In some embodiments, the first media data comes from an external environment of the terminal device, and the second media data comes from an application running in the terminal device.

[0123] In some embodiments, the apparatus 500 is executed in a first application running in a terminal device, and the second acquisition module is further configured to include: determining an application from at least one second application running in a client device; and acquiring second media data from the application.

[0124] In some embodiments, the second acquisition module 520 is further configured to: provide at least one application identifier of at least one second application; and in response to receiving a selection request for an application identifier in the at least one application identifier, determine the application based on the selection request.

[0125] In some embodiments, the second acquisition module 520 is further configured to: generate a prompt word for determining an application based on the first media data and the query request; and determine the application based on an answer to the prompt word from the machine learning model.

[0126] In some embodiments, the second acquisition module 520 is further configured to: determine a relevance between at least any one of the first media data and the query request and at least one second application; and determine an application from the at least one second application based on the relevance.

[0127] In some embodiments, the providing module 530 is further configured to: in response to determining that the screen display interface of the terminal device is switched to the second application, superimpose a screen sharing representation on the screen display interface, the screen sharing representation indicating that the screen display interface is shared with the first application; and superimpose a thumbnail representation of the first application on the screen display interface.

[0128] In some embodiments, the providing module 530 is further configured to: in response to receiving an interaction request for the thumbnail representation, switch the screen display interface of the terminal device to the first application.

[0129] In some embodiments, the first acquisition module 510 is further configured to: determine the correlation between the query request and the first media data in response to determining that the difference between the acquisition time of the first media data and the acquisition time of the query request satisfies a predetermined time range; and determine that the second media data needs to be acquired in response to determining that the correlation satisfies a predetermined threshold condition.

[0130] In some embodiments, the providing module 530 is further configured to: in response to determining that the relevance does not meet a predetermined threshold condition, obtain context data associated with the query request based on the interaction history, where the interaction history is the interaction history between the user providing the query request and the terminal device; and provide a response to the query request based on the context data.

[0131] In some embodiments, the providing module 530 is further configured to: in response to determining that the difference between the acquisition time of the first media data and the acquisition time of the query request does not meet the predetermined time range, based on the interaction history, obtain context data associated with the query request, the interaction history being the interaction history between the user who provides the query request and the terminal device; based on the context data and the query request, determine the query intent; and provide a response to the query request based on the query intent.

[0132] In some embodiments, the first acquisition module 510 is further configured to: determine a text description of the first media data using a machine learning model; and determine a relevance based on the query request and the text description.

[0133] The units and / or modules included in the device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 500 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0134] It should be understood that one or more steps in the above method can be performed by a suitable electronic device or a combination of electronic devices. Such an electronic device or a combination of electronic devices may include, for example, Figure 1A The terminal device 110 and / or server 120 in.

[0135] Figure 6 1 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 6 The electronic device 600 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 6 The electronic device 600 shown can be used to implement Figure 1A terminal device 110 and / or server 120.

[0136] like Figure 6 As shown, the electronic device 600 is in the form of a general electronic device. The components of the electronic device 600 may include, but are not limited to, one or more processors or processing units 610, a memory 620, a storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processing unit 610 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 600.

[0137] The electronic device 600 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 600.

[0138] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 6As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0139] The communication unit 640 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0140] The input device 650 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 660 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 600, or communicate with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0141] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0142] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0143] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0144] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0145] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as an update, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0146] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A query method, comprising: Acquiring first media data by a first acquisition device of the terminal device; In response to determining that second media data needs to be acquired based on the first media data and the query request, a second acquisition device of the terminal device acquires the second media data; as well as A response to the query request is provided based on the first media data and the second media data.

2. The method of claim 1, wherein providing the response to the query request comprises: generating a prompt word for processing the query request based on the first media data, the second media data and the query request; as well as The response is provided based on an answer to the prompt word from a machine learning model. 3 . The method according to claim 1 , wherein the first media data comes from an external environment of the terminal device, and the second media data comes from an application running in the terminal device.

4. The method according to claim 3, wherein the method is executed in a first application running in the terminal device, and acquiring the second media data by the second acquisition device of the terminal device comprises: determining the application from at least one second application running at the client device; as well as The second media data is collected from the application.

5. The method of claim 4, wherein determining the application from the at least one second application running at the client device comprises: providing at least one application identifier of the at least one second application; as well as In response to receiving a selection request for an application identifier among the at least one application identifier, the application is determined based on the selection request.

6. The method of claim 4, wherein determining the application from the at least one second application running at the client device comprises: Based on the first media data and the query request, generating a prompt word for determining the application; as well as The application is determined based on an answer to the prompt word from the machine learning model.

7. The method of claim 4, wherein determining the application from at least one second application running at the client device comprises: Determining a relevance between at least any one of the first media data and the query request and the at least one second application; as well as Based on the relevance, the application is determined from among the at least one second application.

8. The method according to claim 4, further comprising: In response to determining that the screen display interface of the terminal device is switched to the second application, superimposing a screen sharing indication on the screen display interface, the screen sharing indication indicating sharing the screen display interface with the first application; as well as A thumbnail representation of the first application is superimposed on the screen display interface.

9. The method according to claim 8, further comprising: In response to receiving an interaction request for the thumbnail representation, switching the screen display interface of the terminal device to the first application.

10. The method according to claim 1, wherein determining that the second media data needs to be obtained further comprises: In response to determining that a difference between a time when the first media data is obtained and a time when the query request is obtained satisfies a predetermined time range, determining a relevance between the query request and the first media data; as well as In response to determining that the correlation satisfies a predetermined threshold condition, it is determined that the second media data needs to be acquired.

11. The method according to claim 10, further comprising: In response to determining that the correlation does not satisfy the predetermined threshold condition, Acquire context data associated with the query request based on an interaction history, wherein the interaction history is an interaction history between a user who provides the query request and the terminal device; as well as Based on the context data, a response to the query request is provided.

12. The method according to claim 1, further comprising: In response to determining that a difference between a time when the first media data is obtained and a time when the query request is obtained does not satisfy a predetermined time range, obtaining context data associated with the query request based on an interaction history, the interaction history being an interaction history between a user who provided the query request and the terminal device; Determining query intent based on the context data and the query request; as well as A response to the query request is provided based on the query intent.

13. The method according to claim 10, wherein determining the relevance between the query request and the first media data comprises: Determining a text description of the first media data using a machine learning model; as well as The relevance is determined based on the query request and the text description.

14. A device for querying, comprising: A first acquisition module is configured to acquire first media data through a first acquisition device of a terminal device; A second acquisition module is configured to, in response to determining that second media data needs to be acquired based on the first media data and the query request, acquire the second media data by a second acquisition device of the terminal device; as well as A providing module is configured to provide a response to the query request based on the first media data and the second media data.

15. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 13 when executed by the at least one processing unit.

16. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 13.

17. A computer program product, comprising a computer program, wherein the computer program implements the method according to any one of claims 1 to 13 when executed by a processor.

Images