Data processing methods and electronic device

Abstract

A data processing procedure comprises determining a user task based on a user's interaction information and, based on the user task, determining at least one piece of target data from a plurality of data pieces contained in a user's dataset. The plurality of data pieces is determined based on historical, cross-device, multimodal interaction information related to the user and includes prediction task information. The procedure further comprises performing aggregation processing on the at least one piece of target data to obtain an aggregation processing result and generating a target processing result that corresponds to the user task, based on the aggregation processing result.

Description

REFERENCE TO RELATED REGISTRATION

[0001] This application claims priority over Chinese patent application No. 202411367772.9, filed on September 27, 2024, the entire contents of which are hereby incorporated by reference. TECHNICAL AREA

[0002] The present disclosure relates generally to the field of computer technology and in particular to a data processing method and an electronic device. TECHNICAL BACKGROUND

[0003] In daily study and work, users often need to refer back to previously consulted information to help them solve a current task. However, this information is often scattered across various electronic devices, and organizing it uniformly is even more difficult, making it challenging for users to access the information efficiently. Therefore, the question of how to effectively organize and utilize historical information related to the user has become a pressing problem that requires resolution. OVERVIEW OF THE INVENTION

[0004] According to the disclosure, a data processing method is provided which includes determining a user task based on a user's interaction information and determining, based on the user task, at least one piece of target data from a plurality of data pieces contained in a user's data set. The plurality of data pieces is determined based on historical, cross-device, multimodal interaction information related to the user and includes prediction task information. The method further includes performing aggregation processing on the at least one piece of target data to obtain an aggregation processing result and generating a target processing result that corresponds to the user task, based on the aggregation processing result.

[0005] Also according to the disclosure, an electronic device is provided which has a memory in which instructions are stored and a processor configured to execute the instructions to determine a user task based on a user's interaction information and, based on the user task, to determine at least one piece of target data from a plurality of data pieces contained in a data set of the user. The plurality of data pieces is determined based on historical, cross-device, multimodal interaction information related to the user and includes prediction task information.The processor is further configured to execute instructions to perform aggregation processing on the at least one piece of target data in order to obtain an aggregation processing result, and to generate a target processing result based on the aggregation processing result that corresponds to the user task.

[0006] Also according to the disclosure, a data processing method is provided which includes performing a semantic analysis on a user's interaction information to obtain at least one piece of data. The interaction information represents cross-device, multimodal interaction information related to the user. The method further includes predictions based on the at least one piece of data, at least one prediction task corresponding to the at least one piece of data, storing the at least one prediction task in association with corresponding data, and performing clustering processing on the at least one piece of data to obtain a data record of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings required for use in the description of the embodiments are briefly presented below. The drawings described below are exemplary embodiments of this disclosure. A person skilled in the art can derive other drawings from these drawings without any creative effort. Fig. Figure 1 is a flowchart of a data processing procedure according to embodiments of the present disclosure. Fig. Figure 2 is a flowchart of another data processing method according to embodiments of the present disclosure. Fig. Figure 3 is a flowchart showing the use of a data processing method to perform task processing according to embodiments of the present disclosure. Fig. Figure 4 is a schematic diagram showing a target processing result produced by a data processing method according to embodiments of the present disclosure. Fig. Figure 5 is a schematic diagram showing a task area realized by a data processing method according to embodiments of the present disclosure. Fig. Figure 6 is a flowchart showing the use of a data processing procedure to generate a data set according to embodiments of the present disclosure. Fig. Figure 7 is a schematic hardware diagram of an electronic device according to embodiments of the present disclosure. Fig. Figure 8 is a schematic hardware diagram of another electronic device according to embodiments of the present disclosure. DETAILED DESCRIPTION OF THE EXECUTION FORMS

[0008] Various schemes and features of the present disclosure are described below with reference to the accompanying drawings. It is understood that various modifications can be made to the embodiments of the present disclosure. Therefore, the description should not be considered limiting, but only as an example of embodiments. All other embodiments that a person skilled in the art obtains without creative effort based on the embodiments in the present disclosure are within the scope of the present disclosure.

[0009] The following description may use the terms "in one embodiment," "in another embodiment," or "in some embodiments," which can describe a subset of all possible embodiments. It is understood, however, that "some embodiments" can refer to the same subset or to different subsets of all possible embodiments, and that these can be combined without conflict.

[0010] The terms "first", "second", and "third" are used merely to distinguish similar objects and do not specify a particular order of the objects. It is understood that, where permitted, the objects described in connection with "first", "second", or "third" may exist in any order or sequence, so that the embodiments of the present disclosure described herein may be implemented in a different order than that illustrated or described herein.

[0011] Unless otherwise defined, all technical and scientific terms used in this disclosure have the meanings normally understood by a person skilled in the art. The terms used in this disclosure serve only descriptive purposes and are not intended to limit the scope of this disclosure.

[0012] In everyday life, a user is exposed to various types of information scattered across different devices and applications, for example, mobile devices, personal computers (PCs), cloud storage, messaging applications, social applications, etc.

[0013] Because information is scattered across various devices and applications, it is difficult to manage and use it consistently and efficiently, making it hard for users to quickly find the information they need when they want to use it. For example, when working on a task, users often have to switch back and forth between different devices and applications, spending considerable time and energy trying to find and organize relevant information, which hinders the achievement of a state of efficient flow.

[0014] Even when users consciously organize and manage information, they typically search for it in folders or using identifiers. However, due to the complex network relationships between different types of information, the additional data generated during the organization and management process can disrupt the original hierarchical structure or identifier system, making effective management extremely challenging.

[0015] When retrieving historical information, users must extract useful content from a large amount of information (for example, reference materials, web links, chat logs, etc.). Even worse, users may not even remember having encountered certain information and cannot think of precise keywords to search for it. Therefore, they cannot actively find the required content. In this case, the efficiency of information use is significantly reduced.

[0016] To solve the above technical problems, various solutions have been proposed.

[0017] For example, information from various devices related to the user is synchronized to a cloud server to achieve unified storage of historical information.

[0018] In another example, tools for synchronizing multiple devices are used to synchronize information on different user-related devices onto the same device in order to achieve data synchronization across devices.

[0019] In another example, information management and organizational functions are provided by applications, and then different types of information are aggregated for management.

[0020] However, the solutions above still require users to actively store, manage, and search for information, placing high demands on their information management skills (including time and ability requirements); furthermore, managing fragmented information effectively is difficult. Therefore, users still face the problem of low information utilization efficiency.

[0021] The present disclosure provides a data processing method that can be executed by a processor of an electronic device to at least partially alleviate the problems mentioned above. The electronic device may be a server, a laptop, a tablet, a desktop computer, a smart TV, a TV box, a mobile device (such as a mobile phone, a portable video device, a personal digital assistant, a special messaging device, or a portable gaming device), or any other device with data processing capabilities.

[0022] In some embodiments, the data processing method provided by the present disclosure can be implemented as a cross-platform agent application. An agent can run on at least one device associated with the user, for example, the user's mobile phone, laptop, tablet, desktop computer, or a cloud device where user data is stored. Furthermore, the agent can have cross-platform data-reading capability to read and analyze application data from various applications running on the electronic device. To read and analyze application data from various applications running on the electronic device, the agent may require user authorization.

[0023] In one embodiment, which is in Fig. Figure 1, which is a flowchart of a data processing method provided by the present embodiment, the data processing method comprises steps S101 to S104.

[0024] In S101, a user task is determined based on initial interaction information from a user.

[0025] The initial interaction information can be information determined based on a current user interaction process.

[0026] In some embodiments, the initial interaction information can include multimodal information. For example, the initial interaction information can include text information, image information, audio information, video information, user process information, and the like.

[0027] In some embodiments, the initial interaction information can be information obtained from different devices. For example, the initial interaction information can be information obtained from an electronic device currently used by the user, or information obtained from a cloud device for storing user-related interaction information.

[0028] Determining the user task based on the initial interaction information may involve identifying a task that the user expects to solve by performing text recognition, object recognition, semantic analysis, and the like on the initial interaction information.

[0029] In some embodiments, when using an agent to implement the data processing method provided by the present disclosure, the agent can use a multimodal large-scale model to process the initial interaction information in order to determine the user task. For example, if the initial interaction information includes text and image information, the multimodal large-scale model can be used to perform semantic extraction from the text and image information, and the corresponding user task can be determined based on the extracted semantic information.

[0030] In some embodiments, the multimodal large model can include a multimodal large language model.

[0031] In S102, based on the user task, at least one piece of first target data is determined from several pieces of first data contained in a user dataset, wherein the several pieces of first data are determined based on historical interaction information of cross-device multimodality related to the user and the several pieces of first data include prediction task information.

[0032] The historical interaction information of cross-device multimodality can be historical interaction information of multimodality obtained from multiple devices.

[0033] In some embodiments, the multiple devices may include the user's electronic devices, for example a mobile phone, tablet, laptop, desktop computer, camera, user's portable hard drive, etc., and they may also include electronic devices on which information relating to the user is stored, for example a cloud storage device, etc.

[0034] In einigen Ausführungsformen können die Informationen von Multimodalität Textinformationen, Bildinformationen, Audioinformationen, Videoinformationen, aus Nachrichten-Anwendungen oder sozialen Anwendungen gelesene Informationen usw. umfassen.

[0035] Daher können die mehreren Stücke von ersten Daten durch Analysieren und Verarbeiten der in Beziehung zum Benutzer stehenden historischen Interaktionsinformationen vorrichtungsübergreifender Multimodalität erlangt werden. Basierend auf den mehreren Stücken von ersten Daten kann ein Datensatz des Benutzers etabliert werden.

[0036] In einigen Ausführungsformen kann der Datensatz eine Vektordatenbank sein, die nach einer Vektorisierungsverarbeitung und semantischen Analyse der historischen Interaktionsinformationen vorrichtungsübergreifender Multimodalität erlangt wurde.

[0037] In einigen Ausführungsformen kann der Datensatz eine persönliche Wissensbasis (PKB - Personal Knowledge Base) eines Benutzers oder eine Datenbank für eine Abruf-augmentierte Erzeugung (RAG - Retrieval-Augmented Generation) sein, die für die RAG-Technik verwendet wird.

[0038] Die Vorhersageaufgabeninformationen können Aufgabeninformationen sein, die den mehreren Stücken von ersten Daten entsprechen, die basierend auf der Seite oder dem Benutzervorgang, die den mehreren Stücken von ersten Daten entsprechen, bestimmt wurden.

[0039] In some embodiments, the multiple pieces of initial data may include one or more pieces of prediction task information.

[0040] In dem Datensatz können die mehreren Stücke von ersten Daten und mindestens ein damit in Beziehung stehendes Stück Vorhersageaufgabeninformationen einander zugeordnet gespeichert sein. Daher kann beim Abrufen von Daten die Relevanz der mehreren Stücke von ersten Daten für die Abrufinformationen basierend auf den Vorhersageaufgabeninformationen bestimmt werden.

[0041] In some embodiments, determining the at least one piece of first target data from the dataset based on the user task may involve performing a retrieval on the user's dataset based on the semantic information contained in the user task in order to determine, from the multiple pieces of first data, the at least one piece of first target data that is closest to the semantic information of the user task.

[0042] In S103, aggregation processing is performed on at least one piece of the initial target data to obtain an aggregation processing result.

[0043] Performing aggregation processing on the at least one piece of initial target data may include extracting information related to the user task from the at least one piece of initial target data and organizing the extracted information into a logical aggregation processing result.

[0044] In some embodiments, after determining the at least one piece of first target data from the user dataset, the agent can perform aggregation processing on the at least one piece of first target data using a large model.

[0045] In S104, a target processing result is generated based on the aggregation processing result, which corresponds to the user task.

[0046] The target processing result can be a processing result to be presented to the user.

[0047] In some embodiments, the target processing result may include an image generation result, a text generation result, a knowledge tree generation result, a speech generation result, etc., which are related to the user task and are to be presented to the user.

[0048] In some embodiments, the agent can generate the appropriate target processing result using a large model based on the aggregation processing result.

[0049] In some embodiments, the data processing method may further include determining a task scenario that corresponds to the user task and determining a display form of the target processing result based on the task scenario.

[0050] In some embodiments, based on the task scenario, a user interface component (UI component, where UI stands for User Interface) suitable for displaying information can be called to generate the target processing result.

[0051] If the task scenario corresponding to the user task is, for example, a meeting scenario, the target processing result can include at least a summary response corresponding to the user task and a source file link corresponding to the summary response, for example, a link to a web page, an image, or a folder, etc., that is related to the response.

[0052] In another example, if the task scenario corresponding to the user task is an essay reading scenario, the target processing result can include a knowledge tree generated from the retrieval results, where the knowledge tree has knowledge points of different levels that are related to the user task and corresponding source file links.

[0053] In the data processing method provided by this disclosure, the user task can be determined based on initial user interaction information. Based on the user task, at least one piece of initial target data can be determined from the multiple pieces of initial data contained in the user's dataset. These multiple pieces of initial data are determined based on the user's historical interaction information across devices and multimodality, and include the prediction task information. Aggregation processing can be performed on the at least one piece of initial target data to obtain the aggregation processing result. Based on the aggregation processing result, the target processing result corresponding to the user task can be generated.Therefore, the user task can be extracted from the user's interaction information, and the corresponding target processing result can be generated, making the identification and processing of the user task more intelligent. Furthermore, the user's dataset can be generated based on cross-device, multimodal user data to produce the target processing result that corresponds to the user task, thereby enabling the linking of user data from all scenarios, cross-device and cross-mode searching, and the application of personal data.Furthermore, the multiple pieces of initial data in the user's dataset can include the relevant prediction task information, enabling task-oriented organization and management of the user's cross-device multimodal data, thereby making the user's dataset retrieval process more accurate and efficient, and improving the user data usage and management efficiency.

[0054] In some embodiments, the initial interaction information based on a user input process may include specific user input information and / or user-related, cross-device, multimodal application interaction information.

[0055] The user input process can be an input action by the user in a specified application. The user input information can be information based on input by the user in the specified application.

[0056] The specified application may be an application used to implement the data processing procedure provided in this disclosure, for example, an agent.

[0057] Cross-device multimodal application interaction information can be multimodal information obtained from multiple devices related to the user, for example, chat content information between the user and an interacting party obtained from a social application running on the user's mobile phone, web page information viewed by the user obtained from a web application running on the user's tablet, user selection information of a portion of the content on the currently viewed page obtained from the user's PC, and the like.

[0058] In some embodiments, the cross-device, multimodal application interaction information related to the user can be obtained by the agent. Therefore, the agent can be a system-level application running on an electronic device that is capable of obtaining multimodal information across devices and applications.

[0059] In one embodiment, determining the user task based on the user's initial interaction information (S101) can be implemented as step S1011 and / or S1012.

[0060] In S1011, the user task is determined based on the user input information.

[0061] The user input information can be understood semantically and the corresponding user task can be determined.

[0062] In some embodiments, the user input information can first be vectorized using a multimodal large-scale model to obtain a corresponding vector representation. Then, the vector representation of the user input information can be semantically extracted using the large-scale model to obtain the relevant semantic information. Finally, the large-scale model can be used to determine the user task based on the extracted semantic information.

[0063] In S1012, the user task is determined based on the application interaction information.

[0064] The acquired cross-device multimodal application interaction information can be understood semantically and the corresponding user task can be determined.

[0065] In some embodiments, information can first be extracted from the device-spanning multimodal information to obtain relevant information details. For example, text information, icon information, function key operation information, voice information, etc., can be extracted. The extracted information details can then be vectorized using a large-scale model to obtain corresponding vector representations. Subsequently, a semantic extraction can be performed on the vector representations corresponding to the application interaction information, again using a large-scale model, to obtain relevant semantic information. Finally, the user task can be determined based on the extracted semantic information using the large-scale model.

[0066] In the embodiments described above, the user task can be passively determined based on user input information. In some other embodiments, the user task can also be actively determined by collecting and analyzing cross-device, multimodal application interaction information related to usage. Multiple user task determination methods can be provided. Therefore, in the user interaction process, response information for related tasks can be passively or actively provided to the user in response to user input or other application interaction information, thereby improving the flexibility and intelligence of human-computer interaction and thus enhancing the user experience.

[0067] In some embodiments, determining the user task based on the user's initial interaction information (S101) may further include steps S1013 and S1014.

[0068] In S1013, based on the initial interaction information, at least one piece of second target data is determined from the multiple pieces of initial data, which are related to the interaction information.

[0069] Based on the currently obtained initial interaction information, a query can be performed on the user's historical interaction information to determine at least one piece of historical interaction information related to the initial interaction information.

[0070] In some embodiments, the initial interaction information can first be vectorized to obtain the corresponding vector representation. Then, the vector representation corresponding to the initial interaction information can be semantically analyzed to obtain the relevant semantic information. Subsequently, based on the semantic information corresponding to the initial interaction information, a retrieval can be performed on the user's historical interaction information. That is, a retrieval can be performed on the dataset comprising the multiple pieces of initial data to obtain at least one piece of second target data related to the initial interaction information.

[0071] If, for example, the initial interaction information is the user's input "Send an email with a team-building image to the administration department," then a semantic analysis of this initial interaction information can identify at least key semantic information, including "administration department," "team-building image," "Send to administration department," and "Email." Based on this identified semantic information, a query can then be performed on the dataset. This allows for the retrieval of specific department-related information from multiple departments, and the determination of the team-building activity currently specified by the user from multiple pieces of team-building information, and so on.

[0072] In S1014, the user task is determined based on the initial interaction information and at least one piece of the second target data.

[0073] Nach dem Bestimmen des mindestens einen Stücks von zweiten Ziel-Daten, die in Beziehung zu den ersten Interaktionsinformationen stehen, können die Inhalte der ersten Interaktionsinformationen in Kombination mit dem mindestens einen Stück von zweiten Ziel-Daten weiter angereichert werden, um die Benutzeraufgabe genauer zu beschreiben.

[0074] Zum Beispiel kann in dem obigen Fall basierend auf dem bestimmten mindestens einen Stück zweiter Ziel-Daten und den ersten Interaktionsinformationen bestimmt werden, dass die vom Benutzer zu beschreibende Benutzeraufgabe „Sende einer E-Mail des jüngsten Teambuilding-Bildes an die Personalabteilung“ ist.

[0075] Therefore, by combining the user's historical interaction information and further understanding the current initial interaction information, the user's intent can be understood more accurately, and a user task can be generated that is more in line with the user's original intent, thereby providing a more accurate goal-processing result.

[0076] In some embodiments, determining the user task based on the user's initial interaction information (S101) may include steps S1015 to S1017.

[0077] In S1015, application scenario information, corresponding to the initial interaction information, and historical operation information of the user are obtained.

[0078] The application scenario information can be multimodal information collected when obtaining the first interaction information from at least one device.

[0079] In einigen Ausführungsformen können die Anwendungsszenarioinformationen durch Durchführen einer Screenshot-Verarbeitung auf in Beziehung zum Benutzer stehenden Vorrichtungen, Lesen der Interaktionsinformationen des Benutzers in der spezifizierten Anwendung usw. erlangt werden.

[0080] Historical process information may include user interaction habits, preferences, or other information determined based on user interaction information collected within a specific period in the past, such as information about the frequency of the user clicking on messaging applications, that the user's reading time and frequency of image information is higher than that of text information, the user's preference to set the alarm clock every evening after 8 p.m., and so on.

[0081] In some embodiments, the historical process information may have been stored in advance in a user information library of the user.

[0082] In S1016, context information is determined based on the application scenario information and the historical process information, which corresponds to the initial interaction information.

[0083] Contextual information can be task background information related to the user task, contained in the initial interaction information, such as task scenario information, task context information, and the like. This contextual information can aid in understanding the user task.

[0084] In some embodiments, the task scenario information may include email scenarios, meeting scenarios, reading scenarios, project planning scenarios, and the like.

[0085] In some embodiments, the acquired application scenario information and the historical process information can be processed using a multimodal large model to determine the context information that corresponds to the initial interaction information.

[0086] For example, if the application scenario information is a screenshot captured using an agent, the screenshot can be processed by a multimodal large-scale target recognition or text recognition model to determine the task scenario corresponding to the current task. In another example, if the screenshot contains an email icon, the task scenario can be determined to be an email scenario. In yet another example, if the application scenario information is text information within a document being read by an agent, the task scenario can be determined to be a report reading scenario. In yet another example, the text context information corresponding to the portion of text selected by the user in the document can be determined by identifying the text information within the document, and so on.

[0087] In another example, the user's preference for reading image information can be used as contextual information to determine that the user task includes at least one task involving searching for related images.

[0088] In S1017, the user task is determined based on the context information and the initial interaction information.

[0089] In some embodiments, the context information and initial interaction information can be inputted into a large model to determine the user task using the large model.

[0090] The large model can use the contextual information to help understand the user task contained in the initial interaction information, thus enabling the large model to understand the user's intent more accurately.

[0091] In the embodiments described above, the contextual information corresponding to the initial interaction information can be determined by capturing application scenario information related to the initial interaction information and by obtaining historical user habit information. This contextual information can be used to more accurately understand the user task that the user intends to establish through the initial interaction information. Therefore, a target processing outcome can be provided that is more in line with the user's expectations, thereby improving the agent's intelligence in the human-computer interaction process.

[0092] In some embodiments, determining at least one piece of first target data from the multiple pieces of first data contained in the user's dataset, based on the user task (S102), may include:

[0093] S1021: Based on the user task, a retrieval is performed on the multiple pieces of initial data to determine the at least one piece of initial target data; wherein the retrieval includes at least performing a retrieval on the multiple pieces of initial data based on the prediction task information of the user task according to each piece of initial data.

[0094] In some embodiments, the data set can be in the form of a database. After defining the user task, the user task can be sent to the data set to perform a retrieval from the multiple pieces of initial data using a database retrieval function and to determine at least one piece of initial target data.

[0095] In some embodiments, the retrieval process of the multiple pieces of initial data can be implemented through clustering processing.

[0096] In some embodiments, the multiple pieces of first data in the dataset can be vectorized representations obtained by vectorizing the multimodal historical information related to the user across devices using a large-scale model, and the user task can also be a vectorized representation obtained after vectorization processing. Therefore, a retrieval can be performed on the multiple pieces of first data; that is, the at least one piece of first target data that is closest to the vector representation of the user task can be determined from the multiple pieces of first data.

[0097] In some other embodiments, the multiple pieces of first data in the dataset can be vector representations determined based on cross-device multimodal user information and possessing corresponding semantic information. Therefore, the at least one piece of first target data can be determined based on the semantic similarity between the multiple pieces of first data and the user task.

[0098] In some embodiments, a global retrieval of the data record can be performed when performing the retrieval on the user's data record.

[0099] The retrieval can be performed at least on the basis of the prediction task information corresponding to the user task and each piece of initial data; that is, the prediction task information corresponding to the initial data can be used as a retrieval condition to determine the similarity between each piece of initial data and the user task.

[0100] Since the prediction task information is able to represent at least one task related to the initial data, performing the retrieval on the initial data based on the prediction task information can determine the relevance of the multiple pieces of initial data and the user task from the perspective of the task according to each piece of initial data, thereby improving the accuracy and efficiency of the retrieval.

[0101] In some embodiments, the data processing prior to determining the at least one piece of first target data from the multiple pieces of first data contained in the user's data set, based on the user task (S102), may further include steps S105 to S107.

[0102] In S105, a semantic analysis is performed on the historical interaction information in order to obtain at least some initial data.

[0103] As described above, historical interaction information can be multimodal information obtained from multiple devices, for example, image information, text information, application interaction information, etc., which was obtained from devices such as the user's mobile phone, laptop, desktop computer or tablet, or a cloud storage device for user data.

[0104] In some embodiments, a superficial information extraction can be performed on the historical interaction information to identify text information, UI elements, etc. within the historical interaction information.

[0105] After the surface information extraction has been performed on the historical interaction information, vectorization processing can be carried out on the extracted information to obtain corresponding vectorized representation information.

[0106] In some embodiments, vectorization processing can be performed on the information obtained after surface information extraction using a large model.

[0107] In some embodiments, a semantic analysis can be performed on the vectorized representation information using a large model to map cross-device multimodal information into the same semantic space in order to obtain at least one piece of first data with semantic features.

[0108] In some embodiments, performing a semantic analysis on the vectorized representation information of the historical interaction information may include: determining at least one entity information or at least one concept information from the vectorized representation information and determining relationship information between entities or concepts, etc. For example, information such as date, place name, time, and the mapping relationship between this entity information may be determined.

[0109] In S106, based on at least one piece of first data, at least one prediction task corresponding to that piece of first data is predicted, and the at least one prediction task is stored in association with corresponding first data.

[0110] A task prediction can be performed on the at least one piece of initial data in order to determine the at least one prediction task that corresponds to the at least one piece of initial data.

[0111] In some embodiments, based on the semantic information corresponding to the at least one piece of initial data, the task information associated with the current page or user action corresponding to the at least one piece of initial data can be determined using a large-scale model. For example, based on the user's current text interaction content and the UI interface elements extracted from the page, the prediction task corresponding to the current text interaction content can be determined.

[0112] In some embodiments, the initial data may correspond to one or more prediction tasks.

[0113] If the initial data is, for example, the entity information "Calendar," then "Calendar" can be related to the meeting scheduling task in the meeting scenario, to the task of setting reminders in the family scenario, or to the task of sending emails at a scheduled time in the email scenario, and so on. Therefore, tasks such as "Meeting Schedule," "Setting Reminders," or "Sending Emails at a Scheduled Time" can all be set up as predictive tasks that correspond to "Calendar."

[0114] Furthermore, the at least one prediction task can be stored in association with the corresponding initial data, so that when performing a search in the data set, a more accurate retrieval result can be obtained based on the at least one prediction task.

[0115] In S107, clustering processing is performed on at least one piece of initial data to obtain the user's data set.

[0116] Clustering processing can be performed on at least one piece of initial data to establish a mapping relationship between different historical interaction information and to realize structured processing of the multiple pieces of initial data.

[0117] In some embodiments, the clustering processing can be performed on the at least one piece of first data based on any prior art clustering algorithm, for example the k-means clustering algorithm (k-Means algorithm), the mean-shift clustering algorithm (Mean Shift), the density-based spatial clustering analysis with noise (DBSCAN - Density-Based Spatial Clustering of Applications with Noise), or the expectation-maximization clustering algorithm (EM) based on the Gaussian mixture model (GMM).

[0118] In some embodiments, the clustering processing can be performed on at least one piece of initial data to obtain a clustering result in the form of a tree structure.

[0119] In some embodiments, a corresponding multimodal file index can be generated for the multiple pieces of first data in the dataset in order to perform a retrieval of the multiple pieces of first data based on the multimodal file index.

[0120] In the embodiments described above, mapping the historical interaction information of cross-device multimodality into the same semantic space allows for the retrieval of at least one piece of initial data and the prediction task corresponding to that initial data. Clustering processing can then be performed on that initial data to retrieve the user's dataset, thus enabling task-oriented organization and management of the user's historical interaction information. Subsequently, when performing a data retrieval, the retrieval can be carried out on multiple pieces of initial data based on the prediction task information contained within those multiple pieces, thereby improving the efficiency and accuracy of the retrieval.

[0121] In some embodiments, the data processing method may further comprise at least one of steps S108 or S109.

[0122] In S108, the user's data record is updated based on the user task and the target processing result.

[0123] Since the target processing result is able to characterize the adaptation of the user's current interaction information to the historical interaction information, updating the data set based on the user task and the target processing result can learn the user's usage habits and understanding of the knowledge structure and use this to generate subsequent user tasks.

[0124] In some implementations, based on the user task and the target processing result, at least one piece of entity information or at least one piece of concept information, as well as the mapping relationship between the entity information or the concept information, can be determined. The determined entity information or concept information and the mapping relationship information can then be stored in the data record.

[0125] In some embodiments, based on the correspondence between the user task and the target processing result, the mapping relationship between the multiple pieces of initial data stored in the dataset can be adjusted, or a new mapping relationship between the multiple pieces of initial data can be established.

[0126] In S109, in response to a user modification operation on the target processing result, the target processing result is updated and the user's record is updated based on the user task and the updated target processing result.

[0127] The target processing result that corresponds to the issued user task can be displayed to give the user the opportunity to modify the target processing result.

[0128] In some implementations, the output target processing result can be displayed in a pop-up window, a sidebar, or another location. The output target processing result can also be displayed in an interactive interface provided by the agent for executing the data processing procedure, and so on.

[0129] If the output of the target processing is displayed, for example, as a knowledge tree, the user can be allowed to modify or customize each node in the knowledge tree. If the output of the target processing is displayed as an image, the user can be allowed to modify the brightness, color, display position of each object, and other information of the image, and so on.

[0130] Therefore, the target processing result can be updated in response to the user's modification process to bring the target processing result more in line with the user's expectations.

[0131] Furthermore, the updated target processing result can more accurately reflect the processing results expected by the user for the user task. Therefore, updating the user record based on the user task and the updated target processing result can align the user record more closely with the user's understanding of the knowledge structure and their usage habits and preferences.

[0132] Another embodiment of the present disclosure also provides a further data processing method that can be executed by a processor of an electronic device. In some embodiments, the electronic device can be an edge device, for example, a server in a local network. The electronic device can also be an end device with data processing capabilities, for example, a laptop, a tablet, a desktop computer, etc.

[0133] As in Fig. As shown in Figure 2, in some embodiments the data processing procedure comprises steps S201 to S203.

[0134] In S201, a semantic analysis is performed on the user's second interaction information to obtain at least some of the first data, where the second interaction information represents cross-device multimodal interaction information that is related to the user.

[0135] In S202, based on at least one piece of initial data, at least one prediction task corresponding to that piece of initial data is predicted, and the at least one prediction task is stored in association with corresponding initial data.

[0136] In S203, clustering processing is performed on at least one piece of initial data to obtain the user's data set.

[0137] The second interaction information can correspond to the historical interaction information in the embodiments above, and steps S201 to S203 can correspond to steps S105 to S107 above. Therefore, for the specific implementation of steps S201 to S203, reference can be made to the detailed description of steps S105 to S107.

[0138] In the above embodiments, the electronic device implementing the above data processing method can uniformly acquire and organize the cross-device multimodal information of the user and organize and manage the information with the task as the center, thereby realizing the unified management of user data and improving the management efficiency of user data.

[0139] In some embodiments, the data processing method may further comprise at least one of steps S204 or S205.

[0140] In S204, the data set is updated based on third-party, cross-device, multimodal interaction information related to the user.

[0141] After determining the dataset based on the second interaction information, the third, cross-device, multimodal interaction information related to the user can be collected, and a semantic analysis can be performed on this third interaction information. The user's dataset can then be updated based on the results of this semantic analysis.

[0142] In some embodiments, a semantic analysis can be performed on the third interaction information to obtain at least one piece of second data. Then, based on the at least one piece of first data and the at least one piece of second data stored in the dataset, at least one prediction task corresponding to the at least one piece of first data can be re-predicted, and at least one prediction task corresponding to the at least one piece of second data can be predicted. Then, the first data and the at least one prediction task corresponding to the first data can be matched and stored, and the second data and the at least one prediction task corresponding to the second data can be matched and stored to obtain an updated dataset.

[0143] In some embodiments, clustering processing can be performed on the at least one piece of first data and the at least one piece of second data; that is, structural processing can be performed on the updated data set.

[0144] Therefore, by updating the user's data set based on the third interaction information obtained after the second interaction information, the effect of real-time or periodic monitoring of newly generated cross-device multimodal user data and dynamic updating of the data set can be achieved.

[0145] In S205, the record is updated in response to a user update operation on the record.

[0146] The user's output data set can be displayed to allow the user to update at least part of the data in the data set or the mapping relationship between the data.

[0147] In some implementations, the output data set can be displayed in the form of a knowledge tree. For example, the multiple pieces of initial data in the data set can be organized in the form of a tree diagram; and the multiple pieces of initial data, represented by vectorization, can be visualized and output.

[0148] In some embodiments, the update process may include a user-performed deletion, insertion, replacement, or similar operation of at least some of the data in the dataset or the mapping relationship between the data.

[0149] Therefore, in response to the user's update process, at least some of the data in the dataset or the mapping relationship between the data can be updated, so that the user's dataset can be more in line with the user's behavioral habits or knowledge structure.

[0150] The following is related to Fig. 3 and assuming the use of an agent to implement the data processing procedure provided by the present disclosure, the data processing procedure provided by the present disclosure is described in detail as an example. As in Fig. As shown in 3, the embodiment comprises steps S301 to S309.

[0151] In S301, application interaction information is obtained, and then S306 is executed.

[0152] The agent can obtain chat content information from the user and the interactive party via the social application on the user's laptop.

[0153] In S302, user input information is obtained, and then S306 is executed.

[0154] The intelligent agent can obtain information that is entered by the user into the interactive interface provided by the agent.

[0155] In S303 a scene image is captured and then S305 is executed.

[0156] The agent can take a screenshot of a currently running interface of the laptop and use the screenshot image as the scene image.

[0157] In S304, user interaction habits are established, and then S305 is executed.

[0158] The agent can read a user information file related to the user in order to obtain historical interaction habits of the user.

[0159] In S305, context information is determined, and then S306 is executed.

[0160] The agent can analyze the captured scene image and user interaction habits to determine contextual information related to user input information and application interaction information, such as the current task scene.

[0161] In S306 the user task is determined and then S307 is executed.

[0162] The agent can determine the exact user task based on user input information, application interaction information, and context information.

[0163] In S307, a query is performed on the data set to obtain the query results, and then S308 is executed.

[0164] Based on the specific user task, a retrieval can be performed on the user's data set to obtain the retrieval results, which include at least one piece of initial target data.

[0165] In S308, aggregation processing is performed on the retrieval results, and then S309 is executed.

[0166] Furthermore, the agent can extract information related to the user task from the retrieval results and perform aggregation processing on the extracted information to obtain the aggregation results.

[0167] In S309, the target processing result is generated based on the aggregation results.

[0168] After obtaining the aggregation results, the agent can use a large model to generate the target processing output based on the aggregation processing results, which can then be output to the user. The output images can, for example, be displayed.

[0169] For a completed task, an output method for the corresponding target processing result can be determined based on a task scenario that corresponds to the task. For example, if the task scenario corresponding to the user task is a meeting summary scenario, the target processing result can be displayed as a knowledge tree structure. The knowledge tree includes the summary information for each topic in the meeting and the corresponding source file link, and can also include image information related to the meeting. As in Fig. As shown in Figure 4, for a summary task of a conference on human-computer interaction technology (CHI technology), aggregation processing is carried out based on the retrieval results of the data set to generate a corresponding knowledge tree, and links or image information about source files are attached to some nodes of the knowledge tree (for example, Figure 410 and article link 420).

[0170] The above embodiment provides a flowchart for a user task. In some embodiments, when the agent performs one or more user tasks, the user can view the processing status of each task via a task pane provided by the agent. As in Fig. As shown in Figure 5, a task area 500 of the agent includes sub-areas 510, 520 and 530 for displaying the execution status of the respective task.

[0171] Each sub-area can contain corresponding task name information. Fig. For example, the task name in sub-area 510 is “Information about the technology provider to be visited” 511, the task name in sub-area 520 is “Send an email with a team-building photo to the administration department” 521 and the task name in sub-area 530 is “Buy a birthday cake” 531.

[0172] Each sub-area can use icons to indicate the relevant task scenario information. For example, in Fig. 5. Icon 512 in sub-area 510 indicates that the corresponding task scenario is a work scenario; icon 522 in sub-area 520 indicates that the corresponding task scenario is an email scenario; and icon 532 in sub-area 530 indicates that the corresponding task scenario is a social scenario.

[0173] Each sub-area can also include the current execution progress information for the task.

[0174] For example, there is in Fig. The progress bars in sub-area 510 (513) indicate the processing status of the current task. The different colors of the progress bars distinguish the completed portion from the incomplete portion, where the grid is shaded. The shaded portion indicates that the sub-task "Select Reference Information" has been completed, and the empty portion indicates the incomplete portion. Additionally, the percentage of the total task completed is shown as a percentage (here, 30%), and the elapsed execution time is displayed (here, 5 minutes).

[0175] Progress bar 523 is used in sub-area 520 to indicate the processing status of the current task, and the different colors of the progress bar are used to distinguish the completed portion from the incomplete portion. The grid shadow portion indicates that the "Image Selection" sub-task has been completed, the dotted portion indicates that the "Generate Email Draft" portion has been completed, and the empty portion indicates the incomplete portion. Additionally, the percentage of the total task completed is indicated by a percentage (here, 95%), and the elapsed time is shown (here, 5 minutes).

[0176] Progress bar 533 is used in sub-area 530 to indicate the processing status of the current task. The grid shadow indicates that the sub-task "Choose Cake" has been completed, and the empty area indicates the unfinished portion. Additionally, the percentage of the total task completed is shown as a percentage (here, 50%), and the elapsed time is displayed (here, 5 minutes).

[0177] Each sub-area can also contain agent-generated hint information, for example in Fig. 5 the information notes 514 in sub-area 510, the information notes 524 in sub-area 520 and the information notes 534 in sub-area 530.

[0178] Furthermore, task-related links or image information may also be specified in the sub-areas, for example the website link 515 and the image 516 in sub-area 510 in Fig. 5.

[0179] Combined with Fig. Section 6 describes in detail an embodiment of generating a user data record using the data processing method provided by this disclosure. As in Fig. As shown in 6, the embodiment comprises steps S601 to S605.

[0180] S601 retrieves historical user interaction information, and then S602 is executed.

[0181] Historical user interaction information can be user-related, device-spanning, multimodal information.

[0182] In S602, several pieces of initial data are determined, and then S603 is executed.

[0183] Historical user interaction information can be vectorized and semantically analyzed to determine the multiple pieces of initial data. For specific implementation procedures, refer to the detailed description above.

[0184] In S603, at least one prediction task is determined according to each piece of initial data, and then S604 is executed.

[0185] Based on the multiple pieces of initial data, at least one prediction task corresponding to each piece of initial data can be predicted, and then each piece of initial data can be stored in association with the corresponding at least one prediction task.

[0186] In S604, clustering processing is performed on the multiple pieces of initial data, and then S605 is executed.

[0187] Clustering processing can be performed using a large model on the multiple pieces of initial data to achieve structured storage of the multiple pieces of initial data.

[0188] S605 creates a multimodal file index.

[0189] Using the index generation function, based on the cluster processing results of the multiple pieces of initial data, the multimodal file index information can be generated for each initial data piece in order to perform a data retrieval using the multimodal file index.

[0190] The present disclosure also provides an electronic device for implementing the aforementioned data processing method. As in Fig. As shown in Figure 7, the electronic device 700 in one embodiment comprises a communication unit 710 and a processor. The processor includes a first processor 720 and a second processor 730.

[0191] The 710 communication unit can be used to obtain initial interaction information from the user.

[0192] The first 720 processor can be used to determine the user task based on the initial interaction information using a large model.

[0193] The second 730 processor can be used to determine, based on the user task, at least one piece of initial target data from the multiple pieces of initial data contained in a user's dataset. These multiple pieces of initial data can be determined based on the user's historical interaction information across devices and multimodality, and can include relevant prediction task information.

[0194] The first 720 processor can also be used to perform aggregation processing on at least one piece of initial target data using a large model, in order to obtain an aggregation processing result. Based on the aggregation processing result, the target processing result, which corresponds to the user task, can be generated.

[0195] In some embodiments, the electronic device 700 may also have a bus 740. The communication unit 710, the first processor 720, and the second processor 730 can implement data communication via the bus 740.

[0196] In some embodiments, the first interaction information may include: user input information determined based on a user input process, and / or cross-device multimodal application interaction information related to the user.

[0197] The first 720 processor can be used to perform:

[0198] Determining the user task based on user input information; and / or

[0199] Determining the user task based on application interaction information.

[0200] In some embodiments, the first 720 processor can also be used to: to determine at least one piece of second target data that is related to the first interaction information, based on the first interaction information from the multiple pieces of first data; and to determine the user task based on the initial interaction information and at least one piece of the second target data.

[0201] In some embodiments, the first 720 processor can be used to: to obtain the application scenario information, which corresponds to the initial interaction information, and the user's historical operation information; to determine the context information corresponding to the initial interaction information, based on the application scenario information and the historical process information; and to determine the user task based on the context information and the initial interaction information.

[0202] In some embodiments, the second 730 processor can be used to: Based on the user task, perform a retrieval from the multiple pieces of initial data to determine at least one piece of initial target data.

[0203] The retrieval may at least involve performing a retrieval on the multiple pieces of initial data based on the prediction task information according to the user task and each initial date.

[0204] In some embodiments, the first 720 processor can also be used to: to conduct a semantic analysis of the historical interaction information in order to obtain at least some initial data; based on the at least one piece of initial data, to predict the at least one prediction task that corresponds to the at least one piece of initial data and to assign the at least one prediction task to the corresponding at least one piece of initial data; and to perform clustering processing on at least one piece of initial data in order to obtain the user's data set.

[0205] In some embodiments, the first 720 processor can also be used to perform: Based on the user task and the target processing result, the user's data record is updated; in response to the user's modification of the target processing result, updating the target processing result; and / or, Based on the user task and the updated target processing result, the user's record is updated.

[0206] The present disclosure also provides an electronic device for carrying out the above data processing method. As in Fig. As shown in Figure 8, the electronic device 800 in one embodiment has a second communication unit 810 and a third processor 820.

[0207] The second communication unit 810 can be used to obtain the second interaction information of cross-device multimodality that is related to the user.

[0208] The third processor 820 can be used to: perform a semantic analysis on the user's second interaction information to obtain at least one piece of initial data; predict at least one prediction task based on the at least one piece of initial data that corresponds to the at least one piece of initial data; store the at least one prediction task in association with the corresponding at least one piece of initial data; and perform clustering processing on the at least one piece of initial data to obtain the user's data set.

[0209] In some embodiments, the electronic device 800 can also have a bus 830, and the second communication unit 810 and the third processor 820 can implement data communication via the bus 830.

[0210] In some embodiments, the third 820 processor can also be used to perform: based on the third interaction information of cross-device multimodality that is related to the user, updating the data set; and / or, in response to the user's update operation on the record, updating the record.

[0211] The present disclosure further provides an electronic device comprising one or more memories in which computer-readable instructions are stored, and one or more processors configured to execute the instructions in order to carry out a method according to the disclosure, for example one of the example methods described above.

[0212] The description of the above device embodiments is similar to the description of the above method embodiments and exhibits similar advantageous effects. In some embodiments, the functions or units included in the device provided in the exemplary embodiments of this disclosure can be used to perform the processes described in the above method embodiments. For technical details not disclosed in the device embodiments of this disclosure, reference may be made to the description of the method embodiments in this disclosure.

[0213] If the technical solution described in this disclosure involves personal information, the product using the technical solution described in this disclosure may clearly communicate the processing rules for personal information before processing personal information and obtain the voluntary consent of the data subject. If the technical solution described in this disclosure involves sensitive personal information, the product using the technical solution described in this disclosure may obtain separate consent from the data subject before processing sensitive personal information, thereby fulfilling the requirement of "explicit consent".For example, a device for collecting personal information, such as a camera, may have a clear and obvious sign indicating that a personal information collection zone has been entered and that personal information is now being collected. If the individual voluntarily enters the collection zone, this is considered consent to the collection of their personal information. In contrast, on a device for processing personal information, if the processing rules for personal information are communicated through obvious notices / information, personal authorization is obtained via pop-up information or by requesting the individual to upload their personal information.The rules for processing personal information may include information such as the processor of personal information, the purpose of the processing of personal information, the processing method and the type of personal information processed.

[0214] It should be noted that in the embodiments of the present disclosure, if the aforementioned data processing method is implemented in the form of a software function module and sold or used as an independent product, it may also be stored on a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present disclosure may be substantially or partially in the form of a software product that contributes to the prior art. The software product may be stored on a storage medium and may comprise several instructions to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present application.The aforementioned storage medium can include: a flash memory device, a portable hard disk, a read-only memory (ROM), a hard disk, or an optical disc, etc., capable of storing program code. Therefore, the embodiments of this disclosure are not limited to specific hardware, software, or firmware, or a combination thereof.

[0215] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, some or all of the steps in the above procedures can be implemented. The computer-readable storage medium can be volatile or non-volatile.

[0216] The present disclosure also provides a computer program comprising computer-readable code. When the computer-readable code is executed on a computer device, a processor of the computer device can execute the computer-readable code to implement some or all of the processes in the above procedures.

[0217] The present disclosure also provides a computer program product comprising a non-volatile, computer-readable storage medium on which a computer program is stored. When the computer program is read and executed by a computer, some or all of the steps in the above procedures can be implemented. The computer program product can be implemented in the form of hardware, software, or a combination thereof. In some embodiments, the computer program product can be specifically embodied as a computer storage medium. In some other embodiments, the computer program product can be specifically embodied as a software product, for example, as a software development kit (SDK) or the like.

[0218] It should be noted here that the description of the individual embodiments above tends to emphasize the differences between the embodiments, and that similar or identical aspects may be interrelated. The description of the above device, storage medium, computer program, and computer program product embodiments is similar to the description of the above method embodiments and exhibits similar advantageous effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this disclosure, reference may be made to the description of the method embodiments of this disclosure.

[0219] It is understood that the phrase "one embodiment," which appears in several places in the description, means that the specific features, structures, or properties relating to the embodiments are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" does not necessarily refer to the same embodiment at different places in the description. Furthermore, the specific features, structures, or properties may be combined in any suitable way in one or more embodiments.

[0220] It is also understood that the numerical values ​​of the designations of the aforementioned steps / processes in the various embodiments of this disclosure do not indicate any order of execution. The order of execution of the steps / processes is determined by their function and internal logic and does not represent any restriction on the implementation process of the embodiments of this disclosure. The numerical designations in the embodiments of this disclosure serve only for descriptive purposes and do not represent the advantages or disadvantages of any embodiment.

[0221] It is noted that in this disclosure, the terms "comprise," "include," and all other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or device comprising a number of elements may include not only those elements but also other elements not explicitly listed or elements inherent in such a process, method, article, or device. In the absence of further limitations, an element defined by the expression "comprising a..." does not preclude the presence of other identical elements in the process, method, article, or device comprising that element.

[0222] It is understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is only a logical functional division. In actual implementation, other division methods can be used; for example, several units or components can be combined or integrated into another system, or certain features can be ignored or omitted. Furthermore, the coupling, direct coupling, or communication link between the components shown or discussed can be achieved through interfaces, and the indirect coupling or communication link of the device or unit can be electrical, mechanical, or otherwise.

[0223] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. They may be located in a single location or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the solution of the task revealed in this document.

[0224] Furthermore, all functional units in the embodiments of the present application can be integrated into a single processor, or each unit can be used as a separate unit, or two or more units can be integrated into one unit; the above integrated unit can be implemented in the form of hardware or in the form of functional units consisting of software plus hardware.

[0225] A person skilled in the art understands that all or some of the steps for implementing the above method implementation forms can be carried out by hardware associated with program instructions, and that the above program can be stored on a computer-readable storage medium. When the program is executed, the steps of the above method implementation forms are carried out; and the above storage medium includes: a portable storage device, read-only memory (ROM), a storage disk or optical disk, and other media capable of storing program code.

[0226] If the integrated unit described above is implemented as a functional software module and sold or used as an independent product, it can alternatively be stored on a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored on a storage medium and comprises several instructions to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of this application. The aforementioned storage medium includes various media on which program code can be stored, for example, portable storage devices, ROMs, magnetic disks, or optical disks.

[0227] The foregoing describes in detail several embodiments of the present disclosure; however, the present disclosure is not limited to these specific embodiments. Based on the concept of the present disclosure, the person skilled in the art can make various variations and modifications, and these variations and modifications are within the scope of the present disclosure. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] CH 202411367772.9

[0001]

Claims

[1] Data processing procedures, comprehensive: Determining a user task based on a user's interaction information; Determine, based on the user task, at least one piece of target data from a plurality of data pieces contained in a user dataset, wherein the plurality of data pieces is determined based on historical cross-device multimodal interaction information related to the user and includes prediction task information; Performing aggregation processing on at least one piece of target data to obtain an aggregation processing result; and Generating a target processing result that matches the user task, based on the aggregation processing result. [2] Method according to claim 1, wherein: The interaction information includes user input information determined based on a user input process; and Determining the user task based on the user's interaction information includes determining the user task based on user input information. [3] Method according to claim 1, wherein: The interaction information includes cross-device, multimodal application interaction information that is related to the user; and Determining the user task based on the user's interaction information includes determining the user task based on the application interaction information. [4] Method according to claim 1, wherein: that at least one piece of target data is at least one piece of initial target data; and Determining the user task based on the user's interaction information includes: Determine at least one piece of secondary target data related to the interaction information from the majority of data pieces based on the interaction information; and Determining the user task based on the interaction information and at least one piece of secondary target data. [5] Method according to claim 1, wherein determining the user task based on the user's interaction information comprises: Obtaining application scenario information that corresponds to the interaction information and historical operation information of the user; Determining contextual information that corresponds to the interaction information, based on the application scenario information and the historical process information; and Determining the user task based on context information and interaction information. [6] Method according to claim 1, wherein determining the at least one piece of target data comprises: Perform, based on the user task, a retrieval from the plurality of data pieces based on the user task and the prediction task information according to each data piece of the plurality of data pieces, in order to determine the at least one piece of target data. [7] The method of claim 1, which further comprises, before determining the at least one piece of target data: Performing a semantic analysis on the historical interaction information to obtain at least one piece of data; Predictions based on the at least one piece of data, at least one prediction task corresponding to the at least one piece of data, and storing the at least one prediction task in association with corresponding data; and Perform clustering processing on at least one piece of data to obtain the user's data set. [8] Method according to claim 1, further comprising: Updating the data set based on the user task and the target processing result. [9] Method according to claim 1, further comprising: in response to a modification operation on the target processing result, updating the target processing result to obtain an updated target processing result, and updating the record based on the user task and the updated target processing result. [10] Non-volatile computer-readable storage medium on which instructions are stored which, when executed by a processor, cause an electronic device comprising the processor to perform the method according to claim 1. [11] Electronic device comprising: a memory in which instructions are stored; and a processor configured to execute the instructions to: to determine a user task based on a user's interaction information; based on the user task, to determine at least one piece of target data from a plurality of data pieces contained in a dataset of the user, wherein the plurality of data pieces is determined based on historical cross-device multimodal interaction information related to the user and includes prediction task information; to perform aggregation processing on at least one piece of target data in order to obtain an aggregation processing result; and to generate a target processing result that corresponds to the user task, based on the aggregation processing result. [12] Electronic device according to claim 11, wherein: The interaction information includes user input information determined based on a user input process; and The processor is further configured to execute instructions to determine the user task based on the user's interaction information. [13] Electronic device according to claim 11, wherein: The interaction information includes cross-device, multimodal application interaction information that is related to the user; and The processor is further configured to execute instructions to determine the user task based on the application interaction information when determining the user task based on the user's interaction information. [14] Electronic device according to claim 11, wherein: that at least one piece of target data is at least one piece of initial target data; and The processor is further configured to execute instructions to assist in determining the user task based on the user's interaction information: to determine at least one piece of second target data related to the interaction information, based on the interaction information from the majority of data pieces; and to determine the user task based on the interaction information and at least one piece of second target data. [15] Electronic device according to claim 11, wherein the processor is further configured to execute the instructions to determine the user task based on the user's interaction information: To obtain application scenario information that corresponds to the interaction information, and historical operation information of the user; To determine contextual information that corresponds to the interaction information, based on the application scenario information and the historical process information; and to determine the user task based on the context information and the interaction information. [16] Data processing procedures, in full: Performing a semantic analysis on a user's interaction information to obtain at least one piece of data, wherein the interaction information represents cross-device multimodal interaction information related to the user; Predictions based on the at least one piece of data, at least one prediction task corresponding to the at least one piece of data, and storing the at least one prediction task in association with corresponding data; and Perform clustering processing on at least one piece of data to obtain a user data record. [17] The method of claim 16, further comprising: Updating the data set based on other cross-device multimodal interaction information related to the user. [18] The method of claim 16, further comprising: Updating the dataset in response to an update operation on the dataset. [19] Electronic device comprising: a memory in which instructions are stored; and a processor configured to execute the instructions to perform the method according to claim 16. [20] Non-volatile computer-readable storage medium on which instructions are stored which, when executed by a processor, cause an electronic device comprising the processor to perform the method according to claim 16.

Images