Data processing method and electronic equipment

Abstract

The invention discloses a data processing method and electronic equipment, and the method comprises the steps: determining a user task based on the first interaction information of a user; based on the user task, determining at least one piece of first target data from multiple pieces of first data included in the data set of the user; wherein the first data is determined on the basis of cross-device multi-mode historical interaction information related to the user, and the first data has prediction task information; executing aggregation processing on the at least one piece of first target data to obtain an aggregation processing result; and generating a target processing result corresponding to the user task based on the aggregation processing result.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of computer technology, and in particular to a data processing method and electronic equipment. Background Art

[0002] In daily study and work, users need to refer to previously browsed information to assist in solving the current task. However, the information that users have browsed is often scattered in various electronic devices, and it is even more difficult to uniformly organize the information scattered in various electronic devices, resulting in users being unable to efficiently use this information. Therefore, how to effectively organize and use historical information related to users has become an urgent problem to be solved. Summary of the invention

[0003] In view of this, the present application at least provides a data processing method and an electronic device.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] On the one hand, the present application provides a data processing method, comprising:

[0006] Determining a user task based on the first interaction information of the user;

[0007] Based on the user task, at least one first target data is determined from a plurality of first data included in the user's data set; wherein the first data is determined based on historical cross-device multimodal interaction information related to the user, and the first data has prediction task information;

[0008] Performing aggregation processing on at least one first target data to obtain an aggregation processing result;

[0009] Based on the aggregated processing results, the target processing results corresponding to the user tasks are generated.

[0010] In some embodiments, the first interaction information includes at least one of the following: user input information determined based on a user input operation; cross-device multimodal application interaction information related to the user;

[0011] Determining a user task based on the first interaction information of the user includes at least one of the following:

[0012] Determine user tasks based on user input information;

[0013] Determine user tasks based on application interaction information.

[0014] In some embodiments, determining the user task based on the first interaction information of the user further includes:

[0015] Based on the first interaction information, determining at least one second target data related to the first interaction information from the plurality of first data;

[0016] A user task is determined based on the first interaction information and at least one second target data.

[0017] In some embodiments, determining the user task based on the first interaction information of the user includes:

[0018] Acquire application scenario information corresponding to the first interaction information and historical operation information of the user;

[0019] Determine, based on the application scenario information and the historical operation information, context information corresponding to the first interaction information;

[0020] A user task is determined based on the context information and the first interaction information.

[0021] In some embodiments, based on the user task, determining at least one first target data from a plurality of first data included in the user's data set includes:

[0022] Based on the user task, searching the plurality of first data to determine at least one first target data;

[0023] The searching at least includes searching the plurality of first data based on the predicted task information corresponding to the user task and each first data.

[0024] In some embodiments, before determining at least one first target data from a plurality of first data included in a user data set based on a user task, the data processing method further includes:

[0025] Performing semantic analysis on the historical interaction information to obtain at least one first data;

[0026] Based on at least one first data, predict at least one prediction task corresponding to the at least one first data, and store the at least one prediction task in association with the corresponding first data;

[0027] A clustering process is performed on the at least one first data to obtain a data set of the user.

[0028] In some embodiments, the data processing method further includes at least one of the following:

[0029] Update the user's data set based on the user's task and target processing results;

[0030] In response to a user's modification operation on the target processing result, the target processing result is updated; based on the user's task and the updated target processing result, the user's data set is updated.

[0031] On the other hand, the present application also provides a data processing method, comprising:

[0032] Performing semantic analysis on the second interaction information of the user to obtain at least one first data; the second interaction information represents cross-device multimodal interaction information related to the user;

[0033] Based on at least one first data, predict at least one prediction task corresponding to the at least one first data, and store the at least one prediction task in association with the corresponding first data;

[0034] A clustering process is performed on the at least one first data to obtain a data set of the user.

[0035] In some embodiments, the data processing method further includes at least one of the following:

[0036] Based on the third interaction information related to the user across multiple devices, updating the data set;

[0037] In response to a user's update operation on the data set, the data set is updated.

[0038] On the other hand, the present application also provides an electronic device, including a communication unit and a processing unit; the processing unit includes a first processing unit and a second processing unit; wherein,

[0039] A communication unit, configured to obtain first interaction information of a user;

[0040] A first processing unit, configured to determine a user task based on the first interaction information using a large model;

[0041] A second processing unit is used to determine at least one first target data from a plurality of first data included in a user data set based on a user task; wherein the plurality of first data are determined based on historical cross-device multimodal interaction information related to the user, and the first data has corresponding prediction task information;

[0042] The first processing unit is also used to use the large model to perform aggregation processing on at least one first target data to obtain an aggregation processing result; based on the aggregation processing result, generate a target processing result corresponding to the user task.

[0043] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0045] Figure 1 A schematic diagram of the implementation flow of a data processing method provided in this application;

[0046] Figure 2 A schematic diagram of the implementation flow of another data processing method provided by the present application;

[0047] Figure 3 A schematic diagram of an implementation flow of an embodiment of performing task processing according to the data processing method provided by the present application;

[0048] Figure 4 A schematic diagram of a target processing result generated according to the data processing method provided by the present application;

[0049] Figure 5 A schematic diagram of a task panel implemented according to the data processing method provided by the present application;

[0050] Figure 6 A schematic diagram of an implementation flow of an embodiment of generating a data set according to the data processing method provided by the present application;

[0051] Figure 7 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of the present application;

[0052] Figure 8 A schematic diagram of a hardware entity of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0054] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0055] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0057] In daily life, users are exposed to various types of information, which are scattered across different devices and applications, such as mobile devices, personal computers (PCs), cloud drives, news applications, social applications, and so on.

[0058] Since this information is scattered across different devices and applications, it is difficult to manage and use it in a unified manner, which makes it difficult for users to quickly find the information they need when they need it. For example, when completing a task, users need to switch back and forth between different devices and applications, spending a lot of time and energy to find and organize relevant information, which affects their ability to enter an efficient flow state.

[0059] In addition, even if users consciously organize and manage this information, they usually search for information through folders or tags. However, due to the complex network relationships between various types of information, the additional data generated during the organization and management process may far exceed the original hierarchical structure or tag system, making effective management extremely challenging.

[0060] In the process of retrieving historical information, users need to extract useful content from a large amount of information (e.g., reference materials, web links, chat records, etc.). What's worse, users may not even remember that they have seen certain information, and cannot come up with accurate keywords to retrieve them, so they cannot actively find the required content. In this case, the efficiency of information utilization is greatly reduced.

[0061] In order to solve the above technical problems, various solutions have been proposed in the related art, such as:

[0062] Synchronize information on various devices related to users to the cloud server to achieve unified storage of historical information;

[0063] Through multi-device synchronization tools, information on various devices related to the user can be synchronized to the same device to achieve cross-device data synchronization;

[0064] Information management and organization functions are provided through applications, thereby aggregating different types of information together for management.

[0065] However, the above solutions still require users to actively store, manage and search for information, which places high demands on users' information management capabilities (including time requirements and capability requirements), and it is difficult to effectively manage fragmented information. Therefore, for users, there is still the problem of low information utilization efficiency.

[0066] Based on this, the present application provides a data processing method, which can be executed by a processor of an electronic device. The electronic device may refer to a server, a laptop, a tablet computer, a desktop computer, a smart TV, a set-top box, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device) and other devices with data processing capabilities.

[0067] In some embodiments, the data processing method provided by the present application can be implemented as a cross-platform agent application. The agent can run on at least one device related to the user, such as the user's mobile phone, laptop, tablet computer, desktop computer, and cloud device storing user data; at the same time, the agent has cross-application data reading capability to read and analyze application data of different applications running on electronic devices. Here, when the agent reads and analyzes application data of different applications running on electronic devices, it needs to obtain user authorization.

[0068] Figure 1 A schematic diagram of the implementation flow of a data processing method provided in this application, such as Figure 1 As shown, the method includes the following steps S101 to S104:

[0069] Step S101: determining a user task based on the user's first interaction information.

[0070] Here, the first interaction information refers to information determined based on the user's current interaction action.

[0071] In some embodiments, the first interaction information may include information in multiple modes. For example, the first interaction information may be text information, image information, audio information, video information, user operation information, and the like.

[0072] In some embodiments, the first interaction information may be information obtained from different devices. For example, the first interaction information may be information obtained from an electronic device currently used by the user, or may be information obtained from a cloud device for storing interaction information related to the user.

[0073] Determining the user task based on the first interaction information refers to determining the task that the user expects to solve by performing text recognition, object recognition, semantic analysis, and other processing on the first interaction information.

[0074] In some embodiments, when an agent is used to implement the data processing method provided by the present application, the agent may use a multimodal large model to process the first interaction information to determine the user task. For example, when the first interaction information includes text information and image information, the multimodal large model is used to perform semantic extraction on the text information and the image information, and the corresponding user task is determined based on the extracted semantic information.

[0075] In some embodiments, the multimodal large model may include a multimodal large language model.

[0076] Step S102, based on the user task, determine at least one first target data from multiple first data included in the user's data set; wherein the first data is determined based on historical cross-device multimodal interaction information related to the user, and the first data has predicted task information.

[0077] Here, the cross-device multimodal historical interaction information refers to the historical interaction information of multiple modalities obtained from multiple devices.

[0078] In some embodiments, the multiple devices may include the user's electronic devices, such as the user's mobile phone, tablet computer, laptop computer, desktop computer, camera, mobile hard disk, etc.; they may also include electronic devices that store information related to the user, such as cloud disks, etc.

[0079] In some embodiments, the information in multiple modalities may be text information, image information, audio information, video information, information read from news applications or social applications, and the like.

[0080] In this way, after analyzing and processing the historical cross-device multimodal interaction information related to the user, a plurality of first data are obtained; based on the plurality of first data, a data set of the user is established.

[0081] In some embodiments, the data set may be a vector database obtained by vectorizing and semantically analyzing historical cross-device multimodal interaction information.

[0082] In some embodiments, the data set may be a user's Personal Knowledge Base (PKB), or a Retrieval-Augmented Generation (RAG) database used for a RAG technology.

[0083] The predicted task information refers to task information corresponding to the first data determined based on the page or user operation corresponding to the first data.

[0084] In some embodiments, the first data may have one or more prediction task information.

[0085] In the data set, the first data and at least one prediction task information related thereto are associated and stored, so that when performing data retrieval, the relevance of the first data and the retrieval information can be determined based on the prediction task information.

[0086] Thus, in some embodiments, determining at least one first target data from a data set based on a user task may include retrieving the user's data set based on semantic information contained in the user task to determine at least one first target data that is closest to the semantic information of the user task from multiple first data.

[0087] Step S103: performing aggregation processing on the at least one first target data to obtain an aggregation processing result.

[0088] Here, performing aggregation processing on at least one first target data refers to extracting information related to the user task from the at least one first target data and organizing the extracted information into a logical aggregation processing result.

[0089] In some embodiments, after determining at least one first target data from the user data set, the agent performs aggregation processing on the at least one first target data using a large model.

[0090] Step S104: generating a target processing result corresponding to the user task based on the aggregate processing result.

[0091] Here, the target processing result refers to the processing result presented to the user.

[0092] In some embodiments, the target processing result may include a picture generation result, a text generation result, a knowledge tree generation result related to the user's task, a speech generation result, etc. presented to the user.

[0093] In some embodiments, based on the aggregated processing results, the agent generates corresponding target processing results using a large model.

[0094] In some embodiments, the data processing method further includes determining a task scenario corresponding to the user task, and determining a display format of the target processing result based on the task scenario.

[0095] In some embodiments, based on the task scenario, a user interaction (UI) interface component suitable for presenting information is called to generate a target processing result.

[0096] For example, when the task scenario corresponding to the user task is a meeting scenario, the target processing result includes at least a summary answer corresponding to the user task and a source file link corresponding to the summary answer, such as a web page link, picture link, folder link, etc. related to the answer.

[0097] For another example, when the task scenario corresponding to the user task is a paper reading scenario, the target processing result may include a knowledge tree generated based on the search results; wherein the knowledge tree includes knowledge points of various levels related to the user task and corresponding source file links.

[0098] In the data processing method provided by the present application, based on the user's first interaction information, the user task is determined; based on the user task, at least one first target data is determined from multiple first data included in the user's data set, wherein the first data is determined based on the historical interaction information of the cross-device multimodality related to the user, and the first data has predicted task information; aggregation processing is performed on at least one first target data to obtain an aggregation processing result; based on the aggregation processing result, a target processing result corresponding to the user task is generated. In this way, on the one hand, it is realized to extract the user task from the user's interaction information and generate the corresponding target processing result, making the identification and processing process of the user task more intelligent; on the other hand, a user data set is generated based on the user's cross-device multimodal data to generate the target processing result corresponding to the user task, realizing the connection of the full-scene user data, and the cross-device cross-modal search and application of personal data; on the other hand, multiple first data in the user data set have corresponding predicted task information, realizing the task-centered organization and management of the user's cross-device multimodal data, thereby making the retrieval process of the user data set more accurate and efficient, and improving the utilization and management efficiency of user data.

[0099] In some embodiments, the first interaction information includes at least one of the following: user input information determined based on an input operation of the user; and cross-device multimodal application interaction information related to the user.

[0100] User input operation refers to the input operation performed by the user in a specified application; user input information refers to the information input by the user in the specified application.

[0101] Here, the designated application refers to an application used to implement the data processing method provided in this application, for example, an intelligent agent.

[0102] Cross-device multimodal application interaction information refers to multimodal information obtained from multiple devices related to the user. For example, chat content information between the user and the interacting party obtained from the social application running on the user's mobile phone; web page information browsed by the user obtained from the web application running on the user's tablet; user selection information on part of the content on the current browsing page obtained from the user's personal computer; and so on.

[0103] In some embodiments, the cross-device multimodal application interaction information related to the user is obtained through the above-mentioned agent. In this way, the agent is a system-level application running on the electronic device that can obtain cross-device and cross-application multimodal information.

[0104] In this way, the determining of the user task based on the first interaction information of the user, that is, the above step S101, can be implemented as at least one of the following steps S1011 and S1012:

[0105] Step S1011: determining the user task based on the user input information.

[0106] Here, semantic understanding of user input information is performed and the corresponding user tasks are determined.

[0107] In some embodiments, first, the multimodal big model is used to vectorize the user input information to obtain the corresponding vector representation; then, the big model is used to perform semantic extraction on the vector representation of the user input information to obtain the corresponding semantic information; finally, the big model determines the user task based on the extracted semantic information.

[0108] Step S1012: determining the user task based on the application interaction information.

[0109] Here, semantic understanding is performed on the obtained cross-device multimodal application interaction information, and the corresponding user tasks are determined.

[0110] In some embodiments, first, information extraction is performed on cross-device multimodal information to obtain corresponding information details, for example, text information, icon information, function key operation information, voice information, etc.; then, the extracted information details are vectorized using a large model to obtain corresponding vector representations; thereafter, the large model is used to perform semantic extraction on the vector representation corresponding to the application interaction information to obtain corresponding semantic information; finally, the large model is used to determine user tasks based on the extracted semantic information.

[0111] In the above embodiments, user tasks can be passively acquired based on user input information, and user tasks can also be actively determined by collecting and analyzing user-related cross-device multimodal application interaction information, that is, multiple user task determination methods are provided. In this way, during the user interaction process, response information for related tasks can be passively or actively provided to the user in response to user input or other application interaction information, thereby improving the flexibility and intelligence of human-computer interaction, and further improving the user experience.

[0112] In some embodiments, the determining of the user task based on the first interaction information of the user, that is, the above step S101, further includes the following steps S1013 and S1014:

[0113] Step S1013: Based on the first interaction information, determine at least one second target data related to the first interaction information from the multiple first data.

[0114] Here, according to the currently acquired first interaction information, the historical interaction information of the user is retrieved to determine at least one historical interaction information related to the first interaction information.

[0115] In some embodiments, first, the first interaction information is vectorized to obtain a corresponding vector representation; then, the vector representation corresponding to the first interaction information is semantically analyzed to obtain corresponding semantic information; finally, based on the semantic information corresponding to the first interaction information, the user's historical interaction information is retrieved, that is, a data set containing multiple first data is retrieved to obtain at least one second target data related to the first interaction information.

[0116] For example, when the first interactive information is "send team-building picture email to the management department" input by the user, after semantic analysis of the first interactive information, at least key semantic information such as "management department", "team-building picture", "send to management department", and "email" can be determined; then, based on the determined semantic information, the data set can be retrieved to determine the specific management department information related to the team building from multiple management departments, determine which team building the user is currently specifying from multiple team building information, and so on.

[0117] Step S1014: determining the user task based on the first interaction information and the at least one second target data.

[0118] After determining at least one second target data related to the first interaction information, the content of the first interaction information is further enriched in combination with the at least one second target data, thereby more accurately describing the user task.

[0119] For example, in the above case, based on the determined at least one second target data and the first interaction information, it can be determined that the user task to be described by the user is "sending an email with pictures of the most recent team building to the human resources department."

[0120] In this way, by combining the user's historical interaction information and further understanding the current first interaction information, the user's intention can be understood more accurately, and a user task that is more in line with the user's original intention can be generated, thereby providing a more accurate target processing result.

[0121] In some embodiments, the determining of the user task based on the first interaction information of the user, that is, the above step S101, can be implemented as the following steps S1015 to S1017:

[0122] Step S1015: Acquire application scenario information corresponding to the first interaction information and historical operation information of the user.

[0123] Here, the application scenario information refers to multimodal information collected from at least one device when acquiring the first interaction information.

[0124] In some embodiments, application scenario information is obtained by performing screenshot processing on a device related to the user, reading the user's interaction information in a specified application, and the like.

[0125] Historical operation information refers to information about user interaction habits and preferences that is determined based on user interaction information collected within a specified period of time in the past. For example, information about the frequency of user clicks on news applications; users spend more time and more frequently reading image information than text information; users tend to set alarms after 8 p.m. every night; and so on.

[0126] In some embodiments, historical operation information may be pre-stored in a user information library of the user.

[0127] Step S1016: Determine context information corresponding to the first interaction information based on the application scenario information and the historical operation information.

[0128] Here, context information refers to task background information related to the user task included in the first interaction information, such as task scenario information, task context information, etc. Context information can assist in understanding the user task.

[0129] In some embodiments, task scenario information may include email scenarios, meeting scenarios, reading scenarios, project planning scenarios, and the like.

[0130] In some embodiments, the acquired application scenario information and historical operation information are processed using a multimodal large model to determine context information corresponding to the first interaction information.

[0131] For example, when the application scenario information is a screenshot captured by an agent, the multimodal large model can be used to perform object recognition or text recognition on the screenshot to determine the task scenario corresponding to the current task. For example, when the screenshot contains an email icon, the task scenario can be determined to be an email scenario; for another example, when the application scenario information is text information in a document read by an agent, the task scenario can be determined to be a report reading scenario; for another example, by recognizing the text information in the document, the text context information corresponding to the part of the text selected by the user in the document can be determined; and so on.

[0132] For another example, the user's preference for reading picture information may be used as context information, so as to determine that the user's task at least includes a task of searching for related pictures.

[0133] Step S1017: determining the user task based on the context information and the first interaction information.

[0134] In some embodiments, the context information and the first interaction information are input into a large model to determine the user task using the large model.

[0135] Here, the large model can use context information to assist in understanding the user task contained in the first interaction information, so that the large model can understand the user's intention more accurately.

[0136] In the above embodiment, by capturing application scenario information related to the first interaction information and obtaining the user's historical habit information, the context information corresponding to the first interaction information is determined, and the context information is used to more accurately understand the user task that the user expects to establish through the first interaction information. On this basis, a target processing result that is more in line with the user's expectations is provided, thereby improving the intelligence of the agent in the human-computer interaction process.

[0137] In some embodiments, based on the user task, determining at least one first target data from a plurality of first data included in the user's data set, that is, the above step S102, can be implemented as the following step S1021:

[0138] Step S1021, based on the user task, perform retrieval on the multiple first data to determine the at least one first target data; wherein the retrieval at least includes performing retrieval on the multiple first data based on the user task and the predicted task information corresponding to each of the first data.

[0139] In some embodiments, the data set is in the form of a database. Thus, after determining the user task, the user task is sent to the data set to retrieve the plurality of first data using a database search function and determine at least one first target data.

[0140] In some embodiments, the retrieval process of the plurality of first data is implemented by clustering processing.

[0141] In some embodiments, the plurality of first data in the data set is a vectorized representation obtained by vectorizing the multimodal historical information related to the user across devices using a large model, and the user task is also a vectorized representation obtained by vectorizing. Therefore, the plurality of first data is retrieved, that is, at least one first data closest to the vector representation of the user task is determined from the plurality of first data.

[0142] In some embodiments, the plurality of first data in the data set are vector representations determined based on the user's cross-device multimodal information and have corresponding semantic information. Therefore, at least one first target data can be determined based on the semantic similarity between the plurality of first data and the user's task.

[0143] In some embodiments, during the process of performing a search on a user's data set, a global search is performed on the data set.

[0144] Here, the retrieval is performed based at least on the user task and the predicted task information corresponding to each first data, that is, the predicted task information corresponding to the first data is used as a retrieval condition to determine the similarity between each first data and the user task.

[0145] Since the predicted task information can represent at least one task related to the first data, performing a search on the first data based on the predicted task information can determine the relevance of multiple first data to the user task from the perspective of the task corresponding to the first data, thereby improving the retrieval accuracy and efficiency.

[0146] In some embodiments, before determining at least one first target data from a plurality of first data included in the user's data set based on the user task, that is, before the above step S102, the data processing further includes the following steps S105 to S107:

[0147] Step S105: performing semantic analysis on the historical interaction information to obtain at least one first data.

[0148] As mentioned above, historical interaction information refers to multimodal information obtained from multiple devices, such as image information, text information, application interaction information, etc. obtained from the user's mobile phone, laptop, desktop computer, tablet computer, and cloud disk used to store user data.

[0149] In some embodiments, shallow information extraction is performed on the historical interaction information to identify text information, UI elements, etc. in the historical interaction information.

[0150] After shallow information extraction of historical interaction information, vectorization processing is performed on the extracted information to obtain corresponding vectorized representation information.

[0151] In some embodiments, a large model is used to perform vectorization processing on the information obtained after shallow information extraction.

[0152] In some embodiments, a large model is used to perform semantic analysis on the vectorized representation information to map cross-device multimodal information to the same semantic space to obtain at least one first data having semantic features.

[0153] In some embodiments, performing semantic analysis on the vectorized representation information of the historical interaction information may include determining at least one entity information or at least one concept information from the vectorized representation information, and determining relationship information between entities or concepts, etc. For example, determining information such as appointment, place name, time, and the association relationship between these entity information.

[0154] Step S106: based on the at least one first data, predict at least one prediction task corresponding to the at least one first data, and store the at least one prediction task in association with the corresponding first data.

[0155] Here, task prediction is performed on the predicted at least one first data to determine at least one prediction task corresponding to the at least one first data.

[0156] In some embodiments, based on semantic information corresponding to at least one first data, a large model is used to determine task information related to a current page or user operation corresponding to the first data. For example, based on the user's current text interaction content and UI interface elements extracted from the page, the prediction task corresponding to the current text interaction content is determined.

[0157] In some embodiments, the first data corresponds to one or more prediction tasks.

[0158] For example, when the first data is entity information "calendar", in the meeting scenario, "calendar" is related to the meeting reservation task; in the family scenario, "calendar" is related to the task of setting reminders; in the email scenario, "calendar" is related to the task of sending emails at a scheduled time, etc. Therefore, tasks such as "meeting reservation", "setting reminders" and "sending emails at a scheduled time" can all be set as prediction tasks corresponding to "calendar".

[0159] At the same time, at least one prediction task is associated with the corresponding first data and stored, so that when performing a search on the data set, a more accurate search result can be obtained based on the at least one prediction task.

[0160] Step S107: performing clustering processing on the at least one first data to obtain a data set of the user.

[0161] Here, clustering processing is performed on at least one first data to establish association relationships between different historical interaction information, thereby achieving structured processing of multiple first data.

[0162] In some embodiments, clustering processing can be performed on at least one first data based on any clustering algorithm in the art, such as k-means clustering algorithm (k-means algorithm), mean shift clustering algorithm (Mean shift), density-based spatial clustering of applications with noise (DBSCAN), expectation maximization (EM) clustering algorithm based on Gaussian mixture model (GMM), etc.

[0163] In some embodiments, clustering processing is performed on at least one first data to obtain a clustering result with a tree structure.

[0164] In some embodiments, a corresponding multimodal file index is generated for a plurality of first data in a data set, so as to perform retrieval of the plurality of first data based on the multimodal file index.

[0165] In the above embodiment, by mapping the historical interaction information of cross-device multimodality to the same semantic space, obtaining at least one first data, predicting at least one prediction task corresponding to the at least one first data, and performing clustering processing on the at least one first data, a user data set is obtained, thereby realizing task-centered organization and management of the user's historical interaction information. Therefore, when performing subsequent data retrieval, multiple first data can be retrieved based on at least one prediction task information of the first data, thereby improving retrieval efficiency and accuracy.

[0166] In some embodiments, the data processing method further includes at least one of the following steps S108 to S109:

[0167] Step S108: updating the user's data set based on the user task and the target processing result.

[0168] Here, since the target processing result can characterize the adjustment of the user's current interaction information to the historical interaction information, updating the data set based on the user task and the target processing result can learn the user's understanding of the knowledge structure and usage habits, and use them for the generation of subsequent user tasks.

[0169] In some embodiments, based on the user task and target processing results, at least one entity information or at least one concept information, as well as the association relationship between the entity information or the concept information, are determined; then, the determined entity information or concept information and the association relationship information are stored in a data set.

[0170] In some embodiments, based on the correspondence between the user tasks and the target processing results, the association relationship between the plurality of first data stored in the data set is adjusted, or a new association relationship is established between the plurality of first data.

[0171] Step S109, in response to the user's modification operation on the target processing result, updating the target processing result; and updating the user's data set based on the user task and the updated target processing result.

[0172] Here, the target processing result corresponding to the user task is displayed and output to allow the user to modify the target processing result.

[0173] In some embodiments, the output target processing results can be displayed in a pop-up window or sidebar; the output target processing results can also be displayed in an interactive interface provided by an intelligent agent used to execute the data processing method; and so on.

[0174] For example, when the output target processing results are displayed in the form of a knowledge tree, the user is allowed to modify or adjust each node in the knowledge tree; when the output target processing results are displayed in the form of a picture, the user is allowed to modify information such as the brightness, color, display position of each object, etc. of the picture; and so on.

[0175] In this way, in response to the user's modification operation, the target processing result is updated to make the target processing result more consistent with the user's expectations.

[0176] At the same time, the updated target processing results can more accurately reflect the user's expected processing results for the user task. Therefore, updating the user dataset based on the user task and the updated target processing results can make the user dataset more in line with the user's understanding of the knowledge structure, usage habits and preferences.

[0177] On the other hand, the present application also provides a data processing method, which can be executed by a processor of an electronic device. In some embodiments, the electronic device can be a side device, such as a server set in a local area network; it can also be a terminal device with data processing capabilities, such as a laptop, a tablet computer, a desktop computer, etc.

[0178] like Figure 2 As shown, the data processing method includes the following steps S201 to S203:

[0179] Step S201, performing semantic analysis on second interaction information of a user to obtain at least one first data; the second interaction information represents cross-device multimodal interaction information related to the user;

[0180] Step S202, based on the at least one first data, predicting at least one prediction task corresponding to the at least one first data, and storing the at least one prediction task in association with the corresponding first data;

[0181] Step S203: performing clustering processing on the at least one first data to obtain a data set of the user.

[0182] Here, the second interaction information corresponds to the above historical interaction information, and steps S201 to S203 correspond to the above steps S105 to S107 respectively. Therefore, for the specific implementation of steps S201 to S203, please refer to the detailed description of the above steps S105 to S107.

[0183] In the above embodiment, the electronic device that implements the above data processing method uniformly acquires and organizes the user's cross-device multimodal information, and organizes and manages the information in a task-centric manner, thereby achieving unified management of user data and improving the efficiency of user data management.

[0184] In some embodiments, the data processing method further includes at least one of the following steps S204 and S205:

[0185] Step S204: updating the data set based on the third cross-device multimodal interaction information related to the user.

[0186] Here, after determining the data set based on the second interaction information, third cross-device multimodal interaction information related to the user is collected to perform semantic analysis on the third interaction information, and the user's data set is updated based on the semantic analysis result of the third interaction information.

[0187] In some embodiments, semantic analysis is performed on the third interaction information to obtain at least one second data; then, based on at least one first data and at least one second data stored in the data set, at least one prediction task corresponding to at least one first data is re-predicted, and at least one prediction task corresponding to at least one second data is predicted; then, the first data and its corresponding at least one prediction task are associated and stored, and the second data and its corresponding at least one prediction task are associated and stored to obtain an updated data set.

[0188] In some embodiments, clustering processing is performed on at least one first data and at least one second data, that is, structuring processing is performed on the updated data set.

[0189] In this way, by updating the user's data set through the third interaction information obtained after the second interaction information, it is possible to achieve the effect of real-time or periodic monitoring of the user's newly generated cross-device multimodal data and dynamically updating the data set.

[0190] Step S205: in response to the user's update operation on the data set, update the data set.

[0191] Here, the user's data set is displayed and output to allow the user to update at least part of the data in the data set or the association relationship between the data.

[0192] In some embodiments, the output data set is displayed in the form of a knowledge tree. For example, multiple first data in the data set are organized in the form of a tree diagram; and the multiple first data represented by vectorization are visualized and output.

[0193] In some embodiments, the update operation may include a user's deletion operation, insertion operation, replacement operation, etc. on at least part of the data in the data set or the association relationship between the data.

[0194] In this way, in response to the user's update operation, at least part of the data in the data set or the association relationship between the data is updated, so that the user's data set can be more in line with the user's knowledge structure or behavior habits.

[0195] Next, combine Figure 3 , taking the use of an intelligent agent to implement the data processing method provided by this application as an example, an embodiment of the data processing method provided by this application is described in detail. Figure 3 As shown, this embodiment includes the following steps S301 to S309:

[0196] Step S301, obtaining application interaction information; then, executing step S306;

[0197] Here, the agent obtains the chat content information between the user and the other party through the social application from the user's laptop.

[0198] Step S302, obtaining user input information; then, executing step S306;

[0199] Here, the agent obtains the information input by the user in the interactive interface provided by the agent.

[0200] Step S303, capturing a scene image; then, executing step S305;

[0201] Here, the agent takes a screenshot of the current running interface of the laptop and uses the screenshot image as the scene image.

[0202] Step S304, obtaining user interaction habits; then, executing step S305;

[0203] Here, the agent reads the user information file related to the user to obtain the user's historical interaction habits.

[0204] Step S305, determining context information; then, executing step S306;

[0205] Here, the agent analyzes the captured scene images and user interaction habits to determine contextual information related to user input information and application interaction information, such as the current task scenario.

[0206] Step S306, determine the user task; then, execute step S307;

[0207] Here, the agent determines the exact user task based on user input information, application interaction information, and context information.

[0208] Step S307, searching the data set and obtaining the search results; then, executing step S308;

[0209] Here, based on the determined user task, the user's data set is searched, and a search result including at least one first target data is obtained;

[0210] Step S308, performing aggregation processing on the search results; then, executing step S309;

[0211] Here, the agent further extracts information related to the user task from the retrieval results, and performs aggregation processing on the extracted information to obtain an aggregated result.

[0212] Step S309: Generate a target processing result based on the aggregation result.

[0213] Here, after obtaining the aggregated results, the agent uses the large model to generate a target processing result that can be output to the user based on the aggregated processing results, for example, an output picture can be displayed.

[0214] For completed tasks, the output method of the corresponding target processing results can be determined based on the task scenario corresponding to the task. For example, when the task scenario corresponding to the user task is a meeting summary scenario, the target processing result can be displayed and output as a knowledge tree structure; wherein the knowledge tree includes summary information of each topic in the meeting and the corresponding source file links, and can also include picture information related to the meeting. Figure 4 As shown, for the human-computer interaction technology (CHI) conference summary task, aggregation processing is performed based on the retrieval results of the data set to generate a corresponding knowledge tree, and links or picture information about the source files are attached to some nodes of the knowledge tree, for example, picture 410 and paper link 420.

[0215] The above embodiment provides a processing flow for a user task. When the agent executes one or more user tasks, the user can view the processing status of each task through the task panel provided by the agent. Figure 5 As shown, a task panel 500 of an agent includes:

[0216] Sub-panels 510, 520 and 530 for displaying the execution status of each task;

[0217] Each sub-panel includes corresponding task name information: task name “information about the technology supplier to be visited” 511 in sub-panel 510, task name “send team building photo email to management department” 521 in sub-panel 520, task name “buy birthday cake” 531 in sub-panel 530;

[0218] Each sub-panel uses an icon to indicate the corresponding task scenario information: in sub-panel 510, icon 512 is used to indicate that the corresponding task scenario is a work scenario; in sub-panel 520, icon 522 is used to indicate that the corresponding task scenario is an email scenario; in sub-panel 530, icon 532 is used to indicate that the corresponding task scenario is a social scenario;

[0219] Each sub-panel also includes the current execution progress information of the task:

[0220] The progress bar 513 is used in the sub-panel 510 to indicate the processing status of the current task, and the different colors of the progress bar are used to distinguish the completed part from the unfinished part, wherein the grid shaded part indicates the completed sub-task of "selecting reference information", and the blank part indicates the unfinished part; at the same time, the percentage of the completed part to the entire task is indicated by a percentage, i.e., 30%; and the time the task has been executed is displayed, i.e., 5 minutes;

[0221] The progress bar 523 is used in the sub-panel 520 to indicate the processing status of the current task, and the different colors of the progress bar are used to distinguish the completed part from the unfinished part, wherein the grid shaded part indicates that the sub-task of "picture selection" has been completed, the dotted part indicates that the part of "generating email draft" has been completed, and the blank part indicates the unfinished part; at the same time, the percentage of the completed part to the entire task is indicated by a percentage, i.e., 95%; and the time for the task to be executed is displayed, i.e., 5 minutes;

[0222] A progress bar 533 is used in the sub-panel 530 to indicate the processing status of the current task, wherein the shaded portion of the grid represents the completed sub-task of "selecting a cake", and the blank portion represents the unfinished portion; at the same time, a percentage is used to represent the percentage of the completed portion to the entire task, i.e., 50%; and the time the task has been executed is displayed, i.e., 5 minutes.

[0223] Each sub-panel also includes prompt information generated by the agent: for example, instruction information 514 in sub-panel 510, instruction information 524 in sub-panel 520, and instruction information 534 in sub-panel 530.

[0224] At the same time, task-related links or picture information may also be provided in the sub-panel, for example, the web page link 515 and picture 516 in the sub-panel 510 .

[0225] Next, combine Figure 6 , an embodiment of generating a user's data set using the data processing method provided by this application is described in detail. Figure 6 As shown, this embodiment includes the following steps S601 to S605:

[0226] Step S601, obtaining user historical interaction information; then, executing step S602;

[0227] Here, user historical interaction information refers to cross-device multimodal information related to the user.

[0228] Step S602, determining a plurality of first data; then, executing step S603;

[0229] Here, the user historical interaction information is vectorized and semantically analyzed to determine a plurality of first data. For specific implementations, please refer to the detailed description above.

[0230] Step S603, determining at least one prediction task corresponding to each first data; then, executing step S604;

[0231] Here, based on the plurality of first data, at least one prediction task corresponding to each first data is predicted, and each first data is associated with the corresponding at least one prediction task and stored.

[0232] Step S604, performing clustering processing on the plurality of first data; then, executing step S605;

[0233] Here, clustering processing is performed on the plurality of first data using a large model to achieve structured storage of the plurality of first data.

[0234] Step S605: Generate a multimodal file index.

[0235] Here, the index generation function is used to generate multimodal file index information corresponding to each first data based on the clustering processing results of the plurality of first data, so as to perform data retrieval using the multimodal file index.

[0236] Based on the above embodiments, the present application provides an electronic device to implement the above data processing method. Figure 7 As shown, the electronic device 700 includes a communication unit 710 and a processing unit; the processing unit includes a first processing unit 720 and a second processing unit 730; wherein,

[0237] The communication unit 710 is used to obtain first interaction information of the user;

[0238] The first processing unit 720 is used to determine the user task based on the first interaction information using the large model;

[0239] The second processing unit 730 is configured to determine at least one first target data from a plurality of first data included in the user data set based on the user task; wherein the plurality of first data are determined based on cross-device multimodal historical interaction information related to the user, and the first data has corresponding prediction task information;

[0240] The first processing unit 720 is further used to perform aggregation processing on the at least one first target data using a large model to obtain an aggregation processing result; and generate a target processing result corresponding to the user task based on the aggregation processing result.

[0241] In some embodiments, the electronic device 700 further includes a bus 740 ; the communication unit 710 , the first processing unit 720 , and the second processing unit 730 implement data communication via the bus 740 .

[0242] In some embodiments, the first interaction information includes at least one of the following: user input information determined based on the user's input operation; cross-device multimodal application interaction information related to the user;

[0243] The first processing unit 720 is configured to perform at least one of the following:

[0244] Determining the user task based on the user input information;

[0245] The user task is determined based on the application interaction information.

[0246] In some embodiments, the first processing unit 720 is further configured to:

[0247] Based on the first interaction information, determining at least one second target data related to the first interaction information from the plurality of first data;

[0248] The user task is determined based on the first interaction information and the at least one second target data.

[0249] In some embodiments, the first processing unit 720 is configured to:

[0250] Acquire application scenario information corresponding to the first interaction information and historical operation information of the user;

[0251] Determining context information corresponding to the first interaction information based on the application scenario information and the historical operation information;

[0252] The user task is determined based on the context information and the first interaction information.

[0253] In some embodiments, the second processing unit 730 is configured to:

[0254] Based on the user task, searching the plurality of first data to determine the at least one first target data;

[0255] The retrieval at least includes performing retrieval on the plurality of first data based on the predicted task information corresponding to the user task and each of the first data.

[0256] In some embodiments, the first processing unit 720 is further configured to:

[0257] Performing semantic analysis on the historical interaction information to obtain at least one first data;

[0258] Based on the at least one first data, predict at least one prediction task corresponding to the at least one first data, and store the at least one prediction task in association with the corresponding first data;

[0259] Clustering processing is performed on the at least one first data to obtain a data set of the user.

[0260] In some embodiments, the first processing unit 720 is further configured to perform at least one of the following:

[0261] Based on the user task and the target processing result, updating the user's data set;

[0262] In response to the user's modification operation on the target processing result, the target processing result is updated; based on the user task and the updated target processing result, the user's data set is updated.

[0263] On the other hand, based on the above embodiment, the present application also provides an electronic device for executing the above data processing method. Figure 8 As shown, the electronic device 800 includes: a second communication unit 810 and a third processor 820; wherein,

[0264] The second communication unit 810 is used to obtain the second cross-device multimodal interaction information related to the user;

[0265] The third processor 820 is used to perform semantic analysis on the user's second interaction information to obtain at least one first data; based on the at least one first data, predict at least one prediction task corresponding to the at least one first data, and store the at least one prediction task in association with the corresponding first data; perform clustering processing on the at least one first data to obtain the user's data set.

[0266] In some embodiments, the electronic device 800 further includes a bus 830 ; the second communication unit 810 and the third processing unit 820 implement data communication via the bus 830 .

[0267] In some embodiments, the third processor 820 is further configured to perform at least one of the following:

[0268] updating the data set based on third cross-device multimodal interaction information related to the user;

[0269] In response to the user's update operation on the data set, the data set is updated.

[0270] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or units included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0271] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0272] It should be noted that in the embodiment of the present application, if the above-mentioned data processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0273] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0274] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0275] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0276] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0277] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0278] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0279] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0280] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0281] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0282] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0283] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0284] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A data processing method, comprising: Determining a user task based on the first interaction information of the user; Based on the user task, determining at least one first target data from a plurality of first data included in the user's data set; wherein the first data is determined based on historical cross-device multimodal interaction information related to the user, and the first data has prediction task information; Performing aggregation processing on the at least one first target data to obtain an aggregation processing result; Based on the aggregated processing result, a target processing result corresponding to the user task is generated.

2. The method according to claim 1, wherein the first interaction information comprises at least one of the following: user input information determined based on the user's input operation; cross-device multimodal application interaction information related to the user; The determining of the user task based on the first interaction information of the user includes at least one of the following: Determining the user task based on the user input information; The user task is determined based on the application interaction information.

3. The method according to claim 2, wherein determining the user task based on the first interaction information of the user further comprises: Based on the first interaction information, determining at least one second target data related to the first interaction information from the plurality of first data; The user task is determined based on the first interaction information and the at least one second target data.

4. The method according to any one of claims 1 to 3, wherein determining the user task based on the first interaction information of the user comprises: Acquire application scenario information corresponding to the first interaction information and historical operation information of the user; Determining context information corresponding to the first interaction information based on the application scenario information and the historical operation information; The user task is determined based on the context information and the first interaction information.

5. The method according to any one of claims 1 to 3, wherein determining at least one first target data from a plurality of first data included in the user's data set based on the user task comprises: Based on the user task, searching the plurality of first data to determine the at least one first target data; The retrieval at least includes performing retrieval on the plurality of first data based on the predicted task information corresponding to the user task and each of the first data.

6. The method according to any one of claims 1 to 3, before determining at least one first target data from a plurality of first data included in the user's data set based on the user task, the method further comprises: Performing semantic analysis on the historical interaction information to obtain at least one first data; Based on the at least one first data, predict at least one prediction task corresponding to the at least one first data, and store the at least one prediction task in association with the corresponding first data; Clustering processing is performed on the at least one first data to obtain a data set of the user.

7. The method according to any one of claims 1 to 3, further comprising at least one of the following: Based on the user task and the target processing result, updating the user's data set; In response to the user's modification operation on the target processing result, the target processing result is updated; based on the user task and the updated target processing result, the user's data set is updated.

8. A data processing method, comprising: Performing semantic analysis on the second interaction information of the user to obtain at least one first data; The second interaction information represents cross-device multimodal interaction information related to the user; Based on the at least one first data, predict at least one prediction task corresponding to the at least one first data, and store the at least one prediction task in association with the corresponding first data; Clustering processing is performed on the at least one first data to obtain a data set of the user.

9. The method according to claim 8, further comprising at least one of the following: updating the data set based on third cross-device multimodal interaction information related to the user; In response to the user's update operation on the data set, the data set is updated.

10. An electronic device, comprising a communication unit and a processing unit; the processing unit comprises a first processing unit and a second processing unit; wherein, The communication unit is used to obtain first interaction information of the user; The first processing unit is used to determine the user task based on the first interaction information using the large model; The second processing unit is used to determine at least one first target data from a plurality of first data included in the data set of the user based on the user task; wherein the plurality of first data are determined based on the historical cross-device multimodal interaction information related to the user, and the first data have corresponding prediction task information; The first processing unit is further used to perform aggregation processing on the at least one first target data using a large model to obtain an aggregation processing result; and generate a target processing result corresponding to the user task based on the aggregation processing result.

Images