Data processing method and equipment

Abstract

The invention discloses a data processing method and device.The data processing method comprises the steps that in response to a trigger event, first multi-modal information collected by first electronic equipment within a first specified duration is obtained, and portrait information is obtained from second electronic equipment; the first specified time length represents a specified time length taking the trigger event as an end time point; the portrait information represents personalized feature information of the user and / or the enterprise; determining a user task corresponding to the trigger event based on the first multi-modal information and the portrait information, and sending the user task to the second electronic equipment; generating a target processing result corresponding to the user task based on task processing information received from the second electronic equipment; the task processing information is determined from a data set of the user and / or the enterprise based on the user task.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of computer technology, and in particular to a data processing method and device. Background Art

[0002] With the development of artificial intelligence (AI) technology, a variety of AI applications have been proposed in the field of computer technology to achieve corresponding AI functions. However, how to improve the intelligence and efficiency of AI applications in the process of interacting with users is still a problem that needs to be solved. Summary of the invention

[0003] In view of this, the present application at least provides a data processing method and device.

[0004] The technical solution of this application is implemented as follows:

[0005] In one aspect, the present application provides a data processing method applied to a first electronic device, the method comprising:

[0006] In response to a triggering event, first multimodal information collected by the first electronic device within a first specified time period is obtained, and portrait information is obtained from the second electronic device; the first specified time period represents a specified time period with the triggering event as an end time point; the portrait information represents personalized feature information of the user and / or the enterprise;

[0007] Determine a user task corresponding to the trigger event based on the first multimodal information and the portrait information, and send the user task to the second electronic device;

[0008] Based on the task processing information received from the second electronic device, a target processing result corresponding to the user task is generated; the task processing information is determined from a data set of the user and / or the enterprise based on the user task.

[0009] In some embodiments, the data processing method further includes:

[0010] Acquire user operation information collected by the first electronic device within a second specified time period; the second specified time period represents a specified time period with the acquisition action time of acquiring the user operation information as the end time point, and the second specified time period is not greater than the first specified time period;

[0011] Based on the user operation information, a trigger event is determined.

[0012] In some embodiments, determining a user task corresponding to the triggering event based on the first multimodal information and the portrait information includes:

[0013] Determine the user intention and the task scenario corresponding to the user intention based on the first multimodal information and the portrait information;

[0014] Determine user tasks based on user intent and task scenarios.

[0015] In some embodiments, generating a target processing result corresponding to the user task based on the task processing information received from the second electronic device includes:

[0016] Based on the task processing information, a target processing result corresponding to the task scenario is generated.

[0017] In some embodiments, the data processing method further includes:

[0018] Obtain user feedback on target processing results;

[0019] The feedback information is sent to the second electronic device so as to update the portrait information through the second electronic device.

[0020] On the other hand, the present application also provides a data processing method applied to a second electronic device, the method comprising:

[0021] Receiving a user task sent by a first electronic device; the user task is determined based on first multimodal information and portrait information; the first multimodal information is information collected by the first electronic device within a first specified time period; the portrait information represents personalized feature information of the user and / or the enterprise;

[0022] Determining task processing information corresponding to the user task from a data set of the user and / or the enterprise; the data set is determined based on historical cross-device multimodal interaction information related to the user and / or the enterprise;

[0023] The task processing information is sent to the first electronic device.

[0024] In some embodiments, determining task processing information corresponding to a user task from a user's data set includes:

[0025] Based on the user task, determining at least one target data from the data set;

[0026] Aggregation processing is performed on at least one target data to obtain task processing information.

[0027] In some embodiments, the data processing method further includes:

[0028] receiving user feedback information on a target processing result from the first electronic device; the target processing result is generated by the first electronic device based on the task processing information;

[0029] Based on the feedback information, update the portrait information.

[0030] On the other hand, the present application further provides a first electronic device, comprising a first processor and a first communication interface; wherein:

[0031] The first processor, in response to the triggering event, obtains first multimodal information collected by the first electronic device within a first specified time period, and obtains portrait information from the second electronic device; the first specified time period represents a specified time period with the triggering event as an end time point; based on the first multimodal information and the portrait information, determines a user task corresponding to the triggering event; the portrait information represents personalized feature information of the user and / or the enterprise;

[0032] A first communication interface, sending the user task to the second electronic device, and receiving task processing information from the second electronic device; the task processing information is determined from a data set of the user and / or the enterprise based on the user task;

[0033] The first processor also generates a target processing result corresponding to the user task based on the task processing information.

[0034] On the other hand, the present application also provides a second electronic device, including a second processor and a second communication interface; wherein,

[0035] The second communication interface receives a user task sent by the first electronic device; the user task is determined based on the first multimodal information and the portrait information; the first multimodal information is information collected by the first electronic device within a first specified time period; the portrait information represents personalized feature information of the user and / or the enterprise;

[0036] A second processor determines task processing information corresponding to the user task from a data set of the user and / or the enterprise; the data set is determined based on historical cross-device multimodal interaction information related to the user and / or the enterprise;

[0037] The second communication interface also sends the task processing information to the first electronic device

[0038] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0040] Figure 1 A schematic diagram of the implementation flow of a data processing method provided in this application;

[0041] Figure 2 A schematic diagram of the implementation flow of a data processing method provided in this application;

[0042] Figure 3A schematic diagram of an implementation flow of an embodiment of an interaction process between a first electronic device and a second electronic device according to a data processing method provided by the present application;

[0043] Figure 4 A schematic diagram of an application scenario of the data processing method provided by the present application;

[0044] Figure 5 A schematic diagram of an application scenario of the data processing method provided by the present application;

[0045] Figure 6 A schematic diagram of an application scenario of the data processing method provided by the present application;

[0046] Figure 7 A schematic diagram of an application scenario of the data processing method provided by the present application;

[0047] Figure 8 A schematic diagram of an application scenario of the data processing method provided by the present application;

[0048] Fig. 9 A schematic diagram of a hardware entity of a first electronic device provided in this application;

[0049] Fig.10 A hardware entity schematic diagram of a second electronic device provided in this application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0051] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0052] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0054] When a user uses an AI application to process a user task, first, the user inputs the specific task information into the interactive interface provided by the AI ​​application; then, the AI ​​application determines the AI ​​task (i.e., the user task) based on the information input by the user, and searches the relevant database based on the AI ​​task; then, based on the search results, the user task corresponding processing results are generated; finally, the processing results are fed back to the user.

[0055] As can be seen from the above, in the related art, AI applications need to determine user tasks based on information actively input by users. In this way, when users need to use a certain AI function, they need to start the corresponding AI application and perform information input. The operation steps are relatively complicated, which brings a bad user experience.

[0056] Based on this, the present application provides a data processing method applied to a first electronic device, which can be executed by a processor of the first electronic device. The first electronic device may refer to a server, a laptop, a tablet computer, a desktop computer, a smart TV, a set-top box, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device) and other devices with data processing capabilities.

[0057] In some embodiments, the data processing method provided by the present application can be implemented as a cross-platform, cross-application agent application. The agent can run on at least one device related to the user, for example, the user's mobile phone, laptop, tablet computer, desktop computer, and cloud device storing user data; at the same time, the agent has cross-application data reading capabilities to read and analyze application data of different applications running on electronic devices. Here, when the agent reads and analyzes application data of different applications running on electronic devices, it is necessary to obtain user authorization. In some embodiments, the agent can be implemented as a desktop assistant (Luncher) on a personal computer (PC) or a negative screen or widget on a mobile phone and other system-level services.

[0058] Figure 1 A schematic diagram of the implementation flow of a data processing method provided in this application, such as Figure 1 As shown, the method includes the following steps S101 to S103:

[0059] Step S101, in response to a triggering event, obtaining first multimodal information collected by the first electronic device within a first specified time period, and obtaining portrait information from a second electronic device; the first specified time period represents a specified time period with the triggering event as an end time point; the portrait information represents personalized feature information of the user and / or enterprise.

[0060] Here, the trigger event is pre-set to start detecting an AI task related to the user of the first electronic device based on the trigger event.

[0061] In some embodiments, the trigger event may be a system default trigger event, or may be set by the user according to personal usage habits.

[0062] The first specified duration represents a specified duration with the triggering event as the end time point, that is, after the triggering event is determined, the specified duration is traced back with the triggering event as the end time point, so as to determine the first specified duration.

[0063] In some embodiments, the duration information of the first specified duration may be a system default or a user setting.

[0064] Multimodal information refers to various types of information such as text information, image information, audio information, video information, etc. collected by the first electronic device from multiple applications running on it.

[0065] Since using an electronic device to collect and store multimodal information related to a user involves the user's personal privacy and security, in some embodiments, with the user's authorization, the first electronic device collects and stores multimodal information related to the user within a specified period of time.

[0066] In some embodiments, for stored multimodal information related to the user, the first electronic device automatically overwrites the information from a specified time period (for example, 30 seconds) ago with the most recently collected multimodal information, so as to automatically erase the previously collected multimodal information and protect the user's privacy.

[0067] In this way, the first multimodal information is the multimodal information collected by the first electronic device within a specified time period with the triggering event as the end time point.

[0068] Portrait information represents personalized characteristic information of the user and / or the enterprise. In some embodiments, the personalized characteristics of the user may include the user's gender, age, occupation, educational background, skill level, personal hobbies, etc.; the personalized characteristics of the enterprise refer to the personalized characteristics of the group to which the user or the first electronic device belongs, for example, the establishment time, office location, background, service items, product status, profit status and / or other characteristic information related to the group.

[0069] In some embodiments, the portrait information is determined based on cross-device multimodal information related to the user and / or the enterprise. That is, the portrait information is determined by collecting multimodal information on at least one device related to the user and / or the enterprise.

[0070] The second electronic device refers to a device that interacts with the first electronic device to assist in processing the current triggering event.

[0071] In some embodiments, the second electronic device may be an end-side device with stronger data processing capabilities.

[0072] In some embodiments, the second electronic device may be a side device with strong data processing capability in a local area network or a device cluster, for example, a server in a local area network.

[0073] In some embodiments, the second electronic device may be a cloud device.

[0074] In some embodiments, the first electronic device and the second electronic device are the same device or different devices.

[0075] In some embodiments, the second electronic device is a device for storing portrait information of a user and / or an enterprise.

[0076] In some embodiments, the second electronic device receives multimodal information from at least one device related to the user and / or the enterprise, and generates and stores portrait information of the user and / or the enterprise based on the multimodal information.

[0077] Step S102: determining a user task corresponding to the triggering event based on the first multimodal information and the portrait information, and sending the user task to the second electronic device.

[0078] Here, based on the portrait information obtained from the second electronic device, the first multimodal information collected by the first electronic device within the first specified time period is analyzed to determine the task information contained in the first multimodal information; based on the determined task information, a user task is generated and sent to the second electronic device.

[0079] In some embodiments, the user task is query information or knowledge request information for a data set in the second electronic device, so as to determine data related to the user task from the data set based on the query information or knowledge request information.

[0080] In some embodiments, the user task is text information corresponding to the task information, or is low-dimensional feature information corresponding to the semantics of the task information.

[0081] Step S103, generating a target processing result corresponding to the user task based on the task processing information received from the second electronic device; the task processing information is determined from the data set of the user and / or enterprise based on the user task.

[0082] Here, the data set of users and / or enterprises refers to a data set established after analyzing and processing cross-device multimodal information related to the users and / or enterprises.

[0083] In some embodiments, the data set may be a vector database obtained by vectorizing and semantically analyzing cross-device multimodal information related to a user and / or an enterprise.

[0084] In some embodiments, the data set may be a personal knowledge base (PKB) or an enterprise knowledge base of a user and / or an enterprise, or a Retrieval-Augmented Generation (RAG) database used for RAG technology.

[0085] Here, the second electronic device searches the data set based on the user task to determine the task processing information corresponding to the user task.

[0086] The target processing result refers to the processing result presented to the user.

[0087] In some embodiments, the target processing result may include a picture generation result, a text generation result, a picture generation result, or a speech generation result, etc., presented to the user.

[0088] In some embodiments, based on the task processing result, the first device generates a corresponding target processing result using a large model. Here, the large model can be a multimodal large language model.

[0089] In the above-mentioned data processing method applied to the first electronic device provided by the present application, firstly, in response to a trigger event, first multimodal information collected by the first electronic device within a first specified time period is obtained, and portrait information is obtained from the second electronic device; then, based on the first multimodal information and the portrait information, a user task corresponding to the trigger event is determined, and the user task is sent to the second electronic device; finally, based on the task processing information received from the second electronic device, a target processing result corresponding to the trigger event is generated, wherein the task processing information is determined from a data set of users and / or enterprises based on the user task. In this way, on the one hand, in response to a trigger event, the user task corresponding to the trigger event is actively determined from the first multimodal information collected within the first specified time period, without the user having to enter the user task in the specified interactive interface, thereby improving the intelligence and initiative of identifying AI tasks; on the other hand, the first multimodal information is analyzed and understood in combination with the portrait information of the user and / or the enterprise to determine the user task, so that the determined user task is more in line with the personalized needs and user expectations of the user and / or the enterprise, thereby improving the intelligence of data processing; on the other hand, the task processing information corresponding to the specific user task is determined based on the data set of the user and / or the enterprise, and then the target processing result of the user task is generated according to the task processing information, thereby realizing cross-device and cross-modal search and application of personal and / or enterprise data, and making the target processing result of the user task closer to the knowledge structure of the user and / or the enterprise, which is convenient for the user to understand.

[0090] In some embodiments, the data processing method further includes the following steps S104 to S105:

[0091] Step S104, obtaining user operation information collected by the first electronic device within a second specified time period; the second specified time period represents a specified time period with the acquisition action time for obtaining the user operation information as the end time point, and the second specified time period is not greater than the first specified time period.

[0092] Here, the first electronic device cyclically detects the user operation information within the second specified time. The second specified time is a specified time with the acquisition action time of acquiring the user operation information as the end time point, and the time length of the second specified time is not greater than the first specified time. That is, the first electronic device detects the trigger event within the specified time with the current time as the end time point; after determining the trigger event, the first multimodal information within the first specified time is acquired with the time point of the occurrence of the trigger event as the end time point, so as to determine the user task based on the first multimodal information.

[0093] In some embodiments, user operation information refers to screen action information detected within the second specified time period. For example, the user operation information may include a click operation, a slide operation, a box selection operation of a user on a touch display screen of the first electronic device, or gesture information and voice information of a user collected by the first electronic device through a microphone or camera, or a click operation, a slide operation, a key operation, a box selection operation, etc. input by a user through a device such as a mouse, keyboard or touchpad.

[0094] Step S105: determining the trigger event based on the user operation information.

[0095] Here, the user operation information collected within the second time period is analyzed to determine whether there is a preset trigger event.

[0096] In some embodiments, the trigger event includes a user's triggering operation on a designated function key. Here, the designated function key is a function key pre-set based on the data processing method provided in this application. When the user uses the first electronic device, clicking the designated function key can trigger the first electronic device to perform AI task processing based on the first multimodal information.

[0097] In some embodiments, the triggering event includes a user's selection operation on at least one information displayed and output by the first electronic device. For example, in a reading scenario, a user selects part of the text or picture information in an electronic file by using a mouse or a touch screen, and the first electronic device can be triggered to perform AI task processing based on the first multimodal information in response to the selection operation; for another example, in a game scenario, a user makes a preset gesture or issues a voice command, and the first electronic device can be triggered to perform AI task processing based on the first multimodal information in response to the gesture or voice command.

[0098] In some embodiments, the triggering event includes the user reading at least one information displayed and output by the first electronic device for a time period greater than a time threshold. For example, in a reading scenario, based on the collected screen actions, it is determined that the user stays on a certain piece of information for a time period greater than a time threshold, and it is determined that the user may encounter comprehension problems when reading the information, and therefore, the first electronic device is triggered to perform AI task processing based on the context of the file.

[0099] In some embodiments, the triggering event includes a user repeatedly reading at least one piece of information displayed and output by the first electronic device. For example, in a reading scenario, based on the collected screen actions, it is determined that the user repeatedly reads a certain piece of information by scrolling the mouse or sliding the touch screen, and it is determined that the user may encounter comprehension problems when reading the information, and therefore, the first electronic device is triggered to perform AI task processing based on the context of the electronic file.

[0100] In the above embodiment, the trigger event is determined by collecting trigger events actively initiated by the user or the user's designated operations on the first electronic device, and the user task is actively determined based on the trigger event and task processing is performed, so that the first electronic device can actively determine the AI ​​task based on a variety of methods, thereby improving the intelligence of the first electronic device.

[0101] In some embodiments, the step S1021 of determining the user task corresponding to the triggering event based on the first multimodal information and the portrait information may be implemented as the following steps S1021 to S1022:

[0102] Step S1021: Determine the user intention and the task scenario corresponding to the user intention based on the first multimodal information and the portrait information.

[0103] Here, after acquiring the first multimodal information and the portrait information, the user intention is understood for the first multimodal information based on the portrait information, and the task scenario corresponding to the user intention is analyzed.

[0104] In some embodiments, understanding user intent may include, first, extracting information from the first multimodal information to obtain corresponding information details, for example, extracting text information, image information, voice information, etc.; then, using a large model to vectorize the extracted information details to obtain corresponding vectorized representations; thereafter, using the large model to perform semantic extraction on the vectorized representation information to obtain corresponding semantic information; finally, using the large model to understand the extracted semantic information based on the portrait information to determine the user intent.

[0105] For example, during a conversation between a user and other users, the first electronic device collects user conversation information using a microphone; in response to the user triggering an operation on a designated function key, the first electronic device obtains the stored voice information of the last 30 seconds (i.e., the first multimodal information); the first electronic device uses a large model to perform voice recognition on the voice information to obtain a corresponding vectorized representation; the first electronic device uses the large model to perform semantic extraction on the vectorized representation of the voice information to determine the semantic information "what was the sales volume last month" contained in the voice information; at the same time, the first electronic device determines product information, store information, etc. related to the enterprise based on the corporate portrait information of the user's enterprise; finally, the first electronic device determines, based on the above-mentioned semantic information and corporate portrait information, that the user intends to query the sales information of a specific product last month.

[0106] In some embodiments, a large model is used to identify image information, text information, voice information and other information in the first multimodal information to determine the task scenario corresponding to the user's intention.

[0107] In some embodiments, task scenarios may include meeting scenarios, reading scenarios, office scenarios, project planning scenarios, email scenarios, conversation scenarios, game scenarios, entertainment scenarios, and the like.

[0108] For example, when the first multimodal information is a screenshot corresponding to the first electronic device, the screenshot can be processed by the large model for target recognition or text recognition to determine the task scene corresponding to the current task. For example, when the screenshot contains an email icon, the task scene can be determined to be an email scene. For another example, when the first multimodal information is text information read from a document reading application, the task scene can be determined to be a reading scene.

[0109] Step S1022: determining the user task based on the user intention and the task scenario.

[0110] Here, the user intention and its corresponding task scenario are determined as the user task. In some embodiments, the user task represents the query information of the data set of the user and / or the enterprise. For example, the user task is a search vector for retrieving the data set.

[0111] In some embodiments, the generating of the target processing result corresponding to the user task based on the task processing information received from the second electronic device, that is, the above step S103, can be implemented as the following step S1031:

[0112] Step S1031: Generate a target processing result corresponding to the task scenario based on the task processing information.

[0113] Here, when generating the target processing result based on the task processing information received from the second electronic device, the target processing result consistent with the task scenario is generated in combination with the task scenario information in the user task.

[0114] In some embodiments, based on the task scenario, a user interaction (UI) interface component suitable for presenting information is called to generate a target processing result.

[0115] For example, when the task scenario corresponding to the user task is a meeting scenario, the target processing result includes at least a summary answer corresponding to the user task and a source file link corresponding to the summary answer, such as a web page link, picture link, folder link, etc. related to the answer.

[0116] For another example, when the task scenario corresponding to the user task is a reading scenario, the target processing result may include a knowledge tree generated based on the task processing result; wherein the knowledge tree includes knowledge points of various levels related to the user task and corresponding source file links.

[0117] In the above embodiment, based on the task processing information, the target processing result corresponding to the task scenario is generated, so that the display form of the target processing result can be more in line with the scenario requirements of the task scenario, thereby improving the intelligence of the data processing method and improving the user experience.

[0118] In some embodiments, the data processing method provided by the present application further includes the following steps S106 to S107:

[0119] Step S106: obtaining feedback information from the user regarding the target processing result.

[0120] Here, the first electronic device displays and outputs the target processing result so that the user can view and perform further operations on the target processing result.

[0121] In some embodiments, the feedback information may include evaluation information of the target processing result provided by the user, for example, whether the target processing result is useful; it may also include modification information of the target processing result provided by the user, for example, modification information of some knowledge points in the knowledge tree displayed as an output; and so on.

[0122] Step S107: sending the feedback information to the second electronic device so as to update the portrait information through the second electronic device.

[0123] Here, the user's feedback information on the target processing result is sent to the second electronic device, so that the second electronic device analyzes and understands the feedback information, and updates the result of the analysis and understanding to the portrait information.

[0124] For example, if the feedback information is that the user evaluates the target processing result as useful information, the weight of the information related to the target processing result can be increased in the portrait information; otherwise, the weight of the information related to the target processing result can be reduced. For another example, if the feedback information is the user's modification information on the target processing result, the information related to the user's knowledge structure can be adjusted in the portrait information.

[0125] In the above embodiment, the portrait information is updated based on the user's feedback information on the target processing results, so that the portrait information can more accurately describe the user's personalized characteristics, so that when processing the next trigger event, it can assist the large model to better understand the user's intentions and generate user tasks that accurately describe the user's requirements.

[0126] On the other hand, the present application also provides a data processing method applied to a second electronic device. Figure 2 As shown, the data processing method includes the following steps S201 to S203:

[0127] Step S201, receiving a user task sent by a first electronic device; the user task is determined based on first multimodal information and portrait information; the first multimodal information is information collected by the first electronic device within a first specified time period; the portrait information represents personalized feature information of the user and / or enterprise.

[0128] Here, the user tasks and the method for determining the user tasks are the same as the user tasks and the method for determining the user tasks described in the data processing method applied to the first electronic device above, so please refer to the detailed description above.

[0129] Step S202, determining task processing information corresponding to the user task from a data set of the user and / or enterprise; the data set is determined based on cross-device multimodal historical interaction information related to the user and / or enterprise.

[0130] Here, the data set of users and / or enterprises and the method for determining the same are the same as the data set and the method for determining the same described in the data processing method applied to the first electronic device above, so please refer to the detailed description above.

[0131] The second electronic device searches the data set based on the user task to determine task processing information.

[0132] Step S203: sending the task processing information to the first electronic device.

[0133] Here, the second electronic device sends the determined task processing information to the first electronic device through the communication network, so that the first electronic device generates a target processing result based on the task processing information.

[0134] In the data processing method applied to the second electronic device provided by the present application, the second electronic device determines the task processing information corresponding to the user task from the user and / or enterprise data set based on the user task received from the first electronic device, and sends the task processing information to the first electronic device. In this way, the user and / or enterprise data set is stored in the second device, and the data set is determined based on the cross-device multimodal information related to the user and / or enterprise, thereby realizing the unified storage and management of multimodal data scattered in multiple devices and multiple applications, and connecting the user data of all scenarios; at the same time, the task processing information corresponding to the specific user task is determined based on the data set, which realizes the cross-device and cross-modal search and application of personal or enterprise data, and makes the target processing result of the user task closer to the knowledge structure of the user and / or enterprise, which is convenient for users to understand.

[0135] In some embodiments, the determining of the data feedback information corresponding to the data request from the user's data set, that is, the above step S202, can be implemented as the following steps S2021 to S2022:

[0136] Step S2021: determining at least one target data from the data set based on the user task.

[0137] Here, based on the user task, the data set is retrieved to determine at least one target data from the multiple data included in the data set.

[0138] In some embodiments, when the data set is a vector database, searching the data set may include: searching the vector database based on semantic information contained in the user task to determine at least one target data that is closest to the semantic information of the user task from multiple vector data in the vector database.

[0139] Step S2022: performing aggregation processing on the at least one target data to obtain the data feedback information.

[0140] Here, performing aggregation processing on at least one target data refers to extracting information related to the user task from at least one target data and organizing the extracted information into a logical aggregation processing result.

[0141] In some embodiments, after determining at least one target data from the data set, aggregation processing is performed on the at least one target data using a large model.

[0142] In some embodiments, the data processing method further includes the following steps S204 to S205:

[0143] Step S204, receiving feedback information of the user on the target processing result from the first electronic device; the target processing result is generated by the first electronic device based on the task processing information;

[0144] Step S205: update the portrait information based on the feedback information.

[0145] Here, the target processing result and feedback information correspond to the target processing result and feedback information in the data processing method applied to the first electronic device above, so please refer to the detailed description above.

[0146] In some embodiments, after receiving user feedback information from the first electronic device, the second electronic device uses a large model to understand and analyze the feedback information, and updates the portrait information based on the understanding and analysis results.

[0147] Here, the specific implementation method of updating the portrait information based on the user feedback information corresponds to the specific implementation method of updating the portrait information in the data processing method applied to the first electronic device above. Therefore, please refer to the detailed description above.

[0148] In some embodiments, the data processing method applied to the second electronic device provided by the present disclosure corresponds to the data processing method applied to the first electronic device. For technical details not disclosed in the embodiment of the data processing method applied to the second electronic device, please refer to the description of the embodiment of the data processing method applied to the first electronic device of the present application for understanding.

[0149] Next, combine Figure 3 , an embodiment of the interaction process between the first electronic device 310 and the second electronic device 320 for executing the data processing method provided by the present application is described in detail. Figure 3 As shown, this embodiment includes the following steps S331 to S342:

[0150] Step S331, the first electronic device 310 cyclically collects and stores multimodal information within 30 seconds; cyclically detects screen action information within 20 seconds; then, executes step S332;

[0151] Step S332, in response to the triggering event, the first electronic device sends a portrait information request to the second electronic device; thereafter, executing step S333;

[0152] Here, the first electronic device detects the trigger operation of the user-specified function key based on the screen action information within 20 seconds, and takes the trigger operation as a trigger event; in response to the trigger event, the first electronic device sends a portrait information request to the second electronic device.

[0153] Step S333, the second electronic device sends the portrait information to the first electronic device; then, executing step S334;

[0154] Step S334, the first electronic device determines the user intention and task scenario using the large model based on the multimodal information and portrait information within 30 seconds; then, executing step S335;

[0155] Here, the first electronic device obtains multimodal information collected within 30 seconds, and inputs the multimodal information and portrait information received from the second electronic device into the large model; the large model processes the multimodal information and portrait information to obtain user intentions and task scenarios.

[0156] Step S335, the first electronic device determines the knowledge request information using the large model; then, executing step S336;

[0157] Here, the knowledge request information corresponds to the user task.

[0158] The large model determines knowledge request information to be sent to the second electronic device based on the determined user intention and task scenario.

[0159] Step S336, the first electronic device sends knowledge request information to the second electronic device; then, executing step S337;

[0160] Step S337, the second electronic device retrieves the data set based on the knowledge request information to obtain at least one target data; then, executing step S338;

[0161] Step S338, the second electronic device performs aggregation processing on at least one target data using the large model to obtain an aggregation result; then, step S339 is executed;

[0162] Step S339, the second electronic device sends the aggregation result to the first electronic device; then, executing step S340;

[0163] Step S340, the first electronic device generates and displays an output target processing result based on the aggregation result; thereafter, step S341 is executed;

[0164] Step S341, the first electronic device sends user feedback information on the target processing result to the second electronic device; then, executing step S342;

[0165] Step S342: The second electronic device updates the portrait information based on the feedback information.

[0166] Next, combine Figure 4 and Figure 5 , an application scenario of the data processing method provided by the present application is described, wherein the data processing method provided by the present application is implemented by an intelligent agent installed in a laptop computer 410.

[0167] like Figure 4 As shown, a voice message 420 is received in the user's laptop computer 410, and the agent records the voice message 420.

[0168] After the user plays the voice message 420, the user triggers a designated function key on the notebook computer.

[0169] In response to the user's triggering operation on the designated function key, the intelligent agent obtains the information collected within the last 30 seconds, including the voice information 420.

[0170] The intelligent agent uses the large model to perform speech recognition on the voice information 420 and determines that the voice information 420 includes the voice information "I asked Mandy to arrange a sharing meeting. I hope you can share your insights on human-computer interaction, focusing mainly on the latest large language model applications"; at the same time, the intelligent agent retrieves that the user has received a meeting invitation email 430 and a schedule 440 from Mandy.

[0171] Based on the voice message 430, the conference invitation email 430 and the schedule 440 received by the user, the intelligent agent determines that the user task to be performed is "generating a conference report related to human-computer interaction", and the application scenario is a conference report.

[0172] Based on the determined user tasks and application scenarios, the agent searches the user's personal knowledge base to obtain information such as pictures, web links, and papers related to human-computer interaction; uses the big model to aggregate the retrieved information to obtain a logical aggregation result; further, based on the aggregation result, uses the big model to generate a knowledge tree; displays the generated knowledge tree, such as Figure 5 shown.

[0173] Figure 5 In the example, a knowledge tree 510 related to human-computer interaction technology is shown on the user's laptop computer 410 , and links or picture information about source files are attached to some nodes of the knowledge tree 510 , for example, pictures 520 and paper links 530 .

[0174] Next, combine Figure 6 , Figure 7 and Figure 8 , another application scenario of the data processing method provided by the present application is described, wherein the data processing method provided by the present application is implemented by an intelligent agent installed in a laptop computer 610.

[0175] like Figure 6 As shown, in a conference scenario, application information of a laptop computer 610 is projected onto a display screen 620 .

[0176] The agent in the laptop computer 610 reads the projection information and uses a microphone to collect and record the voice information at the conference site. Here, the voice information includes the voice information 630 of the participants.

[0177] In response to the user triggering a designated function key on the laptop computer, the agent obtains information collected within the last 30 seconds, including voice information 630.

[0178] The intelligent agent uses a large model to perform speech recognition on the speech information 640 and determines that the speech information 630 includes the semantic information "What is the expected scale of China's artificial intelligence industry?".

[0179] Based on the voice information 630, the intelligent agent determines that the user task to be performed is "What is the expected scale of China's artificial intelligence industry?", and the application scenario is a conference scenario.

[0180] like Figure 7 As shown, based on the determined user tasks and application scenarios, the intelligent agent searches the enterprise database of the enterprise to obtain email information 710, picture information 720, web link information 730 and stored document information 740, etc. related to the expected scale of China's artificial intelligence industry; uses the big model to perform aggregation processing on the retrieved information to obtain a logical aggregation result; further, based on the aggregation result, uses the big model to generate a target processing result; displays the generated target processing result, such as Figure 8 shown.

[0181] exist Figure 8 In the example, the laptop computer 610 displays the detected user task in card 810, namely "Question: What is the expected scale of China's artificial intelligence industry?"; based on the processing result and in combination with the current application scenario, the output answer is displayed in the sub-card 811 within the card 810; at the same time, in cards 820 and 830, corresponding bar chart analysis and links to relevant information browsed by the user are respectively given; wherein, in card 820, a computer icon 821 is used to indicate that the information comes from the user's laptop computer; and in card 830, a mobile phone icon 831 is used to indicate that the information comes from the user's mobile phone.

[0182] On the other hand, the present application also provides a first electronic device, such as Fig. 9 As shown, the first electronic device 900 includes: a first processor 910 and a first communication interface 920; wherein,

[0183] The first processor 910, in response to a triggering event, obtains first multimodal information collected by the first electronic device 900 within a first specified time period, and obtains portrait information from a second electronic device; the first specified time period represents a specified time period with the triggering event as an end time point; based on the first multimodal information and the portrait information, determines a user task corresponding to the triggering event; the portrait information represents personalized feature information of a user and / or an enterprise;

[0184] The first communication interface 920 sends the user task to the second electronic device and receives task processing information from the second electronic device; the task processing information is determined from the user's data set based on the user task;

[0185] The first processor 910 further generates a target processing result corresponding to the user task based on the task processing information.

[0186] In some embodiments, the first processor 910 is further configured to:

[0187] Acquire user operation information collected by the first electronic device within a second specified time period; the second specified time period represents a specified time period in which the acquisition action time of acquiring the user operation information is an end time point, and the second specified time period is not greater than the first specified time period;

[0188] Based on the user operation information, the trigger event is determined.

[0189] In some embodiments, the first processor 910 is configured to:

[0190] Determining a user intention and a task scenario corresponding to the user intention based on the first multimodal information and the portrait information;

[0191] Based on the user intention and task scenario, the user task is determined.

[0192] In some embodiments, the first processor 910 generates a target processing result corresponding to the task scenario based on the task processing information.

[0193] In some embodiments, the first processor 910 obtains feedback information from the user regarding the target processing result;

[0194] The first communication interface 920 sends the feedback information to the second electronic device so as to update the portrait information through the second electronic device.

[0195] On the other hand, the present application also provides a second electronic device, such as Fig.10 The second electronic device 100 includes: a second processor 1010 and a second communication interface 1020; wherein,

[0196] The second communication interface 1020 receives a user task sent by the first electronic device; the user task is determined based on the first multimodal information and the portrait information; the first multimodal information is information collected by the first electronic device within a first specified time period; the portrait information represents personalized feature information of the user and / or the enterprise;

[0197] The second processor 1010 determines task processing information corresponding to the user task from the user's data set; the data set is determined based on historical cross-device multimodal interaction information related to the user;

[0198] The second communication interface 1020 further sends the task processing information to the first electronic device.

[0199] In some embodiments, the second processor 1010:

[0200] Based on the user task, determining at least one target data from the data set;

[0201] Aggregation processing is performed on the at least one target data to obtain the task processing information.

[0202] In some embodiments, the second communication interface 1020 receives feedback information of the user on the target processing result from the first electronic device; the target processing result is generated by the first electronic device based on the task processing information;

[0203] The second processor 1010 updates the portrait information based on the feedback information.

[0204] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or components included in the device provided in the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0205] It should be noted that in the embodiment of the present application, if the above-mentioned data processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0206] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0207] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0208] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0209] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0210] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0211] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0212] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0213] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0214] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0215] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0216] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0217] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0218] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A data processing method, applied to a first electronic device, comprising: In response to a triggering event, obtaining first multimodal information collected by the first electronic device within a first specified time period, and obtaining portrait information from a second electronic device; The first designated duration represents a designated duration with the trigger event as the end time point; The portrait information represents personalized characteristic information of the user and / or enterprise; Determining a user task corresponding to the trigger event based on the first multimodal information and the portrait information, and sending the user task to the second electronic device; generating a target processing result corresponding to the user task based on the task processing information received from the second electronic device; The task processing information is determined from a dataset of the user and / or the enterprise based on the user tasks.

2. The method according to claim 1, further comprising: Acquire user operation information collected by the first electronic device within a second specified time period; The second specified time duration represents a specified time duration for obtaining the user operation information from an acquisition action time ending at a time point, and the second specified time duration is not greater than the first specified time duration; Based on the user operation information, the trigger event is determined.

3. The method according to claim 1 or 2, wherein determining the user task corresponding to the triggering event based on the first multimodal information and the portrait information comprises: Determining a user intention and a task scenario corresponding to the user intention based on the first multimodal information and the portrait information; Based on the user intention and task scenario, the user task is determined.

4. The method according to claim 3, wherein generating a target processing result corresponding to the user task based on the task processing information received from the second electronic device comprises: Based on the task processing information, a target processing result corresponding to the task scenario is generated.

5. The method according to any one of claims 1 to 3, further comprising: Obtaining feedback information from the user regarding the target processing result; The feedback information is sent to the second electronic device so as to update the portrait information through the second electronic device.

6. A data processing method, applied to a second electronic device, comprising: Receiving a user task sent by a first electronic device; The user task is determined based on the first multimodal information and the portrait information; The first multimodal information is information collected by the first electronic device within a first specified time period; The portrait information represents personalized characteristic information of the user and / or enterprise; Determine task processing information corresponding to the user task from the data set of the user and / or enterprise; The data set is determined based on historical cross-device multimodal interaction information related to the user and / or enterprise; The task processing information is sent to the first electronic device.

7. The method according to claim 6, wherein determining the task processing information corresponding to the user task from the user's data set comprises: Based on the user task, determining at least one target data from the data set; Aggregation processing is performed on the at least one target data to obtain the task processing information.

8. The method according to claim 6 or 7, further comprising: receiving feedback information from the user regarding the target processing result from the first electronic device; The target processing result is generated by the first electronic device based on the task processing information; Based on the feedback information, the portrait information is updated.

9. A first electronic device, comprising a first processor and a first communication interface; wherein: The first processor, in response to a triggering event, acquires first multimodal information collected by the first electronic device within a first specified time period, and acquires portrait information from the second electronic device; The first designated duration represents a designated duration with the trigger event as the end time point; Determining a user task corresponding to the triggering event based on the first multimodal information and the portrait information; The portrait information represents personalized characteristic information of the user and / or enterprise; The first communication interface sends the user task to the second electronic device and receives task processing information from the second electronic device; The task processing information is determined from a data set of the user and / or the enterprise based on the user task; The first processor also generates a target processing result corresponding to the user task based on the task processing information.

10. A second electronic device, comprising a second processor and a second communication interface; wherein: The second communication interface receives a user task sent by the first electronic device; the user task is determined based on the first multimodal information and the portrait information; the first multimodal information is information collected by the first electronic device within a first specified time period; The portrait information represents personalized characteristic information of the user and / or enterprise; The second processor determines task processing information corresponding to the user task from a data set of the user and / or enterprise; The data set is determined based on historical cross-device multimodal interaction information related to the user and / or enterprise; The second communication interface also sends the task processing information to the first electronic device.

Images