Operation processing method and device

Abstract

The invention discloses an operation processing method and device, and belongs to the technical field of man-machine interaction and artificial intelligence. The disclosed operation processing method comprises the following steps: responding to a detected position transfer event for a first object; identifying object information of the first object and application information of applications where the first object is located before and after position transfer by using a first intelligent agent; according to the object information of the first object and application information of applications where the first object is located before and after position transfer, determining a processing intention of the first object, and obtaining a processing intention determination result; determining target processing to be executed on the first object after position transfer by utilizing the first intelligent agent based on the processing intention determination result; and executing the target processing on the first object after position transfer by using a second intelligent agent to obtain a processing result of the first object.

Description

Technical Field

[0001] The present application belongs to the field of human-computer interaction and artificial intelligence technology, and in particular, to an operation processing method and device. Background Art

[0002] In the process of using electronic devices for learning or office work, there are often scenarios where users operate across applications or perform related operations in different locations of the same application. In the known technologies, these operation scenarios are not user-friendly and have problems such as cumbersome user operations, low efficiency, and high time consumption. For example, in the cross-application operation scenario, it takes time to adjust the format when copying a table from Excel to PPT. When copying and pasting in the same application, it is often necessary to adjust the scale, format, etc. after pasting objects such as text to the target location, which affects the user experience. Summary of the invention

[0003] To this end, this application discloses the following technical solutions:

[0004] An operation processing method, comprising:

[0005] In response to detecting a position transfer event of a first object, using a first agent to identify object information of the first object and application information of applications in which the first object is located before and after the position transfer;

[0006] Determine a processing intention for the first object according to the object information of the first object and the application information of the applications in which the first object is located before and after the position transfer, and obtain a processing intention determination result;

[0007] Determine the target processing to be performed on the first object after the position transfer by using the first agent based on the processing intention determination result;

[0008] The target processing is performed on the first object after the position transfer by using the second agent to obtain the processing result of the first object.

[0009] Optionally, the detecting of the position transfer event of the first object includes at least one of the following:

[0010] detecting a position transfer event of dragging the first object from a first position to a second position in the application;

[0011] A position transfer event of dragging an object in a first application to a second application is detected.

[0012] Optionally, the object information of the first object identified by the first agent and the application information of the applications in which the first object is located before and after the position transfer include:

[0013] obtaining an object screenshot obtained by taking a screenshot of the first object, and an area screenshot obtained by taking screenshots of the areas where the first object is located before and after the position transfer;

[0014] The object screenshot, the area screenshot and the first guidance information are input into a large language model to obtain object information of the first object and application information of applications in which the first object is located before and after position transfer, generated by the large language model based on the object screenshot and the area screenshot under the guidance of the first guidance information.

[0015] Optionally, determining the processing intention for the first object according to the object information of the first object and the application information of the applications in which the first object is located before and after the position transfer, includes at least one of the following:

[0016] Inputting object information of the first object, application information of the application in which the first object is located before and after the position transfer, and second guidance information into a large language model, to obtain a first processing intention for the first object generated by the large language model under the guidance of the second guidance information;

[0017] Determine, from the intent understanding data set, a second processing intent that matches the object information of the first object and the application information of the application in which the first object is located before and after the position transfer;

[0018] Among them, the processing intention for the first object includes at least one of the first processing intention and the second processing intention; the intention understanding data set includes at least one intention understanding data, each intention understanding data includes object information of the corresponding object, application information of the application in which the corresponding object is located before and after the corresponding object is transferred, and the processing intention for the corresponding object.

[0019] Optionally, the operation processing method further includes at least one of the following:

[0020] updating the intention understanding dataset based on the object information of the first object, the application information of the applications in which the first object is located before and after the position transfer, and the actual processing intention of the first object;

[0021] Based on the execution frequency information of different processing intentions of the object in the intention understanding data set, the weights of the different processing intentions of the object are determined.

[0022] Optionally, the using the first agent to determine the target processing to be performed on the first object after the position transfer based on the processing intention determination result includes:

[0023] Inputting the target intent and the third guidance information into the large language model, obtaining an operation sequence generated by the large language model under the guidance of the third guidance information to satisfy the target intent; the operation sequence is used to indicate one or more operations to be performed on the first object, and the target intent is an intent obtained based on the result of determining the processing intent;

[0024] The using the second agent to perform the target processing on the first object after the position transfer includes:

[0025] The large action model is used to perform the operations indicated by the operation sequence on the first object after the position is transferred.

[0026] Optionally, the process of obtaining the target intention includes:

[0027] If the processing intention determination result indicates that the processing intention for the first object is a single intention, the single intention is obtained or the first intention of the operating body input is obtained; the target intention is the single intention or the first intention;

[0028] If the processing intention determination result represents that the processing intention for the first object is multiple intentions, an intention selected from the multiple intentions is obtained, or a second intention input by the operating body is obtained; the target intention is the intention selected from the multiple intentions or the second intention.

[0029] Optionally, the operation processing method, before detecting the position transfer event of the object, further includes:

[0030] In response to detecting an event in which a target control is positioned within an application in a target mode, identifying a second object in an area corresponding to the positioned position, and controlling the second object to be in a target state based on the identification result;

[0031] The target state is a state for selecting at least a part of the structure of the second object; and the first object is at least a part of the structure of the second object selected from the second object.

[0032] Optionally, controlling the second object to be in a target state based on the recognition result includes:

[0033] If the recognition result indicates that the second object is text, controlling the second object to be in a first state for selecting text content,

[0034] If the recognition result indicates that the second object is an image, controlling the second object to be in a second state for taking a screenshot of the image content;

[0035] If the recognition result indicates that the second object is audio or video, the second object is controlled to be in a third state for selecting audio or video clips.

[0036] An operation processing device, comprising:

[0037] an identification module, configured to, in response to detecting a position transfer event of a first object, use a first agent to identify object information of the first object and application information of applications in which the first object is located before and after the position transfer;

[0038] A first determination module, configured to determine a processing intention for the first object according to the object information of the first object and the application information of the applications in which the first object is located before and after the position transfer, and obtain a processing intention determination result;

[0039] A second determination module is used to determine the target processing to be performed on the first object after the position transfer by using the first agent to determine the result based on the processing intention;

[0040] The processing module is used to use the second agent to perform the target processing on the first object after the position transfer, so as to obtain the processing result of the first object.

[0041] An electronic device, comprising:

[0042] Memory, used to store computer instruction sets;

[0043] A processor is used to implement the operation processing method provided in any one of the above items by executing a computer instruction set.

[0044] A storage medium carries one or more computer instruction sets, and when the one or more computer instruction sets are executed by an electronic device, the electronic device can implement the operation processing method as described in any one of the above. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0046] Figure 1 It is a flow chart of the operation processing method provided by this application;

[0047] Figure 2is a schematic diagram of a process of using a first agent to identify object information of a first object and application information of applications in which the first object is located before and after position transfer, as provided by the present application;

[0048] Figure 3 It is another flowchart diagram of the operation processing method provided by the present application;

[0049] Figure 4 This is an exemplary appearance of the AI ​​suspension ball provided in this application;

[0050] Figure 5 This is an exemplary implementation process of the second object performing state control provided by this application;

[0051] FIG6(a), FIG6(b) and FIG6(c) are examples of controlling the second object to be in a word-marking interaction state, a screenshot interaction state, and a segment selection interaction state, respectively, provided by the present application;

[0052] Figure 7 This is an example of intelligent operation processing based on position transfer of objects provided by this application;

[0053] FIG8( a ) is an example provided by the present application of dragging a picture in Office to the search input box of a browser;

[0054] FIG8( b ) is an example of performing an operation on the first object after the position is moved provided by the present application;

[0055] FIG8( c ) is an example of calling out and displaying an intention input interface provided by the present application;

[0056] FIG8( d ) is another example of operating the first object after the position is moved provided by the present application;

[0057] Fig. 9 It is a composition structure diagram of the operation processing device provided by this application;

[0058] Fig.10 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] The embodiments of the present application provide an operation processing method and device, which are used to solve the problems of cumbersome user operations, low efficiency, high time consumption, etc. in the known technology when users perform cross-application operations or related operations at different locations in the same application. The operation processing method provided by the embodiments of the present application can be applied to electronic devices in many general or special computing device environments or configurations, such as personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor devices, etc.

[0061] See also Figure 1 As shown in the method flow chart, the operation processing method provided in the embodiment of the present application may at least include the following steps 101 to 104, and these steps are described in detail below.

[0062] Step 101: In response to detecting a position transfer event of a first object, use a first agent to identify object information of the first object and application information of applications in which the first object is located before and after the position transfer.

[0063] The first object is an operable object in the application, which may be, but is not limited to, any type of operable content object such as text, symbols, images, audio and video in the application, or a part of these content objects, such as keywords, phrases, partial screenshots, video clips, etc.

[0064] In the embodiments of the present application, the application is referred to as the application for short, and the first application, the second application, etc. described below all refer to the application.

[0065] The position transfer event for the first object can be an event applied to the first object manually by the user or automatically by a machine based on actual needs to change the position of the first object. Specifically, it can be but is not limited to performing operations such as dragging, gestures and / or voice control on the first object to change its position, thereby generating a position transfer event for the first object to achieve position transfer of the first object.

[0066] The position transfer of the first object may be to transfer the first object to different positions in the same application (i.e., non-cross-application position transfer), or to transfer the first object between different applications (i.e., cross-application position transfer), and there is no limitation on this, and it may be determined according to actual application requirements. For example, drag a sentence in a word document from a text area to a table in the word document as table content, or drag a keyword / sentence in a word document to a search engine's search bar to search for information, or drag a sentence / paragraph in a word document to the body of an email to fill in the body of the email, etc.

[0067] Based on this, optionally, in this step, detecting a position transfer event for the first object can be implemented to include at least one of the following: detecting a position transfer event for transferring the first object from a first position in the application (such as dragging) to a second position, detecting a position transfer event for transferring an object in the first application (such as dragging) to a second application.

[0068] The first application and the second application are different applications.

[0069] The first agent may be an AI (Artificial Intelligence) model, and specifically may be a large model such as a Large Language Model (LLM). The embodiment of the present application mainly takes the first agent as a large language model as an example to illustrate the solution.

[0070] In response to detecting a position transfer event of the first object, this embodiment uses the first agent to identify object information of the first object and application information of the applications in which the first object is located before and after the position transfer.

[0071] The object information of the first object may include, but is not limited to, one or more of the object type of the first object, the content type of the content contained in the object, and the data volume of the object. The object type may include, but is not limited to, text, picture, audio and video, etc., and the content type of the object content may include, but is not limited to, the text content corresponding to the text content, the number or symbol type, etc., and the picture content corresponding to the person, scene, animal, etc.

[0072] The application information of the applications in which the first object is located before and after the position transfer is divided into two cases: non-cross-application position transfer and cross-application position transfer.

[0073] In the case of non-cross-application position transfer, the application information of the applications in which the first object is located before and after the position transfer may include, but is not limited to, the application type / name of the same application in which the first object is located before and after the position transfer (such as office, game, communication application types and specific application names, etc.), different positions in the same application before and after the position transfer (such as text area position, table area position, email body position, email subject position, etc.), and / or the formats corresponding to the position areas before and after the position transfer (such as zoom ratio, alignment, font, text color), etc.

[0074] In the case of cross-application position transfer, the application information of the applications in which the first object is located before and after the position transfer may include, but is not limited to, the type / name of the first application before the position transfer (such as being in a word document before the transfer), the position in the first application and / or the format corresponding to the location area, and the type / name of the second application after the position transfer, the position in the second application (such as dragging a sentence in a word document to the email body or subject position, dragging a picture to the email body or attachment position, etc.) and / or the format corresponding to the location area.

[0075] See also Figure 2 The process of using the first agent to identify the object information of the first object and the application information of the applications in which the first object is located before and after the position transfer can be implemented as follows:

[0076] Step 201: Obtain an object screenshot obtained by taking a screenshot of the first object, and an area screenshot obtained by taking screenshots of the areas where the first object is located before and after the position transfer.

[0077] In response to detecting a position transfer event of the first object, such as a dragging event of the first object, the screenshot function of the electronic device can be automatically called to take a screenshot of the first object, as well as to take screenshots of the areas where the first object is located before and after the position transfer, such as taking a screenshot of the dragged text or picture, and taking screenshots of the application windows where the text / image is located before and after the dragging, etc.

[0078] It is easy to understand that in non-cross-application position transfer, the area screenshots obtained by taking screenshots of the areas where the first object is located before and after the position transfer include different screenshots formed by different position areas where the first object is located in the same application before and after the position transfer. In cross-application position transfer, the area screenshots obtained by taking screenshots of the areas where the first object is located before and after the position transfer include screenshots of application areas (such as application windows) of different applications where the first object is located before and after the position transfer.

[0079] Step 202: input the object screenshot of the first object, the area screenshot and the first guidance information into the large language model, and obtain the object information of the first object and the application information of the applications in which the first object is located before and after the position transfer, which are generated by the large language model based on the object screenshot and the area screenshot under the guidance of the first guidance information.

[0080] Optionally, a prompt word project based on a large language model can be set to guide the large language model to generate first guidance information of object information and application information of the application where the object is located before and after position transfer according to the object screenshot and regional screenshots of the areas where the object is located before and after position transfer (different regional screenshots of the same application or different regional screenshots of different applications).

[0081] The first guidance information set may specifically include, but is not limited to, information such as model roles, role functions, role tasks and / or task samples set around the above-mentioned guidance targets, so as to guide the large language model to generate object information of the object and application information of the application in which the object is located before and after the position transfer by specifying model roles, role functions, role tasks and / or providing corresponding task samples to the large language model.

[0082] On this basis, by inputting the object screenshot of the first object, the area screenshots of the areas where the first object is located before and after the position transfer, and the first guidance information into the large language model, the large language model can obtain the object information of the first object and the application information of the application where the first object is located before and after the position transfer based on the object screenshot and the area screenshot under the guidance of the first guidance information.

[0083] Among them, for non-cross-application position transfer, the large language model can specifically generate and output the object information of the first object, as well as the corresponding application information of different position areas of the same application before and after the position transfer of the first object, such as the type / name of the same application, the different positions of the first object on the same application before and after the position transfer (such as text area position, table area position) and the corresponding formats of the different positions, etc. For cross-application position transfer, the large language model can specifically generate and output the object information of the first object, as well as the relevant information of the first application and the second application where the first object is located before and after the position transfer, such as the type / name of the first application before the position transfer, the corresponding position in the first application and the content format of the corresponding position, the type / name of the second application after the position transfer, the corresponding position in the second application and the content format of the corresponding position, etc.

[0084] Step 102: Determine the processing intention for the first object based on the object information of the first object and the application information of the applications in which the first object is located before and after the position transfer, and obtain a processing intention determination result.

[0085] The processing intention of the first object is used to characterize the purpose of the operation processing of the first object, which may include but is not limited to adaptively adjusting the format at the new location after the transfer, generating table content based on the first object, generating / sending email content, data retrieval, etc.

[0086] In an optional implementation, this step 102 may determine the processing intention for the first object based on a first agent such as a large model such as a large language model. In this implementation, optionally, based on the prompt word engineering of the large language model, second guidance information for guiding the large language model to determine the processing intention for the first object according to the object information of the first object and the application information of the applications in which the first object is located before and after the position transfer can be set.

[0087] The second guidance information set may also include but is not limited to information such as model roles, role functions, role tasks and / or task examples. It is easy to understand that all this information needs to be set around the guidance goal of the second guidance information for the large language model.

[0088] On this basis, the object information of the first object, the application information of the first object before and after the position transfer, and the second guidance information can be input into the large language model, so as to obtain the first processing intention for the first object generated by the large language model under the guidance of the second guidance information. The first processing intention can be used as the processing intention determination result for the first object.

[0089] However, it is not limited thereto. In other implementations, an intent understanding data set may be pre-built, for example, an intent understanding library may be built based on the user's historical operation data. An intent understanding data set such as an intent understanding library includes at least one piece of intent understanding data, each piece of intent understanding data including object information of a corresponding object, application information of the application in which the corresponding object is located before and after the position transfer of the corresponding object (covering cross-application object transfer and non-cross-application object transfer), and processing intent for the corresponding object.

[0090] In this implementation, a second processing intent that matches the object information of the first object and the application information of the application in which the first object is located before and after the position transfer can be determined from the intent understanding data set by query as a result of determining the processing intent for the first object.

[0091] In practical applications, the above two implementations can also be used in combination. For example, when the intention understanding dataset is empty or does not have sufficient data, a large language model is used to determine the first processing intention for the first object. When the intention understanding dataset has sufficient data, in addition to using the large language model to determine the first processing intention for the first object, the second processing intention for the second object is determined based on the intention understanding dataset, and the first processing intention and the second processing intention are used together as the result of determining the processing intention for the first object. In this implementation, the processing intention for the first object includes the first processing intention and the second processing intention.

[0092] Step 103: Determine the target processing to be performed on the first object after the position transfer by using the first agent to determine the result based on the processing intention.

[0093] The processing intent determination result may include one or more processing intents for the first object, depending on the actual intent recognition situation. For example, after dragging a sentence from Word to the body of an email, based on a large language model and / or an intent understanding dataset, the processing intent recognition result for the sentence may include but is not limited to:

[0094] meaning Figure 1 , fill in the sentence as part of the email body and adapt it to the current format of the email body.

[0095] meaning Figure 2 , search the sentence or its keywords in a search engine, fill the search results in the email body as part of the email body, and adapt to the current format of the email body.

[0096] For example, a person's name in the sentence is searched and the obtained person introduction information is filled in the email body, and it is adapted to the current format of the email body; or a technical term in the sentence is searched and the obtained related technical explanation is filled in the email body, and it is adapted to the current format of the email body.

[0097] meaning Figure 3 , search the sentence or its keywords in a search engine and attach the search results to the email.

[0098] For example, a person's name or a technical term in the sentence is searched, and the related documents obtained from the search (such as a paper published by a person, a paper related to a technical term) are attached to the email.

[0099] After obtaining the processing intention determination result for the first object, if the processing intention determination result only includes one processing intention for the first object, that is, only includes a single intention, optionally, the single intention can be directly used as the target intention for the first object to determine the target processing to be performed on the first object after the position is transferred. If the processing intention determination result includes multiple processing intentions for the first object, that is, includes multiple intentions, based on the weights corresponding to different intentions in the multiple intentions, one intention that meets the weight condition (such as the highest weight) can be selected from the multiple intentions as the target intention for the first object to determine the target processing to be performed on the first object after the position is transferred.

[0100] The weights corresponding to different intentions can be provided by the large language model based on its model capabilities when performing intent recognition, or can be set in advance in the intent understanding dataset based on the actual execution frequency of different processing intentions in history. Among them, for different processing intentions corresponding to the same position transfer event of the same object, the higher the actual execution frequency of a certain processing intention in history, the higher the weight of the processing intention, and accordingly, the higher the possibility that the processing intention will be selected as the final target intention in subsequent processing.

[0101] In actual applications, the processing intention (single intention or multiple intentions) for the first object determined based on a large language model and / or an intent understanding data set may not be consistent with the user's actual processing intention for the first object. In this case, optionally, the single intention or multiple intentions contained in the intention recognition result for the first object can be displayed for the user to view and confirm, and an intention interaction interface can be provided to support the user in submitting the actual processing intention required for the first object when the displayed intention is inconsistent with the user's actual processing intention.

[0102] The intention interaction interface can be implemented in the form of a chat window, but is not limited to it, and supports users to submit the actual processing intentions required to the chat window in the form of text input, voice input and / or gestures, etc. And the hiding and calling functions of the chat window can be set so that users can hide or call it according to actual needs.

[0103] Based on this, if the processing intention determination result represents that the processing intention for the first object is a single intention, the single intention or the first intention of the operating body input can be obtained as the target intention for the first object to determine the target processing to be performed on the first object after the position is transferred.

[0104] If the processing intention determination result represents that the processing intention for the first object is multiple intentions, then the intention selected from the multiple intentions can be obtained, or the second intention of the operating body input can be obtained as the target intention for the first object, so as to determine the target processing to be performed on the first object after the position is transferred.

[0105] The operating body may be, but is not limited to, a keyboard / mouse, a user's finger, a touch pen, and the like.

[0106] On this basis, the target intent and the third guidance information can be input into the large language model to obtain an operation sequence generated by the large language model under the guidance of the third guidance information to meet the target intent. The operation sequence is used to indicate one or more operations to be performed on the first object to represent the target processing to be performed on the first object after the position transfer.

[0107] For example, for the position transfer event of "dragging a sentence from Word to the body of the email", it is assumed that the recognized processing intention for the sentence is the above intention. Figure 2 , that is, "search the sentence or its keywords in the search engine, fill the search results in the email body as part of the email body, and adapt to the current format of the email body", then the operation sequence output by the large language model may include: extract keywords from the sentence → construct search conditions containing the extracted keywords → input the search conditions into the search engine for search → identify the current format of the email body → fill the search results in the dragged position of the email body according to the identified format.

[0108] It should be noted that, when an operation sequence includes multiple operations, different operations in the operation sequence may be executed in series or in parallel, depending on the actual relationship between the different operations (such as whether there is a dependency). For example, in the above operation sequence, "constructing search conditions containing the extracted keywords" needs to depend on "extracting keywords from the sentence", so the two are executed in a serial order, while there is no dependency between the two operations "constructing search conditions containing the extracted keywords" and "identifying the current format of the email body", so the two can be executed in parallel. Of course, different operations that do not have a dependency can also be executed in series, which can be determined in combination with the actual resource scheduling situation.

[0109] The third guidance information is used to guide the large language model to generate an operation sequence for satisfying the target intent. In implementation, the third guidance information can also be set based on the prompt word engineering of the large language model, and the set third guidance information can also include but is not limited to information such as model roles, role functions, role tasks and / or task examples, and all of this information is set around the guidance goal of guiding the large language model to generate an operation sequence that satisfies the target intent.

[0110] Step 104: Use the second agent to perform the target processing on the first object after the position transfer, and obtain the processing result of the first object.

[0111] The second intelligent agent can also be an AI model such as a large model, specifically a large action model (Large Action Models, LAM).

[0112] After determining the operation sequence to be performed on the first object after the position transfer, the AI ​​model such as the big action model can be further used to perform the operations indicated by the operation sequence on the first object after the position transfer. In the case where the operations indicated by the operation sequence include multiple operations, different operations in the operation sequence can be performed on the first object after the position transfer in a corresponding required serial and / or parallel manner, thereby automatically generating a processing result for the first object.

[0113] For example, in the known technology, the copy and paste operation in the same application often needs to adjust the format again, such as copying the white background color text in the ppt to the black background area, the format needs to be adjusted to adapt to the new background color; when copying the text in the word document to the table, the format needs to be adjusted to adapt to the table format. For this usage scenario in the same application, the embodiment of the present application can use the first intelligent agent such as the large language model to identify the object information of the transferred (position-shifted) text, and the area information of the area where the text is located before and after the position transfer (such as the white background area and the black background area in the ppt before and after the text transfer, and the text area and the table area in the word before and after the text transfer), to determine the processing intention for the object of the position-shifted text, and the operation sequence for satisfying the processing intention, and use the second intelligent agent such as the large action model to automatically perform the operation indicated by the operation sequence on the position-shifted text, so as to automatically generate the processing result of the position-shifted text. For example, the color of text transferred to the black background area of ​​​​the PPT is automatically adjusted to make it adapt to the background color of the black background area, and the format of text transferred to the table area of ​​​​the Word is automatically adjusted in terms of font size, font, alignment and / or color to make it adapt to the format of the table area.

[0114] For another example, in the known technology, copying and pasting across applications is often not the result expected by the user. For example, when copying text from ppt and pasting it into WeChat, the copied text will become a picture; when copying a table from excel into ppt, it takes time to adjust the format. For this usage scenario, the first agent such as a large language model can be used to identify the object information of the position transfer object such as text and table and the application information of the application before and after the position transfer, so as to determine the processing intention of the cross-application transfer object such as text and picture, and the operation sequence for satisfying the processing intention, and the second agent such as a large action model can be used to automatically perform the operation indicated by the operation sequence on the text, picture and other objects after the cross-application position transfer, so as to automatically generate the processing results of the text and picture after the cross-application position transfer. For example, the text transferred from ppt to WeChat is automatically formatted so that it maintains the text type and adapts to the current format of the area in WeChat, and the text transferred from excel to ppt is automatically formatted so that it adapts to the current format of the position in ppt.

[0115] To summarize, the operation processing method provided in the embodiment of the present application, for the position transfer event of the first object, uses the first intelligent agent to identify the object information of the first object and the application information of the applications in which the first object is located before and after the position transfer, determines the processing intention for the first object according to the recognition result, and determines the target processing to be performed on the first object after the position transfer that matches the processing intention, and then uses the second execution body to perform the target processing on the first object after the position transfer, thereby realizing intelligent and automatic generation of processing results for the first object after the position transfer, thereby reducing the tediousness of user operations in cross-application or non-cross-application object position transfer operation scenarios, improving operation efficiency, and reducing time consumption.

[0116] It should also be noted that the method of the embodiment of the present application supports one-step transfer of any type of operable object between different locations of the same application or between different applications through shortcuts such as dragging, voice control and / or gesture control, without the need for cumbersome operations as in the known technology to achieve the required cross-application / non-cross-application object position transfer, thereby further improving the convenience of user operation and enhancing user experience.

[0117] In an optional embodiment, the operation processing method provided by the present application may further include at least one of the following steps 1-1 to 1-2:

[0118] 1-1. Update the intention understanding dataset based on the object information of the first object, the application information of the applications in which the first object is located before and after the position transfer, and the actual processing intention of the first object.

[0119] The actual processing intention can be the single intention or the first intention of the operating body input when the processing intention determination result represents the processing intention for the first object as a single intention, or can also be the intention selected from the multiple intentions or the second intention of the operating body input when the processing intention determination result represents the processing intention for the first object as multiple intentions, depending on the actual application situation.

[0120] Based on the object information of the first object, the application information of the application in which the first object is respectively located before and after the position transfer, and the actual processing intention of the first object, the intention understanding data set is updated, which may include but is not limited to adding the object information of the first object, the application information of the application in which the first object is respectively located before and after the position transfer, and the actual processing intention of the first object as one piece of intention understanding data to the intention understanding data set when the object information of the first object, the application information of the application in which the first object is respectively located before and after the position transfer, and the actual processing intention of the first object are not contained in the intention understanding data set. When the object information of the first object, the application information of the application in which the first object is respectively located before and after the position transfer, and the actual processing intention of the first object are contained in the intention understanding data set, the frequency information of the object information of the first object, the application information of the application in which the first object is respectively located before and after the position transfer, and the actual processing intention of the first object in the intention understanding data set is updated.

[0121] By updating the intention understanding dataset based on the object information of the first object, the application information of the first object before and after the position transfer, and the actual processing intention of the first object, the data of the intention understanding dataset is continuously enriched and improved, so as to provide richer data basis for subsequent intention recognition of object position transfer events.

[0122] 1-2. Determine the weights of different processing intentions of the object based on the execution frequency information of different processing intentions of the object in the intention understanding data set.

[0123] There may be multiple different processing intentions for the same position transfer event of the same object. In this case, the weights of the different processing intentions of the object can be determined based on the execution frequency information of the different processing intentions of the same position transfer event of the same object in historical operations. Among them, the higher the execution frequency of the processing intention, the higher its corresponding weight, and vice versa, the lower the execution frequency of the processing intention, the lower its corresponding weight. Subsequently, when responding to the position transfer event of the first object to identify the processing intention of the first object, the weights of the different processing intentions of the objects in the intention understanding data set can be combined to sort the various processing intentions contained in the processing intention identification result of the first object. Optionally, the higher the weight of the processing intention, the higher its sorting position, so that the user can confirm whether there is an intention option that meets his actual intention and make a selection.

[0124] It is easy to understand that as users continuously apply position movement events to objects according to actual needs, and actually execute / adopt corresponding processing intentions to operate and process the objects after the position movement event, the execution frequency information of different processing intentions of objects in the intention understanding dataset can be dynamically updated.

[0125] This embodiment can continuously enrich and improve the data of the intent understanding data set by updating the intent understanding data set based on the object information of the first object, the application information of the applications in which the first object is located before and after the position transfer, and the actual processing intention of the first object, thereby providing a richer data basis for the subsequent identification of the processing intention of the object position transfer event. By determining the weights of different processing intentions of the object based on the execution frequency information of different processing intentions of the object in the intent understanding data set, it is convenient to sort the various intentions contained in the processing intention identification results of the first object based on the weights of different processing intentions when identifying the processing intention of the first object in response to the position transfer event of the first object, so that the user can quickly and efficiently confirm whether there is an intention option that meets his actual intention and make a selection, further improving the user's operating efficiency.

[0126] In an alternative embodiment, see Figure 3 As shown in the flowchart of the operation processing method, the operation processing method provided by the present application may further include the following processing before detecting the position transfer event of the object:

[0127] Step 301: In response to detecting an event of positioning a target control in an application in a target mode, identifying a second object in an area corresponding to the positioning position, and controlling the second object to be in a target state based on the identification result.

[0128] The target state is a state for selecting at least a part of the structure of the second object; and the first object is at least a part of the structure of the second object selected from the second object.

[0129] The target mode is used for the position transfer event of the object. Based on the operation processing method provided in the embodiment of the present application, the object where the position transfer event occurs is automatically processed to generate a mode of the processing result of the object. In actual application, the user can choose to turn on or off the target mode according to needs or habits. When the target mode is turned off, the user can transfer the object and perform the required processing on the object after the position transfer based on the conventional operation method. For example, after copying a table from Excel to PPT, manually adjust the format of the table copied to PPT to adapt it to the format requirements of PPT, or when performing a text copy operation in the same application, manually adjust the text format copied to the new location to adapt it to the format requirements of the new location.

[0130] The target control is used to locate the position in the application, and supports users to move the target control in the application by dragging, voice control and / or gesture control, and by moving the target control to the desired position, the desired position is located. The target control can be, but is not limited to, implemented as a floating ball. In the embodiment of the present application, it is specifically implemented as an AI floating ball that can be moved. See Figure 4 , provides an exemplary appearance of the AI ​​floating ball, and supports position movement by dragging, voice control and / or gesture control.

[0131] When the user needs to select a second object or a component of the second object within the application in order to transfer the position of the selected first object (the first object is the second object or a part of the second object), the user can first move the target control such as the AI ​​floating ball to the position of the second object by dragging the AI ​​floating ball, so as to use the target control to locate the area where the required second object is located.

[0132] In response to detecting an event of positioning a target control within an application in target mode, this embodiment identifies a second object in an area corresponding to the positioning position, wherein the type of the second object can be specifically identified, such as whether the second object is text, image, audio or video, and on this basis, the second object is controlled to be in a target state based on the identification result.

[0133] The target state is specifically a state for selecting at least a part of the component structure of the second object, and the target state matches the type of the second object.

[0134] Among them, if the recognition result indicates that the second object is text, the target state may be a first state for selecting text content, and the second object may be controlled to be in the first state for selecting text content accordingly. The first state may specifically be, but is not limited to, a word-marking interaction state that supports the selection of one or more keywords, phrases, sentences, paragraphs, etc. in the second object, so as to support the user to select part or all of the text content of the second object through word-marking interaction and other methods.

[0135] If the recognition result indicates that the second object is an image, the target state may be a second state for taking a screenshot of the image content, and the second object may be controlled to be in the second state for taking a screenshot of the image content. The second state may specifically be, but is not limited to, a screenshot interaction state that supports taking a screenshot of part or all of the image content of the second object, so as to support selecting part or all of the image content of the second object based on screenshot interaction or the like.

[0136] If the recognition result indicates that the second object is audio or video, the target state may be a third state for selecting audio or video segments, and the second object may be controlled to be in the third state for selecting audio or video segments accordingly. The third state may specifically be, but is not limited to, a segment selection interaction state for supporting the selection of segments of the audio or video progress bar of the second object, so as to support the selection of some or all audio or video segments of the second object based on the segment selection interaction mode.

[0137] See also Figure 5 , provides an exemplary implementation process of the state control of the second object in this embodiment, wherein, in target mode, the user can drag the AI ​​floating ball to the desired position in the application for positioning, and in response to detecting the event of positioning the AI ​​floating ball as a target control in the application in target mode, specifically, a screenshot of the positioning area can be taken and the position (x, y) of the AI ​​floating ball can be detected and marked, and then the object (second object) in the area can be identified with the position (x, y) as the starting point, for example, the content type in the window can be identified.

[0138] On this basis, according to the content type of the identified second object, the second object is controlled to be in the corresponding target state, wherein, if the second object is text, the second object can be controlled to be in a word-marking interaction state, as shown in FIG6(a), in which the user can select part or all of the text content (such as keywords, sentences, paragraphs) in the second object by interacting with the second object by marking words. If the second object is an image, the second object can be controlled to be in a screenshot interaction state, as shown in FIG6(b), in which the user can select part or all of the image content of the second object by taking a screenshot of part or all of the image content of the second object. If the second object is audio or video, the second object can be controlled to be in a segment selection interaction state capable of selecting segments of the audio and video progress bar, as shown in FIG6(c), in which the user can select part or all of the audio and video clips of the second object by selecting segments of the audio and video progress bar of the second object.

[0139] The selected part or all of the text, images, audio and video clips and other objects are used as the first object for subsequently applying a position transfer event to it, and based on the method of the embodiment of the present application, by utilizing an intelligent agent to perform intention recognition, operation sequence determination and other processing, the corresponding operation is automatically executed on the first object after the position transfer, thereby obtaining an intelligent processing result for the first object after the position transfer.

[0140] It should be noted that the above-mentioned word-marking interaction state, screenshot interaction state, paragraph selection interaction state, etc. are only exemplary implementation methods of the target state and are not used to limit the target state. During implementation, as long as the second object is controlled to be in a state where part or all of its content can be selected based on the recognition result, it falls within the protection scope of the embodiments of the present application.

[0141] This embodiment sets a target control for positioning in an application, and in response to detecting an event of positioning the target control in the application in target mode, identifies the second object in the area corresponding to the positioning position, and then controls the second object to be in a target state based on the identification result. This allows users to quickly and efficiently select part or all of the component structure of the second object in the positioned area, so as to use the selection result as the first object to apply subsequent position transfer events and agent-based automated operation processing to it, thereby further improving the user's operating efficiency when performing position transfer of objects and related operation processing based on this in the same application or across applications.

[0142] An application example of the present application method is provided below.

[0143] In this example, the target mode is specifically implemented as an intelligent drag mode, which can support the position transfer of the required first object based on the dragging method and the intelligent operation processing based on this.

[0144] See also Figure 7 As shown, a flowchart of intelligent operation processing based on position transfer of the required object is shown. In this example, the intelligent drag mode can be started / opened first. After that, the user drags content C (i.e., the first object, which can be but not limited to text, image, audio and video or part of these contents) from window A to window B according to actual needs to realize the position transfer of content C. On this basis, the large language model LLM can be further used to identify content C and window A and window B, and determine the processing intention for content C based on the recognition result, wherein one or more processing intentions (I1, I2...) of the user for content C can be determined and displayed based on the large language model and / or the intention understanding database. When multiple intentions are identified, the display order of each intention can be adjusted based on the weight corresponding to each intention, supporting the user to select the required intention from them or input the required intention by themselves, and then determining the operation sequence for satisfying the required intention, and using the large action model LAM to perform the operation processing indicated by the operation sequence on the content C after the position is moved, to obtain the processing result of the content C after the position is moved. Then, the relevant information of content C and the two windows, as well as the execution intention, is recorded, and the intention understanding library is updated based on the recorded information (such as adding records to the intention understanding library, or adjusting the intention weight, etc.).

[0145] The application scenarios applicable to the above process may include but are not limited to:

[0146] Scenario 1: Drag a picture / video or part of a picture / video in Office to a folder and save it directly;

[0147] Scenario 2: Drag a picture or part of a picture in Office to the browser and search for the same picture;

[0148] Scenario 3: Adjust the text color, size, etc. of the dragged content within Office to automatically adapt to the current format of the location after dragging.

[0149] Taking scenario 2 as an example, see the example of FIG8(a), the user drags a picture in the office to the search input box of the browser.

[0150] On this basis, the processing intention for the image is determined based on the method of this application. If the determined processing intention is a single intention and the user has no objection, the operation processing for satisfying the single intention is automatically determined and executed on the moved image, such as constructing search conditions based on the image to search for the same image, etc.

[0151] During the execution process, as shown in FIG8(b), the AI ​​floating ball icon can be used and controlled to be in a predetermined state to represent that a corresponding operation (such as a search for the same image) is being performed on the first object after the position is moved (such as the image in FIG8(a)).

[0152] After dragging the image, multiple intents may be identified. In this case, the user can be provided with intent options for selecting multiple intents, and the display order of each option can be determined based on their corresponding weights. The user can click on one of the intent options to select the intent, so as to automatically perform matching operations on the dragged image based on the selected intent. Alternatively, when the user's actual desired intent does not exist in the multiple intent options, the intent input interface can be called up. Optionally, the user can click the AI ​​floating ball icon on the interface to call up the intent input interface, as shown in the schematic diagram of Figure 8(c). On this basis, the actual desired intent is input through the intent input interface, and then the dragged image is automatically subjected to operations matching the input intent, as shown in Figure 8(d). This example reduces the tediousness and time-consuming nature of user operations and improves operational efficiency by performing intent recognition and automated operation processing on dragged objects based on agents.

[0153] Corresponding to the above method, the present application embodiment also provides an operation processing device, whose composition structure is as follows: Fig. 9 As shown, including:

[0154] An identification module 901 is used for, in response to detecting a position transfer event of a first object, using a first agent to identify object information of the first object and application information of applications in which the first object is located before and after the position transfer;

[0155] A first determination module 902 is used to determine a processing intention for the first object according to the object information of the first object and the application information of the applications in which the first object is located before and after the position transfer, and obtain a processing intention determination result;

[0156] A second determination module 903 is used to determine the target processing to be performed on the first object after the position transfer by using the first agent to determine the result based on the processing intention;

[0157] The processing module 904 is used to use the second agent to perform the target processing on the first object after the position transfer, and obtain the processing result of the first object.

[0158] In an optional implementation, the apparatus further includes a detection module for detecting a position transfer event of the first object, wherein the detection module detects a position transfer event of the first object, including at least one of the following:

[0159] detecting a position transfer event of dragging the first object from a first position to a second position in the application;

[0160] A position transfer event of dragging an object in a first application to a second application is detected.

[0161] In an optional implementation manner, the identification module 901 is specifically configured to:

[0162] obtaining an object screenshot obtained by taking a screenshot of the first object, and an area screenshot obtained by taking screenshots of the areas where the first object is located before and after the position transfer;

[0163] The object screenshot, the area screenshot and the first guidance information are input into a large language model to obtain object information of the first object and application information of applications in which the first object is located before and after position transfer, generated by the large language model based on the object screenshot and the area screenshot under the guidance of the first guidance information.

[0164] In an optional implementation manner, the first determining module 902 is specifically configured to:

[0165] Inputting object information of the first object, application information of the application in which the first object is located before and after the position transfer, and second guidance information into a large language model, to obtain a first processing intention for the first object generated by the large language model under the guidance of the second guidance information;

[0166] Determine, from the intent understanding data set, a second processing intent that matches the object information of the first object and the application information of the application in which the first object is located before and after the position transfer;

[0167] Among them, the processing intention for the first object includes at least one of the first processing intention and the second processing intention; the intention understanding data set includes at least one intention understanding data, each intention understanding data includes object information of the corresponding object, application information of the application in which the corresponding object is located before and after the corresponding object is transferred, and the processing intention for the corresponding object.

[0168] In an optional embodiment, the above device further includes a post-processing module, which is used to perform at least one of the following:

[0169] updating the intention understanding dataset based on the object information of the first object, the application information of the applications in which the first object is located before and after the position transfer, and the actual processing intention of the first object;

[0170] Based on the execution frequency information of different processing intentions of the object in the intention understanding data set, the weights of the different processing intentions of the object are determined.

[0171] In an optional implementation manner, the second determining module 903 is specifically configured to:

[0172] Inputting the target intent and the third guidance information into the large language model, obtaining an operation sequence generated by the large language model under the guidance of the third guidance information to satisfy the target intent; the operation sequence is used to indicate one or more operations to be performed on the first object, and the target intent is an intent obtained based on the result of determining the processing intent;

[0173] The processing module 904 is specifically configured to: use the large action model to perform the operations indicated by the operation sequence on the first object after the position is transferred.

[0174] In an optional implementation manner, the process of obtaining the target intention includes:

[0175] If the processing intention determination result indicates that the processing intention for the first object is a single intention, the single intention is obtained or the first intention of the operating body input is obtained; the target intention is the single intention or the first intention;

[0176] If the processing intention determination result represents that the processing intention for the first object is multiple intentions, an intention selected from the multiple intentions is obtained, or a second intention input by the operating body is obtained; the target intention is the intention selected from the multiple intentions or the second intention.

[0177] In an optional embodiment, the device further includes a control module for identifying a second object in an area corresponding to the positioning position in response to detecting an event of positioning the target control in the application in the target mode before detecting the event of position transfer of the object, and controlling the second object to be in the target state based on the recognition result;

[0178] The target state is a state for selecting at least a part of the structure of the second object; and the first object is at least a part of the structure of the second object selected from the second object.

[0179] In an optional implementation manner, when the control module controls the second object to be in the target state based on the recognition result, it is specifically used to:

[0180] If the recognition result indicates that the second object is text, controlling the second object to be in a first state for selecting text content,

[0181] If the recognition result indicates that the second object is an image, controlling the second object to be in a second state for taking a screenshot of the image content;

[0182] If the recognition result indicates that the second object is audio or video, the second object is controlled to be in a third state for selecting audio or video clips.

[0183] The present application also discloses an electronic device, the composition structure of the electronic device, such as Fig.10 As shown, including at least:

[0184] A memory 10, for storing a computer instruction set;

[0185] A set of computer instructions can be implemented in the form of a computer program.

[0186] The processor 20 is used to implement the operation processing method provided by any of the above method embodiments by executing a computer instruction set.

[0187] The processor 20 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a neural network processor (NPU), a deep learning processor (DPU) or other programmable logic devices, etc.

[0188] The electronic device includes a display device and / or includes a display interface and can be connected to an external display device.

[0189] Optionally, the electronic device also includes a camera assembly, and / or is connected to an external camera assembly.

[0190] In addition, the electronic device may also include components such as a communication interface and a communication bus. The memory, the processor and the communication interface communicate with each other through the communication bus.

[0191] The communication interface is used for communication between electronic devices and other devices. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0192] An embodiment of the present application also provides a storage medium, which carries one or more computer instruction sets. When the one or more computer instruction sets are executed by an electronic device, the electronic device can implement the operation processing method as described in any one of the above.

[0193] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0194] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0195] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or the part that makes a creative contribution can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0196] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0197] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An operation processing method, comprising: In response to detecting a position transfer event of a first object, using a first agent to identify object information of the first object and application information of applications in which the first object is located before and after the position transfer; Determine a processing intention for the first object according to the object information of the first object and the application information of the applications in which the first object is located before and after the position transfer, and obtain a processing intention determination result; Determine the target processing to be performed on the first object after the position transfer by using the first agent based on the processing intention determination result; The target processing is performed on the first object after the position transfer by using the second agent to obtain the processing result of the first object.

2. The operation processing method according to claim 1, wherein the detecting of the position transfer event of the first object comprises at least one of the following: detecting a position transfer event of dragging the first object from a first position to a second position in the application; A position transfer event of dragging an object in a first application to a second application is detected.

3. The operation processing method according to claim 1, wherein the object information of the first object and the application information of the first object before and after the position transfer are respectively identified by the first agent, comprises: obtaining an object screenshot obtained by taking a screenshot of the first object, and an area screenshot obtained by taking screenshots of the areas where the first object is located before and after the position transfer; The object screenshot, the area screenshot and the first guidance information are input into a large language model to obtain object information of the first object and application information of applications in which the first object is located before and after position transfer, generated by the large language model based on the object screenshot and the area screenshot under the guidance of the first guidance information.

4. The operation processing method according to claim 1, wherein determining the processing intention for the first object according to the object information of the first object and the application information of the applications in which the first object is located before and after the position transfer, comprises at least one of the following: Inputting object information of the first object, application information of the application in which the first object is located before and after the position transfer, and second guidance information into a large language model, to obtain a first processing intention for the first object generated by the large language model under the guidance of the second guidance information; Determine, from the intent understanding data set, a second processing intent that matches the object information of the first object and the application information of the application in which the first object is located before and after the position transfer; in, The processing intention for the first object includes at least one of the first processing intention and the second processing intention; the intention understanding data set includes at least one intention understanding data, each intention understanding data includes object information of the corresponding object, application information of the application in which the corresponding object is located before and after the corresponding object is transferred, and the processing intention for the corresponding object.

5. The operation processing method according to claim 4, further comprising at least one of the following: updating the intention understanding dataset based on the object information of the first object, the application information of the applications in which the first object is located before and after the position transfer, and the actual processing intention of the first object; Based on the execution frequency information of different processing intentions of the object in the intention understanding data set, the weights of the different processing intentions of the object are determined.

6. The operation processing method according to claim 1, wherein the step of using the first agent to determine the target processing to be performed on the first object after the position transfer based on the processing intention determination result comprises: Inputting the target intent and the third guidance information into the large language model, obtaining an operation sequence generated by the large language model under the guidance of the third guidance information to satisfy the target intent; the operation sequence is used to indicate one or more operations to be performed on the first object, and the target intent is an intent obtained based on the result of determining the processing intent; The using the second agent to perform the target processing on the first object after the position transfer includes: The large action model is used to perform the operations indicated by the operation sequence on the first object after the position is transferred.

7. The operation processing method according to claim 6, wherein the process of obtaining the target intention comprises: If the processing intention determination result indicates that the processing intention for the first object is a single intention, obtaining the single intention or obtaining the first intention input by the operating body; The target intention is the single intention or the first intention; If the processing intention determination result indicates that the processing intention for the first object is multiple intentions, obtaining an intention selected from the multiple intentions, or obtaining a second intention input by an operating body; The target intent is an intent selected from the multiple intents or is the second intent.

8. The operation processing method according to claim 1, before detecting the position transfer event of the object, further comprising: In response to detecting an event in which a target control is positioned within an application in a target mode, identifying a second object in an area corresponding to the positioned position, and controlling the second object to be in a target state based on the identification result; The target state is a state for selecting at least a part of the structure of the second object; and the first object is at least a part of the structure of the second object selected from the second object.

9. The operation processing method according to claim 8, wherein the step of controlling the second object to be in a target state based on the recognition result comprises: If the recognition result indicates that the second object is text, controlling the second object to be in a first state for selecting text content, If the recognition result indicates that the second object is an image, controlling the second object to be in a second state for taking a screenshot of the image content; If the recognition result indicates that the second object is audio or video, the second object is controlled to be in a third state for selecting audio or video clips.

10. An operation processing device, comprising: an identification module, configured to, in response to detecting a position transfer event of a first object, use a first agent to identify object information of the first object and application information of applications in which the first object is located before and after the position transfer; A first determination module, configured to determine a processing intention for the first object according to the object information of the first object and the application information of the applications in which the first object is located before and after the position transfer, and obtain a processing intention determination result; A second determination module is used to determine the target processing to be performed on the first object after the position transfer by using the first agent to determine the result based on the processing intention; The processing module is used to use the second agent to perform the target processing on the first object after the position transfer, so as to obtain the processing result of the first object.

Images