Object manipulation method and device

Abstract

A method and device for object manipulation, relating to the field of terminal technology, are disclosed. This method reduces the complexity of performing a series of repetitive operations on various objects, improving operational efficiency. It also reduces the difficulty of creating shortcuts and expands the scope of shortcut usage. The method includes: the device receiving a first user operation for processing a first object. The device can perform batch operations on a second object (other than the first object) within a batch of objects to which the first object belongs, based on the first user operation. The device responds to the processing result of the first user operation on the first object, and the processing result of the batch operation on the second object is consistent. Alternatively, the device can also create shortcuts based on the first user operation.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to object operation methods and devices. Background Technology

[0002] In the course of daily use of devices for learning or work, users may encounter scenarios where they need to perform a series of repetitive operations or processes on various objects. In such scenarios, users typically have to perform a large number of repetitive operations, such as, but not limited to, opening, copying, pasting, renaming, creating, and other processing operations. This processing method is very complex, cumbersome, and inefficient. Summary of the Invention

[0003] This application provides an object manipulation method and device that can reduce the complexity of performing a series of repetitive operations on various objects and improve operational efficiency.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] In a first aspect, an object operation method is provided, the method comprising: a device receiving a first user operation performed on a first object in batch data (or described as batch objects); the device performing a batch operation on a second object in the batch data based on the first user operation, wherein the second object is an object other than the first object, and the processing result of the device on the first object in response to the first user operation is consistent with the processing result of the device on the second object using the batch operation.

[0006] Based on the above technical solution, after receiving a first user operation performed on a first object in batch data, the device can determine the user's operation intention based on the first user operation. Then, it can perform batch operations on other objects in the batch data besides the first object based on the first user operation. The processing result of the first user operation on the first object is consistent with the processing result of the device on other objects besides the first object using batch operations. In other words, the device can perform batch operations on batch data on behalf of the user based on the user's demonstration. This eliminates the need for the user to perform numerous repetitive operations, reducing the complexity of batch operations and improving batch processing efficiency and intelligence.

[0007] In one possible design, the device performs a batch operation on a second object in the batch data based on the first user operation, including: the device outputs n processing steps corresponding to the batch operation based on the first user operation, where n is an integer greater than or equal to 1; and the device performs the n processing steps on the second object.

[0008] In this way, the device can determine the processing steps corresponding to the batch operations based on the first user operation, and then perform these processing steps on other objects besides the first object. This ensures that the processing results of the device on other objects are consistent with the user's demonstration result for the first object. The device can then perform batch operations on behalf of the user, eliminating the need for the user to perform a large number of repetitive operations, reducing the complexity of batch operations, and improving batch processing efficiency and intelligence.

[0009] In one possible design, the first user operation includes at least one of the following: operation of an input device equipped with the device, and voice commands. The device outputs n processing steps corresponding to the batch operation based on the first user operation, including: the device outputting the n processing steps based on at least one of the following: operation of the input device equipped with the device, the voice commands, and content displayed on the interface when the device receives the first user operation. Optionally, the voice commands included in the first user operation can be voice expressing the user's overall intention to process the first object, or voice explaining the processing steps corresponding to certain operations in the executed first user operation. That is, when the user performs the first user operation, there is no need to reorganize language to clearly express each of the n processing steps corresponding to the first user operation, in order to input voice commands to express each processing step. This does not increase the difficulty for the user in using voice commands, making the use of voice commands more flexible, and further does not increase the difficulty of demonstrating the operation.

[0010] In one possible design, the content displayed on the interface when the device receives the first user's operation can be obtained through screenshots or screen recordings.

[0011] In this way, the device can identify the processing steps of batch operations based on the operation of the equipped input device, voice commands, and the content displayed on the interface when the device receives a user's demonstration. In other words, during the demonstration process, the user can control the device using the equipped input device and also use voice commands for assisted explanation, making the interaction with the device more natural and reducing the user's learning cost. Combining this rich information for recognition allows the device to identify the operation steps of batch operations more quickly, improving recognition efficiency.

[0012] In one possible design, before the device performs a batch operation on the second object in the batch data based on the first user operation, the method further includes: upon receiving the first operation in the first user operation, the device outputs a first reminder message, the first reminder message being used to indicate that the device has identified a first processing step, or the first reminder message being used to confirm with the user whether the identified first processing step is accurate, the first processing step being a processing step identified based on the first operation.

[0013] In this way, when a user performs a demonstration operation and receives a user action, the device can identify the corresponding processing step based on that action. It can then indicate to the user that the identified processing step has been found, or confirm with the user whether the identified processing step is accurate. In other words, the device can progressively identify processing steps in batch operations. This allows the device to promptly identify the corresponding processing step when a user performs an operation and output that processing step to the user for confirmation. This real-time identification and confirmation method improves the accuracy of the identified processing steps, making the interaction more natural and intelligent.

[0014] In one possible design, after the device outputs the first reminder message, the method further includes: receiving a second user operation, the second user operation being used to indicate to the device whether the first processing step was accurate.

[0015] In this way, the device can also receive user input indicating whether the identified processing steps are accurate. The device can determine whether the identified processing steps are accurate based on the input, which can improve the accuracy of the identified processing steps and thus improve the accuracy of batch operations.

[0016] In one possible design, the first notification message includes the first processing step, and the second user action includes an operation to edit the first processing step.

[0017] In this way, the device directly outputs the recognized processing steps to the user, who can then edit them directly. This simplifies the process of correcting the device's recognition results, saves time, and improves efficiency. Furthermore, compared to methods like using voice commands to correct recognition results, it reduces the difficulty for the device to understand and process instructions.

[0018] In one possible design, the second user operation includes at least one of voice commands and operation of an input device equipped with the device.

[0019] In one possible design, the device outputs n processing steps corresponding to the batch operation based on the first user operation, including: the device outputs the n processing steps based on the first user operation and the second user operation.

[0020] In this way, when there is a second user operation, that is, a user's correction operation on the processing steps, the device can also correct the processing steps of the batch results based on the demonstration operation and the user's correction operation on the processing steps, which can improve the accuracy of the obtained processing steps, thereby improving the accuracy and efficiency of batch operations.

[0021] In one possible design, before the device performs a batch operation on the second object in the batch data based on the first user operation, the method further includes: receiving a third user operation, the third user operation being used to indicate the range of the batch data to the device; and / or; outputting a second reminder message, the second reminder message being used to confirm the second object in the batch data to the user; and receiving a fourth user operation, the fourth user operation being used to indicate the second object to the device.

[0022] In this way, the device can also receive user instructions on the range of data to be processed in batches, making it easier for users to determine which data constitutes the batch data. Alternatively, the device can proactively confirm the objects to be processed in batches with the user and receive user instructions on the objects to be processed in batches, thus making it easier for the device to determine the objects to be processed in batches and improving the accuracy of batch operations.

[0023] In one possible design, before the device receives a first user operation performed on a first object in the batch data, the method further includes: the device receiving information indicating that the first user operation has started; and / or, before the device performs a batch operation on a second object in the batch data based on the first user operation, the method further includes: the device receiving information indicating that the first user operation has ended; the information indicating that the first user operation has started and the information indicating that the first user operation has ended are used by the device to identify the first user operation.

[0024] In this way, the device can also receive information to indicate the start of execution of the first user operation, and / or receive information to indicate the end of execution of the first user operation, which makes it easier for the device to determine which user operations are the first user operations, i.e. demonstration operations, so as to determine the steps of the batch operations based on the demonstration operations.

[0025] In one possible design, the device performs the n processing steps on the second object, including: the device determining a subject keyword to be replaced included in each of the n processing steps; the device determining a subject keyword of the second object, wherein the subject keyword of the second object has the same attributes as the subject keyword to be replaced; the device replacing the subject keyword to be replaced included in each processing step with the subject keyword of the second object; and the device sequentially performing each processing step after replacing the subject keyword.

[0026] Optionally, the main keywords can be determined based on user input or calculated by an algorithm.

[0027] In this way, the device can determine the main keyword to be replaced in each processing step of the batch operation. This main keyword is also the main keyword of the first object. It can also determine the main keyword of the second object, and then replace the main keyword in each processing step with the main keyword of the second object, thus determining the batch operation flow for the second object. By sequentially executing each processing step after the keyword replacement, the batch processing of the second object can be achieved. This allows the device to perform various repetitive processing steps on the objects of batch operations on behalf of the user, thereby improving the efficiency of batch operations.

[0028] In one possible design, each of the n processing steps includes an operation instruction and an operation object corresponding to the operation instruction. Before the device outputs the n processing steps corresponding to the batch operation based on the first user operation, the method further includes: the device identifying the operation instruction and the operation object corresponding to the operation instruction in each processing step based on the first user operation.

[0029] In this way, each processing step in a batch operation includes both an operation instruction and an operation object; that is, a processing step consists of an operation instruction and its corresponding operation object. Thus, the device can identify each operation step by recognizing the operation instruction and operation object within each step, thereby determining the flow of the batch operation.

[0030] In one possible design, the device can identify the operation object corresponding to the operation instruction based on the operation instruction in each processing step.

[0031] In one possible design, after the device outputs n processing steps corresponding to the batch operation based on the first user operation, the method further includes: the device creating the n processing steps as shortcut instructions.

[0032] In this way, the device can directly create shortcut instructions based on the processing steps corresponding to a batch operation, making the creation of shortcut instructions more timely and efficient.

[0033] In one possible design, before the device creates the n processing steps as shortcuts, the method further includes: the device receiving a fifth user operation, the fifth user operation being used to set at least one of setting the triggering method of the shortcut, setting the name of the shortcut, and editing the n processing steps.

[0034] In this way, the device allows users to make necessary supplementary inputs before creating shortcuts, such as setting the trigger method, setting the shortcut name, and modifying the processing steps, which can make the generated shortcuts more accurate and better meet the actual needs of users.

[0035] In one possible design, the batch operation is performed on a single application, or the batch operation is performed on multiple applications.

[0036] In one possible design, after the device receives a first user operation for processing a first object in a batch of objects, the method further includes: the device sending the n processing steps to a target device, the n processing steps being used by the target device to process objects in the target device, the processing result of the target device using the n processing steps on the objects in the target device being consistent with the processing result of the device in response to the first user operation on the first object; or, the device sending a shortcut command created based on the n processing steps to the target device, the shortcut command being used to trigger the target device to execute the n processing steps on objects in the target device. In this way, based on the user's demonstration operation, the device can directly share the identified demonstration operation steps or the created shortcut command with other devices, allowing users of other devices to directly trigger the device to execute the n processing steps or the shortcut command, thereby improving the operational efficiency of other devices and expanding the scope of use of the shortcut command.

[0037] Secondly, a method for object manipulation is provided. The method includes: a device receiving a first user operation input in a first time period for processing a first object; the device creating a shortcut based on the first user operation, the shortcut triggering processing of a second object in a second time period following the first time period; the processing result of the shortcut on the second object being consistent with the processing result of the device on the first object in response to the first user operation. In this way, the user can create a shortcut by executing a demonstration operation, reducing the difficulty of shortcut generation and improving its efficiency. When a user needs to perform a series of operations multiple times, the user can create this series of operations as a shortcut. Subsequently, the user does not need to repeatedly execute this series of operations; by triggering the shortcut, the device can perform the series of operations on behalf of the user, reducing the complexity of performing repetitive batch operations on various objects and improving the efficiency of batch processing operations.

[0038] Thirdly, an apparatus is provided that has the function of implementing the method as described in any of the designs of any of the preceding aspects. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the described function.

[0039] Fourthly, an apparatus is provided, comprising: a processor and a memory coupled to the processor, the memory for storing program code including instructions, the processor reading the instructions from the memory to cause the apparatus to perform the method as designed in any of the preceding aspects.

[0040] Optionally, the memory can be coupled to the processor or independent of the processor.

[0041] In one possible design, the device may also include a display screen that can be used to perform display operations.

[0042] Fifthly, a computer-readable storage medium is provided, the computer-readable storage medium including a computer program that, when the computer program is run on a device, causes the device to perform the method as described in any of the preceding aspects.

[0043] A sixth aspect provides a computer program product comprising: a computer program or instructions that, when executed on a computer, cause the computer to perform the method as designed in any of the preceding aspects.

[0044] A seventh aspect provides a chip system including at least one processor and at least one interface circuit, the at least one interface circuit being used to perform transceiver functions and send instructions to at least one processor, wherein when at least one processor executes instructions, at least one processor performs the method as described in any of the designs in any of the preceding aspects.

[0045] It should be noted that the technical effects of any of the designs in the third to seventh aspects mentioned above can be found in the technical effects of the corresponding designs in the first or second aspects, and will not be repeated here. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the structure of a device provided in an embodiment of this application;

[0047] Figure 2 Figure 5c is a schematic diagram of the interface provided in the embodiment of this application. Figures 1 to 6 ;

[0048] Figure 5d A flowchart illustrating a device identification operation instruction provided in an embodiment of this application;

[0049] Figure 6 This application provides a schematic diagram of a device identification operation object process.

[0050] Figures 7 to 9 Interface illustration provided for embodiments of this application Figures 7 to 9 ;

[0051] Figure 10 A flowchart illustrating an object manipulation method provided in an embodiment of this application;

[0052] Figure 11 A flowchart illustrating the generation of shortcut commands provided in this application embodiment;

[0053] Figure 12 This application provides a schematic diagram of a shortcut command editing interface.

[0054] Figure 13 A flowchart illustrating yet another object manipulation method provided in an embodiment of this application;

[0055] Figure 14 This is a schematic diagram of the structure of another device provided in an embodiment of this application;

[0056] Figure 15 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0057] In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between the related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. A and B can be singular or plural.

[0058] In the description of this application, unless otherwise stated, "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, a and b and c, where a, b, and c can be single or multiple.

[0059] Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0060] To address the problem of users performing numerous repetitive operations when manipulating various objects, one possible solution is to use shortcuts. However, shortcuts can only handle simple operations, such as opening an application or playing music. For complex operations, such as copying a file, pasting it to a specific path, and then renaming it, users still need to perform many repetitive actions. This solution cannot solve the aforementioned problem.

[0061] Another possible solution is to utilize the "macro" function provided by office software. Macros can be created by recording a series of actions performed by the user within the software and converting them into corresponding code, or by having the user manually write the code. This allows users to use the created macros to perform certain repetitive tasks. However, this solution requires users to write code or debug the code converted from screen recordings, making it relatively difficult for them to operate. Therefore, this solution does not reduce the complexity of the operation.

[0062] Based on this, embodiments of this application provide an object operation method that can reduce the complexity of performing a series of repetitive operations on various objects, and improve batch processing efficiency and intelligence.

[0063] The technical solutions provided in this application embodiment can be applied to device 100 or to a system including device 100.

[0064] For example, device 100 may be a laptop computer, desktop computer, tablet computer, mobile laptop computer, handheld computer, ultra-mobile personal computer (UMPC), netbook, as well as personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device, and / or smart city device. Optionally, device 100 may be a fixed device or a portable device.

[0065] Optionally, the operating system installed on device 100 may include, but is not limited to, the following: Alternatively, other operating systems may be used. This application does not impose specific limitations on the specific type of device or the operating system installed.

[0066] For example, Figure 1 A schematic diagram of the structure of a device 100 provided in an embodiment of this application is shown.

[0067] like Figure 1 As shown, device 100 may include processor 110, memory 120, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, display screen 150, antenna, wireless communication module 160, audio module 170, speaker 170A, microphone 170B, input device 180, etc.

[0068] Processor 110 may include one or more processing units, such as application processors, modem processors, graphics processors, image signal processors, controllers, video codecs, digital signal processors, baseband processors, and / or neural network processors. These different processing units may be independent devices or integrated into one or more processors.

[0069] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0070] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0071] In some embodiments, processor 110 may include one or more interfaces, such as USB interface 130.

[0072] The charging management module 140 receives charging input from the charger. While charging the battery 142, the charging management module 140 can also supply power to the device 100 through the power management module 141.

[0073] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, memory 120, display 150, and wireless communication module 160, etc.

[0074] The wireless communication function of device 100 can be implemented through an antenna, wireless communication module 160, modem processor, and baseband processor.

[0075] Antennas are used to transmit and receive electromagnetic wave signals. Each antenna in device 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0076] The wireless communication module 160 can provide solutions for wireless communication applications on device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via an antenna, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via the antenna.

[0077] In some embodiments, the antenna of device 100 is coupled to the wireless communication module 160, enabling device 100 to communicate with networks and other devices via wireless communication technologies. The wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc.

[0078] The display screen 150 is used to display images, videos, etc. The display screen 150 includes a display panel. In some embodiments of this application, the display screen 150 can be used to display at least one of batch data, identified demonstration operations, user-inputted instructions, etc. A description of batch data, demonstration operations, and user-inputted instructions is provided below.

[0079] The memory 120 can be used to store computer executable program code, which includes instructions. The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc. The data storage area may store data created during the use of the device 100, etc. Furthermore, the memory 120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the device 100 by running instructions stored in the memory 120 and / or instructions stored in memory disposed within the processor.

[0080] In some embodiments of this application, the memory 120 may be used to store shortcut instructions for creation, which are described in detail below.

[0081] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0082] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. In some embodiments of this application, the speaker 170A can be used to output audio for user interaction.

[0083] Microphone 170B, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. In some embodiments of this application, microphone 170B can be used to collect voice commands input by a user.

[0084] Input device 180 may include a keyboard, mouse, etc. The keyboard is used to input English letters, numbers, punctuation marks, etc., into device 100, thereby issuing commands and inputting data to device 100. The mouse is an indicator for positioning on the horizontal and vertical coordinates of the display system of device 100, used to input instructions to device 100. Input device 180 can be connected to device 100 via a wired connection, such as through a GPIO interface or a USB interface. Input device 180 can also be connected to device 100 wirelessly, such as through Bluetooth or infrared.

[0085] Understandable Figure 1The illustrated structure does not constitute a specific limitation on device 100. In other embodiments of this application, device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The processing steps or functional characteristics of the illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0086] The technical solutions involved in the following embodiments can all be applied to applications with, for example, Figure 1 Implemented in the device with the structure shown.

[0087] In some embodiments, the device can receive a user's demonstration operation, which may refer to an operation performed by the user on a specific piece of data (or an object) within a batch of data (or described as a batch of objects). This demonstration operation can be used to perform one or more processing operations on the data. Exemplary examples include, but are not limited to, opening, copying, pasting, renaming, creating, editing, and downloading. The device can perform batch operations on the batch data based on the user's demonstration operation. The processing result of the device on each piece of data in the batch data is consistent with the processing result of the demonstration operation on that piece of data. In this way, the device can determine the user's operational intent based on the user's demonstration operation, and thus perform batch operations on the batch data on behalf of the user. Users are no longer required to perform a large number of repetitive operations, reducing the complexity of batch operations and improving batch processing efficiency and intelligence.

[0088] In this embodiment, the operation performed uniformly by the device on batch data is called a batch operation. A batch operation may also include one or more processing operations. Optionally, the order and type of these processing operations may be the same as in the exemplary operation.

[0089] The following is a demonstration of the user's operation, with reference to the accompanying diagram.

[0090] For example, such as Figure 2 As shown in (1), the device can display interface 200, in which batch data 201 is displayed, including multiple resumes named after different people. Figure 2 Five examples are shown in (1), such as Xiaohong's resume, Xiaoming's resume, etc., which will not be introduced one by one here.

[0091] Understandable Figure 2The example shown in (1) uses PDFs as an example of batch data. In practical applications, the data included in the batch data can be presented in various types, such as but not limited to PDFs, text documents, tables, images (in various formats), audio, video, folders, application installation packages, URLs, emails, applications, compressed files, etc. Optionally, the data included in the batch data can only include data of the same type, such as only PDFs. Of course, it can also include multiple different types of data at the same time, such as images, text documents, audio, etc. The embodiments of this application do not impose specific restrictions on the data types included in the batch data.

[0092] Optional, Figure 2 In the example shown in (1), the batch data 201 presented in interface 200 is assuming it is stored in a device. In other embodiments, the batch data can also be stored in other devices. In other words, a device can perform batch operations on batch data stored in other devices. Optionally, devices can establish connections via wired or wireless communication. For example, after a mobile phone and a computer establish a connection, the computer can perform batch operations on the data in the mobile phone. Optionally, in this embodiment, during the user's demonstration operation, the various interfaces presented by the device can be created by the device itself, or they can be transmitted to the device by other devices in scenarios such as multi-screen collaboration.

[0093] When users want to include, for example Figure 2 When the individual resumes in the batch data 201 shown in Figure (1) are stored in their respective folders, the user can first use any one of the resumes to perform a demonstration operation. In some embodiments, to facilitate the device's identification of demonstration operations, the device identifies which operations performed by the user are considered demonstration operations, thereby further improving the accuracy of batch operations. After receiving an instruction to start performing a demonstration operation (or described as information indicating the start of a first user operation), the device can identify the user operation following the instruction as a demonstration operation, and the device will then begin to execute the record of the demonstration operation. For example, the aforementioned instruction to start performing a demonstration operation can be a voice instruction, a gesture instruction, an action instruction input through a physical input device, etc.

[0094] Taking voice commands as an example, such as Figure 2As shown in Figure (1), a user can input a voice command into the device such as "Please save these resumes into each person's individual folder. Listen carefully, I'll only demonstrate once." This voice command includes keywords such as "Listen carefully, I'll only demonstrate once" to indicate the start of the demonstration operation. Correspondingly, after recognizing the keywords in the voice command, the device can determine that the operation the user is about to perform is a demonstration operation, and can then begin recording the user's demonstration operation. Optionally, to inform the user that the device has begun recording the user's demonstration operation, the device can also output a reminder message to remind the user. For example, this reminder message can be as follows: Figure 2 The reminder message 210 shown in (2) is shown in the middle.

[0095] Next, the user can begin performing the demonstration operation. For example, the user saves Xiaohong's resume to Xiaohong's folder. The demonstration operation can be performed as follows: Figure 2 Middle (3) to Figure 4 As shown in (2). First, the user can perform actions such as... Figure 2 (3) Copying Xiaohong's resume 202: For example, the user can select Xiaohong's resume 202 with the mouse and right-click to bring up the following... Figure 2 As shown in (3), the copy control 220 allows users to copy Xiaohong's resume 202 by clicking on it with the mouse. Alternatively, users can copy Xiaohong's resume 202 through other operations, such as using the keyboard shortcut "ctrl+c" or, if the device is a touchscreen, clicking on the touchscreen to bring up the copy control. This allows users to perform various complex demonstrations by executing the demonstration actions, compared to using voice commands, and also reduces the cognitive load of organizing language when using voice commands.

[0096] During the aforementioned copying operation, the user simultaneously pointed the mouse at the display location of Xiaohong's resume 202 and input the following into the device: Figure 2 The voice command “First, copy this resume” is shown in (3). The device responds to the user’s copy operation and copies Xiaohong’s resume. In this way, the user uses natural voice to explain and assist during the demonstration operation, which makes the interaction between the user and the device more natural and reduces the user’s learning cost. Compared with the method of using only voice commands, the device can recognize the operation steps in the demonstration operation more quickly and improve the recognition efficiency. Optionally, in the embodiments of this application, when the user inputs a voice command to the device, in order to let the user know that the device has received the voice command, the device can output the recognized voice command accordingly, which will be uniformly described here.

[0097] Optionally, in this embodiment, during the demonstration operation, to facilitate clearer identification of the user's demonstration operation, the device can instruct (through voice prompts or display screen reminder messages) the user to input more comprehensive information, such as: inputting clearer and more complete voice commands, inputting more complete operation steps, or demonstrating simultaneously using voice commands and physical input devices, etc., so that the device can more accurately identify the demonstration operation and improve the accuracy of batch operations. For example: such as Figure 2 The demonstration operation shown in Figure (1) involves the display of a control 10 on the interface, which can be used to prompt the user to input voice commands. In this way, the device actively prompts the user to input voice commands to further clarify the operation steps in the demonstration operation, which can improve the accuracy and efficiency of the recognized operation steps.

[0098] like Figure 2 As shown in (3), after receiving the user's voice command, the device simultaneously outputs the recognized voice command "First, copy the resume here"221. Furthermore, the user can open a folder named "Employee Information," and the device responds to this operation by displaying the following... Figure 2 The opening interface 230 of the "Employee Information" folder shown in (4) is shown. As shown in the opening interface 230, the "Employee Information" folder includes folders for each person, such as "Xiao Hong" folder 231, "Xiao Ming" folder 232, etc. The folders here are named after the people. Each folder will not be described in detail.

[0099] Next, as Figure 2 As shown in (4), the user can continue to perform the operation of opening the "Little Red" folder 231. In response to this operation, as shown in (4), the user can perform the operation of opening the "Little Red" folder 231. Figure 3 As shown in (1), the device displays the opening interface 300 of the "Little Red" folder 231. During the process of the user opening the "Little Red" folder 231, the device can also detect the user's operation of selecting the "Little Red" folder 231 with the mouse, and such as... Figure 2 The voice command “Paste here into the folder corresponding to the employee” is shown in (4).

[0100] The opened interface 300 displays folders named "Work Documents," "Personal Information," etc., which will not be listed here. Figure 3 As shown in (1), the user can continue to perform the operation of opening the folder 301 named "Personal Information" while simultaneously inputting a voice command to the device, "There is a subfolder called 'Personal Information' here." In response to this operation, as shown... Figure 3As shown in (2), the device displays an opening interface 310 for the "Personal Information" folder 301. The user can then create a new folder within the opening interface 310. Optionally, such as... Figure 3 As shown in (2), users can also create a new folder by right-clicking to bring up the new folder control 311. Of course, other operations can also be used to create a new folder. During this process, such as Figure 3 As shown in (2), the user simultaneously inputs the voice command "Create a new folder here" into the device. The device then responds to this operation as follows: Figure 3 As shown in (3), the device displays a newly created folder 320.

[0101] Next, the user renamed the newly created folder 320 to "Job Application Resume". During this renaming process, as follows... Figure 3 As shown in (3), the user simultaneously input the voice command "name it as onboarding resume". In response to this operation, as shown... Figure 3 As shown in (4), the device renames the newly created folder 320 to "Job Application Resume". Then, the user... Figure 3 The “Resume” folder 330 shown in (4) was opened, and in response to this operation, as follows: Figure 4 As shown in (1), the device displays the "Resume" folder opening interface 400.

[0102] The user continues to paste the previously copied Xiaohong's resume 202 into the opened interface 400. For example, the user can right-click on the opened interface 400 to bring up, for example, Figure 4 The paste control 401 shown in (1) allows users to paste Xiaohong's resume 202 by clicking the paste control 401 with the mouse. Alternatively, users can paste Xiaohong's resume 202 through other methods, including but not limited to using the keyboard shortcut "ctrl+v" or, if the device is a touchscreen, clicking the touchscreen to bring up the paste control. During the paste operation, such as... Figure 4 As shown in (1), the user simultaneously inputs a voice command into the device: "Put the resume I just copied here." In response to this operation, as shown... Figure 4 As shown in (2), the device copies Xiaohong's resume 202 into the opened interface 400.

[0103] At this point, the user's demonstration operation has concluded. In some embodiments, to facilitate the device's identification of demonstration operations and determine which operations are demonstration operations, the device may also, upon receiving an instruction indicating the end of demonstration operation execution (or described as information indicating the end of the first user operation), identify user operations preceding that instruction as demonstration operations. In other words, operations performed after the instruction indicating the end of demonstration operation execution are considered not demonstration operations. For example, the aforementioned instruction indicating the end of demonstration operation execution may also be a voice instruction, a gesture instruction, or, through means such as... Figure 1 The input device 180 shown inputs action commands, etc. Again, taking voice commands as an example, such as... Figure 4 As shown in (2), the user can input voice commands such as "like this, help me with batch operations" into the device. Accordingly, after the device recognizes the voice command, it can determine that the demonstration operation has been completed.

[0104] It's understandable that the above demonstration uses file distribution and storage as an example of batch operations. In real-world applications, there may be many more scenarios involving batch operations. In some scenarios, batch operations can be performed within the same application.

[0105] For example, Table 1 shows examples of operation steps and voice commands included in the demonstration operation in a scenario where batch operations provided in the embodiments of this application are executed in the same application.

[0106] Table 1

[0107]

[0108] As shown in Table 1, in a batch operation scenario where each employee creates a new folder and two subfolders, the steps in the demonstration operation can include, in sequence: "Open the employee list and copy a name," "Navigate to a specific path and create a new folder," "Rename the new folder with the copied name," and "Create two more folders within the new folder and rename them 'ID Photo' and 'Documents,'" etc. The voice commands in the demonstration operation can include, in sequence: "Copy a name from the employee list," "Create a new folder at this path," "Name this folder with the employee's name," and "Create the two subfolders 'ID Photo' and 'Documents' here," etc., which will not be described in detail here.

[0109] In other embodiments, batch operations may be performed across applications. For example, Table 2 shows examples of operation steps and voice commands included in exemplary operations in scenarios where batch operations are performed across applications, as provided in the embodiments of this application.

[0110] Table 2

[0111]

[0112]

[0113] As shown in Table 2, in a batch operation scenario of sending customized emails to each customer, the steps in the demonstration operation can include, in sequence: "Open the customer list containing names and email addresses," "Create a new email and copy the email address of the first customer to the recipient field," "Copy and paste the pre-written email content," and "Copy the customer name from the customer list and modify the customer name in the subject, salutation, and body." The voice commands in the demonstration operation can include, in sequence: "Send an email to each customer in the customer list," "Use this template for the content," and "Fill in this customer's name in these places," etc., which will not be described in detail here.

[0114] Table 2 only shows two examples of application scenarios for performing batch operations across applications. In practice, many more scenarios can be included. For example: downloading each person's photo from attachments and saving it to each person's subfolder; downloading all attachments and images from a reimbursement website and pasting them into a PowerPoint presentation; splitting an Excel spreadsheet by content or components for information distribution; downloading each student's assessment results from an online questionnaire website and archiving them by class, etc.

[0115] Understandable. Figure 2 Middle (3) to Figure 4 The example shown in (2) is based on the execution of batch operations within the same application, namely the file manager. Taking the scenario of sending customized emails to each customer as an example, this paper introduces the demonstration of the operations performed on the user when batch operations are executed in different applications.

[0116] For example, as shown in Figure 5a(1), the device can display an interface 500, which displays batch data 501, including the names of different customers and the email addresses of each customer. In this example, the interface 500 can be presented by the device in response to the user's opening operation of the customer list 502 shown in Figure 5a(1). The interface 500 can be the running interface of an office application, etc. As shown in Figure 5a(1), the user can input a voice command such as "Help me send emails to these customers, watch carefully, I will only demonstrate once" to instruct the demonstration operation to start. In response to the voice command, the device begins to record the user's demonstration operation and can output a reminder message 510 as shown in Figure 5a(2).

[0117] Taking sending an email to Xiaobai as an example, the process of this demonstration operation can be shown in Figure 5a (3) to Figure 5c (4). First, as shown in Figure 5a (3), the user can perform a copy operation on the email address "xiaobai@123.com". For example, the user can select the email address "xiaobai@123.com" with the mouse and right-click to bring up the copy control 520 shown in Figure 5a (3). The user can click on the copy control 520 with the mouse to copy the email address "xiaobai@123.com". For the implementation of other copy operations, please refer to the implementation described above. During the execution of this copy operation, the user can simultaneously point the mouse to the display position of the email address "xiaobai@123.com" and input the voice command "First, copy this email address" as shown in Figure 5a (3) to the device. In response to the aforementioned copy operation, the device copies the email address "xiaobai@123.com". Correspondingly, the operation step that the device can recognize is "copy 'xiaobai@123.com'", and the device can output operation step 530 as shown in Figure 5a (4). For an introduction to the operation steps output in this scenario, please refer to the following text. Figure 2 Middle (3) to Figure 4 The demonstration operation shown in (2) introduces the operation steps of the device output. Similarly, in this scenario, how the device performs batch operations after the demonstration operation is completed can also be referred to the introduction of batch operation implementation below.

[0118] Furthermore, the user can perform operations such as clicking the email application icon 531 shown in Figure 5a (4) (this operation is not shown in Figure 5a (4)). In response to this operation, the device can display the email application's running interface 540 as shown in Figure 5b (1). Then, the user can perform operations such as clicking the "Compose Email" control 541 displayed in the running interface 540 to create a new email, and at the same time input a voice command such as "Create email here" as shown in Figure 5b (1) to the device. In response to the user's operation of creating a new email, the device can display the email editing interface 550 as shown in Figure 5b (2) for the user to fill in the recipient, email content, etc. Correspondingly, the operation step that the device can recognize is "Create email", and the device can output the operation step 551 as shown in Figure 5b (2).

[0119] Next, the user copied the email address earlier. xiaobai@123.com"Paste into the recipient field of the email editing interface 550. For the implementation of the paste operation here, please refer to the pasting implementation described above. During the paste operation, as shown in Figure 5b (3), the user simultaneously inputs the voice command "Paste the address you just copied here" into the device. In response to this operation, as shown in Figure 5b (3), the device will " xiaobai@123.com "Paste to the recipient. Accordingly, the operation steps that the device can recognize are "paste xiaobai@123.com to the recipient of the new email", and the device can output the operation steps 560 shown in Figure 5b (4).

[0120] Then, as shown in Figure 5c(1), the user continues to fill in the email subject and email body in the email editing interface 550. For example, the user can open the template 570 shown in Figure 5c(1) and directly copy the email subject and email body saved in the template 570 to the corresponding position in the email editing interface 550 (this operation is not shown in Figure 5c(1)). At the same time, while filling in the email subject and email body, as shown in Figure 5c(1), the user simultaneously inputs the voice command "Fill in the subject and body" to the device. In response to the aforementioned operation of the user filling in the email subject and email body, the operation steps that the device can recognize are "Paste the email body and email subject in the template into the new email", and the device can output the operation steps 580 shown in Figure 5c(2).

[0121] Next, the user can copy the name "Xiaobai" displayed on interface 500 and replace "xx" as shown in Figure 5c (1). During this process, the user can simultaneously input the voice command "Modify this" into the device (the aforementioned operation is not shown in Figure 5c). In response to the aforementioned operation, the device can recognize the operation step as "Replace xx in the email body and email subject with Xiaobai in the customer list", and the device can output the operation step 590 as shown in Figure 5c (3). Finally, the user can perform the operation of sending an email, such as clicking the send button 561 displayed on interface 550 as shown in Figure 5b (3) with the mouse to send the email. Correspondingly, in response to this operation, the device can recognize the operation step as "Send a new email", and the device can output the operation step 600 as shown in Figure 5c (4). At this point, the user's demonstration operation is completed. The demonstration operations shown in Figure 5a (3) to Figure 5c (4) span multiple applications such as office applications and email applications.

[0122] It is understood that the examples of batch operations executed in the same application and the examples of batch operations executed in different applications in this application can be referred to for mutual reference.

[0123] Optionally, before receiving a user's demonstration operation and performing batch operations on batch data based on that demonstration operation, the device can first enable the batch operation function. For example, the device can receive a user's operation to wake up the voice assistant, and in response to this operation, the device can wake up the voice assistant. After confirming that the voice assistant is enabled, the user can enter a voice command such as "Please save these resumes to each person's respective folder" to enable the batch operation function.

[0124] Alternatively, users can also enable the batch operation function by activating the shortcut tabs displayed on the interface through the voice assistant. This application embodiment does not limit the method of enabling the device's batch operation function. Optionally, when the device enables the batch operation function, the device may display as follows: Figure 2 , Figure 3 The controls shown are used to display various information during batch operations (such as, but not limited to, received voice commands, recognized operation steps, and messages confirming information to the user). Figure 2 The rectangular control 20 shown in (1) enables the device to perform various interactions with the user in batch operations using the control.

[0125] In some embodiments, the device can determine the steps of a batch operation based on a demonstration operation, and then execute the batch operation based on the determined steps. The demonstration operation may include a series of operation steps. Each operation step may include an operation instruction and an operation object. The device can identify the operation instruction and operation object in the operation steps based on the demonstration operation.

[0126] The implementation of device identification operation commands will be introduced below. For example, Figure 5d This illustration shows a flowchart of a device identification operation instruction provided in an embodiment of this application. Figure 5d As shown, the process includes the following steps:

[0127] S501, The demonstration operation of the equipment has been initiated.

[0128] In some embodiments, the device may determine that a demonstration operation has begun upon receiving an instruction instructing the commencement of such operation. A description of this instruction can be found above.

[0129] S502, Equipment acquires demonstration operation information.

[0130] The demonstration operation information comprises various information obtained by the user during interaction with the device while performing a demonstration operation. This demonstration operation information can be generated based on the demonstration operation. For example, it may include at least one of the following: actions of the physical input device, voice commands, and interactive interface information. Including this rich information in the demonstration operation information improves the accuracy of the recognized demonstration operation.

[0131] For example, in this embodiment of the application, physical input devices include, but are not limited to, various physical devices such as mice, keyboards, touchpads, touchscreens, and styluses. The actions of a physical input device can refer to the operations performed by the user on the device when performing the demonstration operation, such as, but not limited to, right-clicking, sliding, left-clicking, key presses on the keyboard, and touch operations on the stylus / touchpad / touchscreen. By controlling the device through these operations, the demonstration operation can be executed. Specifically, when the user uses a physical input device to control the device to perform the demonstration operation, the demonstration operation information may include the actions of the physical input device. Conversely, when the user does not use a physical input device to control the device, the demonstration operation information may not include the actions of the physical input device.

[0132] Voice commands can include natural speech spoken by the user, carrying various information related to the demonstrated operation, such as pronouns (e.g., "this," "here," "these," "these few," "it," "they"), the object of the operation (e.g., "resume," "folder," "table," "image," "data," "application," "interface"), operation instructions (e.g., "copy," "open," "paste," "rename"), and purpose (e.g., "copy it," "create a new folder here," "put in their respective folders"). For example, voice commands can be like... Figure 2 The voice command shown in (3) is "First, copy the resume here," as in Figure 3 The voice command shown in (1) is "There is a subfolder called 'Personal Information' here". Figure 3 The example shown in (2) is "Create a new folder here". When a user interacts with the device using voice commands to perform a demonstration operation, the demonstration operation information may include the voice commands. However, when the user does not interact with the device using voice commands, the demonstration operation information may not include the voice commands.

[0133] The interactive interface information may include one or more interfaces (or windows) presented on the device during the user's demonstration operation. Optionally, this interactive interface information can be obtained through screen recording or screenshots. As the demonstration operation is performed, the interfaces presented on the device and the content within those interfaces will change accordingly. Throughout the demonstration operation, regardless of the method the user uses to interact with the device, the demonstration operation information can include the interactive interface information.

[0134] S503, the equipment identifies operation commands based on the demonstration operation information.

[0135] The device can recognize operation commands based on at least one of the following: actions from the physical input device, voice commands, and interactive interface information. For example, operation commands may include, but are not limited to, various commands such as open, copy, paste, rename, create, edit, and download.

[0136] For example, regarding actions on a physical input device, the device can identify the operation command based on the included operation events. These operation events can include at least one of the following: operation type, operation location on the screen, and operation content on the screen. Taking a keyboard as an example, the operation type can be a key icon, the operation location can be the screen area where the key is pressed, and the operation content can be the content entered by the key. Taking a mouse as an example, the operation type can be a single mouse click (e.g., left click, right click), multiple mouse clicks, mouse scrolling, etc., the operation location can be the position where the mouse cursor hovers on the screen, and the operation content can be the content selected by the mouse, the content clicked by the mouse, etc.

[0137] For example, for voice commands, the device can identify the presence of an operation command based on whether it contains operation command keywords, and it can also identify the type of operation command based on the operation command keywords included in the voice command. It is understood that different operation commands can correspond to different operation command keywords. For example, operation command keywords could be such as "open," "copy," "paste," or "rename." Optionally, the device can use automatic speech recognition (ASR) technology, natural language processing (NLP) technology, or text-to-speech (TTS) technology to process the voice commands in order to identify the operation command keywords within the voice commands.

[0138] For example, regarding interactive interface information, the device can identify operation commands by observing whether the content changes across different interfaces (which could be different frames from the same screen recording or different screenshots) and the content of those changes. Optionally, the device can employ various technologies such as neural networks, deep learning (e.g., the Transformer model), and artificial intelligence (AI) to process and identify interactive interface information.

[0139] Optionally, when the demonstration operation information includes multiple types of information such as actions from physical input devices, voice commands, and interactive interface information, the device can recognize the operation commands based on each type of information separately, and then summarize the operation commands recognized based on each type of information to obtain the final result. The operation commands recognized based on different types of information can be completely identical, completely different, or partially identical.

[0140] In some embodiments, steps S502 to S503 can be executed periodically or sequentially. That is, during the user's demonstration operation, the device can synchronously execute steps S502 to S503, i.e., synchronously identify the operation command. Thus, during the demonstration operation, if the user executes an operation command, the command can be promptly identified by the device, and the identified command can be promptly output to the user for confirmation. In other words, the device can progressively identify operation commands. Therefore, when the demonstration operation ends, the identification of all operation commands in that process can also be completed, allowing for immediate commencement of batch operations, thereby improving the efficiency of batch operations.

[0141] For example, assuming the device starts recognizing operation commands from time t and executes the recognition every 0.5 milliseconds, the device can perform the first recognition operation at time t+0.5 milliseconds. The resulting example operation information obtained by the device can be the example operation information corresponding to the operation from time t to time t+0.5 milliseconds. Then, the device can perform the second recognition operation at time t+1 milliseconds, and the resulting example operation information can be the example operation information corresponding to the operation from time t+0.5 milliseconds to time t+1 milliseconds. And so on. In this embodiment, the device can identify operation commands existing within a certain time period by periodically acquiring example operation information over that time period. Optionally, there may be one or multiple operation commands.

[0142] Optionally, in this embodiment, the demonstration operation information acquired by the device over a certain period may simultaneously include multiple elements such as the actions of the physical input device, voice commands, and interactive interface information. The timing of acquiring this demonstration operation information can be the same. For example, if a user performs a demonstration operation using both a physical input device and a voice command at the same time (e.g., at the same instant or within the same time period), the device can acquire the actions of the physical input device, voice commands, and interactive interface information at the same time. Thus, when the device identifies operation commands based on each type of information, it can also determine whether the operation commands identified based on these different types of information acquired at the same time are consistent, thereby improving the accuracy of operation command recognition. Optionally, the device can perform the aforementioned judgment operation through various methods such as statistics, index tables, and network models.

[0143] If the operation instructions identified based on different types of information are consistent, the device can determine that the identified operation instructions are accurate. For example, if the operation instruction identified based on the physical input device's action at time t is "copy," and the operation instruction identified based on the voice command at time t is also "copy," then the device can determine that the operation instruction is "copy." Conversely, if the operation instructions identified based on different types of information are inconsistent, the device can determine that the identified operation instructions are inaccurate. For example, if the operation instruction identified based on the physical input device's action at time t is "copy," while the operation instruction identified based on the voice command at time t is "paste," then the device can determine that the identified operation instruction may be inaccurate.

[0144] In some embodiments, the device can also output (display on a screen or announce via voice) the recognized operation commands. Optionally, the device can perform an operation to output the recognized operation command after performing an operation command recognition operation. In this way, during the execution of the demonstration operation, if the user executes an operation command, the device can output the recognized operation command in real time, which can make batch operations more intelligent and improve the human-computer interaction experience. Of course, the device can also output the operation commands recognized in multiple recognition operations at the same time after performing multiple operation command recognition operations, or the device can output the recognized operation commands after the demonstration operation is completed. There is no restriction on the timing of the device outputting the recognized operation commands.

[0145] In some embodiments, when the device determines that the identified operation command is inaccurate (e.g., multiple identification results occur simultaneously, or the accuracy of the identified operation command does not meet the requirements), it can also confirm with the user whether the identified operation command is accurate. For example: combined with Figure 2In the operation of copying Xiaohong's resume 202 shown in (3), if the device recognizes the operation instruction as paste, the device can output to the user (either through the display screen or voice output) messages such as "You have been detected to have performed a paste operation. Please confirm if it is correct" or "Is this a paste operation?" to confirm with the user whether the recognized operation instruction is accurate.

[0146] Optionally, the device can also receive confirmation information from the user regarding the operation instruction. This confirmation information can be used to indicate whether the operation instruction is correct, and at least one of the correct operation instructions. The confirmation information can be a voice message or information input by the user into the device through a physical input device. For example, the voice message can be a voice message indicating that the operation instruction is correct, such as "correct," "this is it," "yes," "OK," or "that's right." It can also be a voice message indicating that the operation instruction is correct, such as "incorrect," "not this," "no," "NO," or "wrong." Furthermore, it can be a voice message indicating that the operation instruction recognized by the device is inaccurate, such as "incorrect, the operation instruction is xx" or "no, it is to perform xx operation," indicating that there is a correct operation instruction.

[0147] Information that users input into the device terminal through physical input devices can be such as “correct”, “OK”, “yes”, “√”, etc., to indicate that the operation command recognized by the device is correct. It can also be such as “incorrect”, “no”, “×”, “incorrect, the operation command is xx”, “the operation command is xx”, etc., to indicate that the operation command recognized by the device is incorrect, or it can also be used to indicate the correct operation command.

[0148] Optionally, when the device recognizes an incorrect operation command, the user can directly modify the erroneous operation command output by the device to the correct one. In this way, the device can also determine that the recognized operation command is inaccurate based on the user's operation.

[0149] Of course, in other embodiments, regardless of whether the operation command recognized by the device is accurate, the device can perform an operation to confirm with the user whether the recognized operation command is accurate. This can further improve the accuracy of the operation commands determined by the device.

[0150] S504. The demonstration operation of the equipment has been completed.

[0151] In some embodiments, the device may determine that the demonstration operation has ended upon receiving an instruction indicating that the demonstration operation has ended. A description of this instruction can be found above. It is understood that if the demonstration operation has not ended, the device may continue to return to steps S502 to S503 until the demonstration operation ends.

[0152] Figure 5d The embodiment shown is an example of synchronously identifying operation instructions during the execution of a demonstration operation. In other embodiments, the device may also uniformly identify the operation instructions included in the demonstration operation after the demonstration operation is completed.

[0153] The implementation of the recognition operation instructions has been introduced above. The implementation of the recognition operation objects will be introduced below.

[0154] It is understandable that the object of the operation can be data in various storage formats, such as, but not limited to, PDFs, text documents, tables, images (in various formats), audio, video, folders, application installation packages, URLs, emails, applications, compressed files, etc. For example, the object of the operation could be a PDF file, a data segment in an Excel spreadsheet that meets a series of conditions, a webpage, or an email, etc.

[0155] In some embodiments, there may be a correspondence between operation instructions and operation objects; for a given operation object, there may be a corresponding operation instruction. The operation instruction corresponding to the operation object can be used to process that operation object. In some embodiments, the device can identify the operation object based on the operation instruction.

[0156] For example, in this embodiment, Figure 6 This application provides a schematic diagram of a device identification operation object according to an embodiment of the present application. Figure 6 As shown, the process includes the following steps:

[0157] S601, The demonstration operation of the equipment has been initiated.

[0158] For an explanation of step S601, please refer to [link / reference]. Figure 5d The following is a description of step S501.

[0159] S602, The device has confirmed that it has recognized the operation command.

[0160] Optionally, the operation instructions in step S602 can be based on the device... Figure 5d The method shown identifies the object. Optionally, when the device recognizes an operation command, it may perform the operation to identify the operation object. When the device does not recognize an operation command, it may not perform the operation to identify the operation object.

[0161] S603, the device identifies the object to be operated based on the operation command.

[0162] In some embodiments, after recognizing an operation command, the device can obtain sample operation information corresponding to the operation command. For example, the sample operation information corresponding to the operation command may be sample operation information within a certain period of time in which the operation command is located. Optionally, the sample operation information may also include at least one of the following: actions of the physical input device, voice commands, and interactive interface information. For a description of these information, please refer to [reference needed]. Figure 5d The relevant description at step S502 is shown below. Optionally, the example operation information corresponding to different operation commands can be different. The device can identify the operation object corresponding to the operation command based on this example operation information.

[0163] For example, regarding actions performed by physical input devices, the device can also identify the object of the operation based on the operation events included therein. A description of these operation events can be found in step S503. Specifically, the device can identify the object of the operation based on the content displayed on the screen, such as content selected by the mouse, content clicked by the mouse, content displayed in the mouse hover area, and content entered via buttons.

[0164] For example, for voice commands, the device can identify the target of the operation based on whether the voice command includes keywords indicating the target. These keywords can be pronouns, such as nouns including but not limited to "this," "here," "these," "these few," "it," and "they," which can refer to the target. The device can parse the pronoun in the voice command and the content displayed on the interface when the pronoun is received (such as, but not limited to, the file's save path, the content of a document, etc.) to obtain the target. Optionally, the content displayed on the interface when the pronoun is received can be determined based on information from the interactive interface. Of course, the device can also use other algorithms, such as neural networks or large AI models, to process the voice command and obtain the target.

[0165] For example, for interactive interface information, the device can also use optical character recognition (OCR) to process the image to identify the object being operated on. Alternatively, the device can directly obtain the interactive interface information and the objects being operated on through internal system operations.

[0166] Optionally, when the demonstration operation information corresponding to the operation instruction includes multiple pieces of information such as the actions of the physical input device, voice commands, and interactive interface information, the device can combine multiple pieces of this information to identify the operation object, thereby improving the accuracy of the identified operation object.

[0167] In some embodiments, the device can perform an operation command recognition operation once, and after recognizing the operation command, it can perform an operation object recognition operation to identify the operation object corresponding to the recognized operation command, thus enabling the recognized operation command and operation object to be matched. In this way, combined with... Figure 5d In the example shown, if the operation instruction identification operation is executed at a time or periodically, steps S602 to S603 can also be executed at a time or periodically. That is, during the user's demonstration operation, the device can synchronously execute steps S602 to S603, i.e., synchronously identify the operation object.

[0168] In this way, during the execution of the demonstration operation, if the user performs an operation on a certain object, the object can be promptly identified by the device, and the identified object can be output to the user for confirmation in a timely manner. Thus, when the demonstration operation ends, the identification of the object in the demonstration operation process is also completed, allowing batch operations to begin immediately, thereby improving the efficiency of batch operations.

[0169] For example: combining Figure 5d The example shown illustrates a scenario where the device begins recognizing operation commands at time t, executing this process every 0.5 milliseconds. The device can perform the first operation command recognition at time t+0.5 milliseconds. After recognizing several operation commands through this operation, the device can then perform the first operation object recognition operation based on these commands. Next, the device can perform the second operation command recognition at time t+1 milliseconds. After recognizing several more operation commands through this operation, the device can then perform the second operation object recognition operation based on these newly recognized commands. Optionally, in each operation object recognition operation, the device can identify one or more operation objects.

[0170] In some embodiments, the device may also output (display on a screen or announce via voice) the recognized object. Optionally, after performing an object recognition operation, the device may perform an operation to output the recognized object.

[0171] In this way, during the execution of the demonstration operation, if the user performs an operation on a certain object, the device can output the identified object in real time. This makes batch operations more intelligent and improves the human-computer interaction experience. Of course, the device can also output the operation objects identified in multiple operation recognition operations at the same time after the demonstration operation is completed. Alternatively, the device can output the identified operation objects after the demonstration operation is completed. There are no restrictions on when the device outputs the identified operation objects.

[0172] Optionally, operation instructions and operation objects can be output simultaneously. For example, operation instructions and their corresponding operation objects can be output at the same time. That is, the device can construct the operation instructions and their corresponding operation objects into a complete operation step and output it to the user. (For example, combining...) Figure 2 The demonstration operation of copying Xiaohong's resume 202 shown in (3) assumes that the operation instruction recognized by the device is "copy" and the operation object is "resume-Xiaohong.pdf". The device can output information such as "copy resume-Xiaohong.pdf" which includes both the operation instruction and the corresponding operation object.

[0173] Of course, operation instructions and operation objects can also be output separately, such as when combined. Figure 2 The demonstration operation of copying Xiaohong's resume 202 shown in (3) assumes that the operation instruction recognized by the device is "copy" and the operation object is "resume-Xiaohong.pdf". The device can output information such as "execute copy operation" and "operation object is resume-Xiaohong.pdf" which respectively include the operation instruction and the operation object. There is no restriction on the order of output of the operation instruction and the operation object.

[0174] In some embodiments, the device may also confirm with the user whether the identified object of operation is accurate. For example: in conjunction with Figure 2 In the example operation of copying Xiaohong's resume 202 shown in (3), the device can output messages to the user (displayed on the screen or announced by voice) such as "It was detected that you performed an operation on resume-Xiaohong.pdf. Please confirm whether it is accurate" or "Is the operation performed on resume-Xiaohong.pdf?" to confirm with the user whether the identified operation object is accurate.

[0175] Similarly, the device can also receive confirmation information from the user regarding the operation object. This confirmation information can be used to indicate whether the identified operation object is accurate or a correct operation object. The confirmation information for the operation command can be a voice message or information input by the user into the device via a physical input device. For the implementation of voice messages and information input into the device via a physical input device, refer to the similar implementations described above for indicating whether the operation command is correct and for correct operation commands.

[0176] Optionally, if the device identifies an inaccurate operation object, the user can modify the incorrect operation object output by the device to the correct one. In this way, the device can also determine that the identified operation object is inaccurate based on the user's action.

[0177] In some embodiments, the device can confirm with the user whether the operation instruction and the operation object are accurate separately. In other embodiments, the device can also confirm with the user whether they are accurate at the same time. For example, the device can directly output a message to the user that includes the operation instruction and the corresponding operation object, that is, directly output the operation steps consisting of the operation instruction and the corresponding operation object, so as to confirm with the user whether the operation instruction and the operation object are correct at the same time, that is, it can directly confirm with the user whether the identified operation steps are accurate.

[0178] For example: combining Figure 2 The demonstration operation of copying Xiaohong's resume 202 shown in (3) allows the device to output messages to the user such as "It was detected that you have performed a copy operation on resume-Xiaohong.pdf. Please confirm whether it is correct" or "Is it copying resume-Xiaohong.pdf?", so as to confirm with the user whether the identified operation instruction and operation object are correct, that is, to confirm with the user whether the operation steps are correct.

[0179] For example: combination Figure 3 When the user inputs the voice command "Paste this to the corresponding employee's folder" as shown in Figure (1), the device can recognize the operation command and the corresponding operation object based on the voice command, and then recognize the operation steps composed of the operation command and the operation object. Accordingly, such as Figure 7 As shown in (1), the device can output messages to the user such as "Do you want to paste into / Employee Information / Xiaohong?" 700 (which can be used as an example of the first reminder message) to confirm with the user whether the identified operation instructions and operation objects are correct, that is, to confirm with the user whether the identified operation steps are correct.

[0180] In this way, outputting the operation command and its corresponding operation object simultaneously to the user for confirmation—that is, outputting the complete identified operation steps to the user for confirmation—can confirm the correct operation steps and improve the accuracy of operation step recognition. When subsequent devices perform batch operations based on these correct operation steps, the accuracy of batch operations can be improved. Furthermore, compared to separate confirmation methods, fewer information outputs are required, avoiding repeated disturbances to the user and improving the efficiency of human-computer interaction. By progressively recognizing operation steps and confirming their correctness with the user step by step, intelligent recording of demonstration operation processes can be achieved, improving the accuracy of recognition results.

[0181] Similarly, the device can also receive confirmation information from the user regarding operation instructions, operation objects, and operation steps. This confirmation information can be used to indicate whether the operation instructions or operation objects are accurate, whether the operation steps are correct, and at least one of the following: correct operation instructions, correct operation objects, and correct operation steps. For example: combined with Figure 7As shown in (1), "Do you want to paste it into / Employee Information / Xiaohong?" 700, Figure 7 As shown in (2), the device can receive voice commands 710 (which can be used as an example of a second user operation) such as "No, I want to find a subfolder called 'Personal Information' in the corresponding employee's folder" input by the user, and is used to indicate at least one of the following: incorrect operation steps and correct operation steps.

[0182] Optionally, the device can present as follows: Figure 7 After the voice command 710 shown in (2), you can continue to receive commands such as Figure 3 The user operation shown in (1) to open the folder named "Personal Information" 301, and as shown in (1) Figure 3 The voice command "Create a new folder here" shown in (2) is an example. The device can determine the correct operation steps based on user input instructions to indicate whether the operation steps recognized by the device are accurate, demonstration steps performed by the user, etc.

[0183] Optionally, when the device recognizes that the operation instructions or operation objects in the operation steps are incorrect (or described as incorrect operation steps), the user can also directly modify the operation instructions or operation objects in the operation steps output by the device to the correct operation instructions or operation objects, or describe it as directly modifying the operation steps output by the device to the correct operation steps.

[0184] For example: combining Figure 7 The message "Do you want to paste it into / Employee Information / Xiaohong?" shown in (1) 700, the operation instructions (such as "paste") and the operation object (such as "Employee Information / Xiaohong") included in the operation steps can be editable. This editable format allows the user to perform editing operations to modify the content. For example, if the user confirms that the operation object "Employee Information / Xiaohong" is incorrect, they can perform operations such as clicking the mouse on the operation object. Then, in response to this operation, such as Figure 7 As shown in (3), the object is in an editable state, and the user can edit it and modify it as follows. Figure 7 The operation object shown in (3) is “ / Employee Information / Xiaohong / Personal Information / Resume” (which can be used as an example of a second user operation). Accordingly, based on the user's operation, the device can obtain the correct operation object and thus obtain the correct operation steps.

[0185] S604, The demonstration operation of the equipment has been completed.

[0186] For an explanation of step S604, please refer to [link / reference]. Figure 5dThe following describes step S504. Similarly, if the demonstration operation is not completed, the device may continue to return to steps S602 to S603 until the demonstration operation ends.

[0187] Figure 6 The illustrated embodiment exemplifies the synchronous identification of the operation object during the execution of the demonstration operation. In other embodiments, the device may also perform unified identification of the operation objects included in the demonstration operation at the end of the demonstration operation. There are no restrictions on the timing of the device performing the identification corresponding to the operation.

[0188] Figure 6 The illustrated embodiment uses the identification of the operation object based on the operation command as an example, which facilitates the correspondence between the identified operation object and the operation command. In other embodiments, the device can also directly identify the operation object based on the demonstration operation information. For example, the device can use various algorithms such as neural networks, deep learning, and large AI models to identify the operation object based on the demonstration operation information.

[0189] Furthermore, the device can determine the operational steps included in the demonstration operation, i.e., the flow of the demonstration operation, based on the recognized operation instructions and the corresponding operation objects. For example: combining... Figure 2 Middle (3) to Figure 4 The demonstration operation shown in (2) includes the following steps in sequence: copying Xiaohong's resume 202, opening the folder named "Employee Information", opening the folder named "Xiaohong" 231, opening the folder named "Personal Information" 301, creating a new folder 320, renaming the new folder to "Job Application Resume", opening the "Job Application Resume" folder, and pasting Xiaohong's resume 202, etc.

[0190] In the above embodiments, the device first identifies the operation command and the operation object, and then determines the operation steps included in the demonstration operation based on the identified operation command and operation object. In other embodiments, the device can also directly determine the operation steps included in the demonstration operation based on the demonstration operation information using various AI algorithms such as neural networks and deep learning.

[0191] In some embodiments, the device can output the steps of the identified demonstration operation to the user for viewing, such as by outputting a first reminder message. This makes it easier for the user to verify whether the operation steps identified by the device are accurate.

[0192] In this embodiment, as one possible implementation, the device can output all the operation steps in the determined demonstration operation to the user for viewing.

[0193] As another possible implementation, the device can output the determined partial operation steps (such as key operation steps) or the operation steps after summarizing them to the user. The partial operation steps output here, and the summarized operation steps, can also be implemented as the batch operation steps described later. This avoids outputting too many operation steps, which would negatively impact the human-computer interaction experience.

[0194] In this embodiment, combined with Figure 2 Middle (3) to Figure 4 The demonstration operation shown in (2) is as follows: Figure 2 As shown in (3), when the user performs a copy operation on Xiaohong's resume 202 (which can be used as an example of the first operation in the first user operation), the operation steps that the device can recognize are "copy resume-Xiaohong.pdf". Accordingly, in response to the user's copy operation on Xiaohong's resume 202, the device can output as follows: Figure 8 The operation step 800 shown in (1) can be used as an example of the first processing step.

[0195] like Figure 3 As shown in (2), when a user performs the operation of creating a new folder in the open interface 310 (which can also be used as an example of the first operation in the first user operation), the operation steps that the device can recognize are "create a new folder in / personal information". Accordingly, in response to the user's operation of creating a new folder in the open interface 310, the device can output as follows: Figure 8 The operation step 810 shown in (2) can also be used as an example of the first processing step. Optionally, when the device outputs the operation step determined later, it can also simultaneously display the operation steps that have been output earlier, which will be explained uniformly here. That is to say, the device can display the complete flow of operation steps, which can help the user confirm whether the execution order of the operation steps is accurate. For example Figure 8 As shown in (2), the device also displays the previously output operation steps "Copy resume from / resume-Xiaohong.pdf" 811.

[0196] Understandable. Figure 8 The operation steps 800 shown in (1) are... Figure 8 The operation steps 811 shown in (2) are the same operation steps. However, since Xiaohong's resume 202 and the newly created folder 320 are in different storage paths, in order to make the output operation steps clearer and more accurate, the device will... Figure 8 The presentation of operation steps 800 in (1) has been updated to... Figure 8 The presentation of operation steps 811 shown in (2).

[0197] like Figure 3As shown in (3), the user renames the newly created folder 320 to "Job Application Resume". The operation step that the device can recognize is "Rename the newly created folder to "Job Application Resume". Accordingly, in response to the user's operation of renaming the newly created folder 320 to "Job Application Resume", the device can output as follows: Figure 8 The operation steps 820 shown in (3) are as follows.

[0198] like Figure 4 As shown in Figure (1), when the user pastes Xiaohong's resume 202 into the "Job Application Resume" folder, the device recognizes the operation step as "Paste Resume - Xiaohong to / Job Application Resume". Correspondingly, in response to the user's operation of pasting Xiaohong's resume 202 into the "Job Application Resume" folder, the device can output the following: Figure 8 The operation steps 830 shown in (4) are as follows.

[0199] Among them, such as Figure 8 in (1) to such as Figure 8 The message shown in (4) that contains “We have detected the following actions of you and the steps to take” can be used as an example of a first reminder message.

[0200] Optional, such as Figure 8 The operation steps output by the device can be modified by the user, including but not limited to modifying the content of the operation steps (such as operation instructions, operation objects, etc.), deleting operation steps, adding operation steps, and changing the order of operation steps. In this way, users can directly edit the operation steps output by the device, which can simplify the process of correcting the device's recognition results, save time, and improve efficiency.

[0201] by Figure 8 For example, in the case of (1), Figure 8 The operation steps 800 shown in (1) can be presented in an editable format, which allows users to modify the content in operation steps 800. Alternatively, only the operation instructions and operation objects in operation steps 800 can be presented in an editable format (e.g., using at least one of bold font, underline, etc. to represent an editable format, or other representation styles can also be used) to support user modification.

[0202] For example: Figure 8 As shown in (1), the device can also present an undo control 801, which can be used to support the user in deleting operation step 800, or to support the user in undoing at least one of operation step 800 and the steps after operation step 800, so that the device can return to the state before or when operation step 800 was executed. For example: Figure 8As shown in (1), the device can also present an add control 802. The operation control 802 can be used to support the user to add a new operation step before, after, or between two steps.

[0203] Figure 8 Middle (1) to Figure 8 In the example shown in (4), the device synchronously outputs the determined operation steps during the user's demonstration operation. Optionally, the device can output the operation step once it recognizes it. In this way, if the user performs an operation step during the demonstration operation, the device can output the recognized operation step in real time, making batch operations more intelligent and improving the human-computer interaction experience. Of course, the device can also output all recognized operation steps at once after recognizing multiple operation steps. Alternatively, the device can output the recognized operation steps after the demonstration operation is completed, without any restrictions on the timing of the device's output of the recognized operation steps.

[0204] Furthermore, when the demonstration operation is completed, the device can perform batch operations based on the determined operation steps included in the demonstration operation. The implementation of batch operations is described below.

[0205] In some embodiments, when the demonstration operation finishes execution, the device can determine when to begin executing batch operations. As one possible implementation, batch operations can begin automatically within a preset time elapsed since the demonstration operation ended.

[0206] As another possible implementation, the device can begin performing batch operations upon receiving an instruction to initiate such operations. For example, this instruction could be a voice command, a gesture command, or an action command input via a physical input device. Taking a voice command as an example, for instance, combined with... Figure 2 Middle (3) to Figure 4 The demonstration operation in (2) shows that voice commands can be used as follows: Figure 4 As shown in (2), “Like this, help me with batch operations”, the device begins to perform batch operations in response to the user’s voice command. It is understood that in this example, the voice command used to indicate the end of the demonstration operation and the voice command used to indicate the start of the batch operation are the same. In other embodiments, the voice command used to indicate the end of the demonstration operation and the voice command used to indicate the start of the batch operation may be different.

[0207] In some embodiments, the device may determine the object of the batch operation, i.e., the range of batch data, before performing the batch operation. As one possible implementation, the device may receive an operation specifying the object of the batch operation, and in response to this operation, the device may determine the object of the batch operation. Exemplarily, the operation specifying the object of the batch operation may be a voice command, a gesture command, or an action command input via a physical input device. For example, taking a voice command as an example, combined with... Figure 2 In the example shown in (1), the voice command for performing a batch operation could be something like, "Help me save these resumes into each person's individual folder" (which could be an example of a third-user operation). In response to this voice command, the device can determine that the object of the batch operation is... Figure 2 The batch data 201 shown in (1) includes resumes.

[0208] Optionally, when determining the objects to be executed in a batch operation, the device may also determine the objects to be executed in a batch operation based on the operation used to specify the objects to be executed in the batch operation, and the content presented on the interface when the operation is received.

[0209] In some embodiments, the device may confirm with the user whether the identified batch operation objects are accurate, or describe it as confirming with the user the identified batch operation objects. For example: in combination with Figure 2 Middle (3) to Figure 4 In the scenario shown in (2), a batch operation is performed on batch data 201. This occurs after the device receives a voice command such as "Like this, perform a batch operation for me." In response to this voice command indicating the start of the batch operation, such as... Figure 9 As shown in (1), the device can output a reminder message 900 to the user (which can be used as an example of a second reminder message). The reminder message 900 includes the identified batch operation execution objects 902, such as "Resume-Xiaoming.pdf", "Resume-Xiaolan.pdf", etc., a total of 4 resumes, and information 901 used to remind the user to confirm whether the batch operation execution objects are accurate.

[0210] Optionally, the device can also receive confirmation information from the user regarding the objects of the batch operations identified by the device. This confirmation information can be used to indicate whether the objects identified by the device for batch operations are accurate, and at least one of the correct objects. The confirmation information for the operation instructions can be a voice message or information input by the user into the device through a physical input device.

[0211] For example: taking voice messages as an example, combined with Figure 9The reminder message 900 described in (1) is used when the user confirms that the batch operation execution objects 902 included in the reminder message 900 are correct. The device can receive user input such as "Yes, that's it" (...). Figure 9 (2) Taking this as an example), voice commands such as "Yes" and "No problem" (which can be used as examples of fourth user operations) are used to indicate that the object of the batch operation recognized by the device is correct. When the user confirms that the object of the batch operation recognized by the device is incorrect, the user can also perform the operation of selecting the correct object of the batch operation.

[0212] Optionally, the operation of selecting the correct batch operation execution object can be to directly change the batch operation execution object output by the device, or it can be to jump back to another display interface to reselect (such as selecting by screenshot or selecting by cursor within the first time period) the batch operation execution object (which can also be used as an example of the fourth user operation).

[0213] For other implementations of voice messages, information input into the device via a physical input device, etc., please refer to the similar implementations of voice messages and information input into the device via a physical input device described above for indicating whether an operation command is correct and at least one of correct operation commands.

[0214] Furthermore, once the device identifies the objects to be processed in a batch (hereinafter referred to as batch objects), the device can begin performing batch operations on these objects. For example, Figure 10 A flowchart illustrating an object manipulation method provided in an embodiment of this application is shown. Figure 10 As shown, the method includes the following steps:

[0215] S1001, Keyword for the batch processing object extracted by the equipment.

[0216] In some embodiments, the device can determine the main keyword to be replaced in each step of a batch operation. In this application embodiment, there is no limitation on the timing of the device determining the main keyword to be replaced. The steps included in the batch operation can be obtained by summarizing the operation steps determined by the device, and the steps of the batch operation can be part or all of the operation steps of the demonstration operation. Compared to the determined operation steps of the demonstration operation, the steps of the batch operation summarized based on the determined operation steps of the demonstration operation can be more accurate and concise.

[0217] For example, a user might perform some incorrect steps during a demonstration operation, but then cancel those incorrect steps. The steps for the demonstration operation determined by the device might include these incorrect steps. However, the steps for the batch operation derived by the device based on these demonstration operation steps may not include these incorrect steps.

[0218] Similarly, before performing batch operations, the device can also output the steps of the batch operation to the user for viewing. For example, combined with... Figure 2 Middle (3) to Figure 4 The steps for batch operations output by the device, as shown in the demonstration operation in (2), can also be performed as follows. Figure 8 The operation steps 800 to 830 are shown in (4). For the implementation of the output batch operation steps, please refer to the implementation of the device outputting the determined operation steps described above.

[0219] In this process, the main keywords to be replaced in each step are the information that the device will replace during batch operations. It is understood that some steps may contain main keywords that need to be replaced, while others may not. For a single step, there can be one or more main keywords to be replaced. Optionally, the main keywords included in different steps can be the same or different; when multiple main keywords are included in the same step, these multiple main keywords may be different. For example, main keywords can include, but are not limited to, information with various attributes such as name, title, age, time, quantity, email address, email subject, email body name, recipient address, patent / document title, and storage path.

[0220] For example: combining Figure 8 The example shown in (4) uses the following steps in a batch operation: "Copy resume-Xiaohong.pdf from / Resume", "Create a new folder in / Personal Information", "Rename the new folder to Onboarding Resume", and "Paste resume-Xiaohong to / Onboarding Resume". For the step "Copy resume-Xiaohong.pdf from / Resume", the main keyword to be replaced is "Resume-Xiaohong", and the attribute is the resume name. The steps "Create a new folder in / Personal Information" and "Rename the new folder to Onboarding Resume" do not include the main keyword to be replaced. For the step "Paste resume-Xiaohong to / Onboarding Resume", the main keyword to be replaced is "Resume-Xiaohong", and the attribute is also the resume name.

[0221] Optionally, the device can determine the main keywords to be replaced in each step of the batch operation based on the attributes of the main keywords input by the user. For example, the user can input the attributes of the main keywords into the device using various methods such as voice, gestures, or physical input devices. For instance, using voice, the user can input a voice command such as "name according to time + name + ID photo," carrying the attributes of the main keywords. In response to this command, the device can determine that the attributes of the main keywords to be replaced are "time" and "name." Of course, the device can also calculate the main keywords to be replaced in each step based on other algorithms (such as, but not limited to, large AI models, neural networks, etc.).

[0222] In some embodiments, for each batch processing object, the device can extract the main keywords of that batch processing object. In some embodiments, the device can extract the main keywords of the object based on the batch processing object's storage attributes and the attributes of the main keywords to be replaced in each step of the batch operation.

[0223] For example, the storage attributes of the processed object can include, but are not limited to, various parameters such as name, storage format, storage location, size, and display position. For example, combined with... Figure 8 Example of batch operation steps shown in (4): The device determines that the attribute of the main keyword to be replaced in these steps is resume name, with the batch processing object as... Figure 2 Taking the file “Resume-Xiaolan.pdf” shown in (1) as an example, the device can extract the main keywords of the object based on the name “Resume-Xiaolan.pdf”, namely “Resume-Xiaolan.pdf”. Taking the batch processing object as Figure 2 Taking “Resume-Xiaolan.pdf” as an example, the device can extract the main keyword of the object as “Resume-Xiaolan.pdf” based on the name of “Resume-Xiaolan.pdf”. For another example: combined with the above example where the attributes of the main keyword to be replaced are “time” and “name”, the device can extract the time and name of each batch object based on the attributes of the main keyword.

[0224] In some embodiments, when the device determines the subject keyword to be replaced in each step of a batch operation, or extracts the subject keyword of each batch object, the device may also confirm with the user whether the obtained subject keyword is correct.

[0225] For example, combining Figure 8In the example of the batch operation steps shown in (4), assuming the user determines that the attribute of the main keyword to be replaced in the batch operation is the resume name, the device can output a message to the user such as "The keyword to be replaced is the resume name. Please confirm whether it is correct" to confirm whether the obtained main keyword is correct. Alternatively, the device can also directly highlight, use different colored fonts, borders, etc. to directly highlight the main keyword in such a way as... Figure 8 The steps of the batch operation shown in (4) are marked. There are no restrictions on how the device confirms with the user whether the obtained main keywords are correct.

[0226] Optionally, the device can also receive confirmation information from the user regarding the main keyword. This confirmation information can be used to indicate whether the main keyword is correct, and at least one of the correct main keywords. The confirmation information for this operation instruction can be a voice message or information input by the user into the device via a physical input device. Optionally, when the device determines that the main keyword is incorrect, the user can directly change the main keyword output by the device to the correct one. In this way, based on the user's action, the device can obtain the correct main keyword.

[0227] 1002. The device performs batch operations based on the main keywords and batch operation steps.

[0228] In some embodiments, the device can sequentially replace the main keywords to be replaced in the batch operation steps with the main keywords extracted based on each batch processing object, and then execute the batch operation steps after replacing the keywords, thereby sequentially realizing the operation on each batch processing object. When the operation on all batch processing objects is completed, the batch operation can be realized.

[0229] For example, combining Figure 8 Example of batch operation steps shown in (4), the main keyword to be replaced in the batch operation steps is "resume-Xiaohong", and the batch processing object is as follows. Figure 2 Taking “Resume-Xiaolan” as an example, as shown in (1), the device is based on the extracted main keyword of the object, which is “Resume-Xiaolan”. The device can replace “Resume-Xiaohong” in the batch operation steps with “Resume-Xiaolan”. Correspondingly, the batch operation steps after replacing the main keyword include four steps: “Copy Resume-Xiaolan.pdf from / Resume”, “Create a new folder in / Personal Information”, “Rename the new folder to Onboarding Resume”, and “Paste Resume-Xiaolan to / Onboarding Resume”. Then the device can execute the batch operation steps to complete the processing of “Resume-Xiaolan”. This makes the device’s processing of Xiaolan’s resume consistent with the user’s processing of Xiaohong’s resume during the demonstration.

[0230] Similarly, for Figure 2 Except for "Resume-Xiaolan" shown in (1), the device can process each batch object according to the aforementioned process, and thus complete the processing of other batch objects such as... Figure 2 Batch operation of batch data 201 shown in (1).

[0231] In some embodiments, during the process of performing batch operations on batch processing objects based on batch operation steps, errors may occur, such as the inability to find a certain folder or a certain email address. In this embodiment, the device can use a fuzzy processing method, for example, searching for data similar to the missing data in the existing data and processing it. Alternatively, the device can output the error items that occur during the execution process to the user to confirm how to handle them. Optionally, the device can also continue to perform batch operations based on the user's confirmation information.

[0232] In some embodiments, to further improve the accuracy of batch operations, the device may first perform batch operations on a portion (which may be one or more) of the batch objects and confirm with the user whether the batch operations performed on that portion of the batch objects are correct. If the user confirms that the operations are correct, then the device performs batch operations on the remaining batch objects. Alternatively, if the user confirms that the operations are incorrect, the device may also receive a correction request from the user and then perform batch operations on some or all of the batch objects based on the user's correction request. Of course, in other embodiments, the device may also directly perform batch operations on all batch objects to improve the efficiency of batch operations.

[0233] In some embodiments, after completing batch operations, the device may also output a reminder message to notify the user that the batch operations have been completed. Optionally, the reminder message may include at least one of batch objects whose operations have been completed and batch objects whose operations have not been completed. For example, in combination with... Figure 9 As shown in (2), when the user confirms that the batch operation object 902 included in the reminder message 900 is correct, the device, in response to the operation, performs actions such as... Figure 8 After completing the operation on all batch processing objects as shown in (4), the device can output the following: Figure 9 The reminder message 920 shown in (3) indicates that all batch processing objects with completed operations, such as “Resume-Xiaolan.pdf”, “Resume-Xiaoming.pdf”, etc.

[0234] In some embodiments, the device may also create shortcuts based on exemplary operations. The device may receive a first user operation input during a first time period for processing a first object, and the device creates shortcuts based on the first user operation. Exemplarily, the first user operation may be an operation performed by the user when the shortcut is created, as described below, and may include, but is not limited to, operations on an input device equipped with the device, voice commands, or at least one of these.

[0235] Optionally, the device can identify the processing steps corresponding to the user's operation in the first time period, and then create the identified processing steps as shortcuts. These shortcuts trigger processing of the second object in a second time period, which follows the first time period. The device's processing result for the second object in response to the shortcut is consistent with the device's processing result for the first object in response to the first user's operation. Both the first and second objects can be objects within the device. Optionally, the first and second objects can exist as different objects in the same batch of data, or they can exist independently.

[0236] Alternatively, the processing result of other devices on the second object in response to the shortcut command is consistent with the processing result of the device on the first object in response to the first user operation. The first object is an object in the device, and the second object is an object in other devices besides the device. That is, the device can send the shortcut command to the other device.

[0237] Shortcuts can consist of a series of actions, each of which can be a simple task, such as sending a message, or a more complex script.

[0238] In this way, the device can directly create shortcuts using the steps determined based on the demonstration process, making shortcut creation more timely and efficient. Subsequently, users can directly trigger the device to perform a series of operations on their behalf using shortcuts, eliminating the need for users to perform multiple repetitive operations, thus reducing the complexity of repeatedly executing a series of operations and improving operational efficiency.

[0239] In this embodiment, as one possible implementation, the demonstration operation can be executed within the aforementioned batch operation scenario; that is, the shortcut command can be created within the aforementioned batch operation scenario. In this implementation, the device can not only execute batch operations based on the user's demonstration operation, but also create shortcut commands for the user based on the determined steps of the batch operation or the steps of the demonstration operation.

[0240] As another possible implementation, the demonstration operation could also be a demonstration operation specifically performed for creating shortcuts. That is, shortcuts do not necessarily have to be created in the scenario of batch operations described above.

[0241] For example, Figure 11 This illustration shows a flowchart of a method for generating shortcut commands according to an embodiment of this application. Figure 11 As shown, the process includes the following steps:

[0242] S1101, The device converts the steps of batch operations into instruction codes.

[0243] Understandably, in a scenario where the demonstration is specifically designed for creating shortcuts, Figure 11 The steps of the batch operation in the embodiments described herein can also be implemented as the steps of the demonstration operation.

[0244] The instruction code can be used to determine each step in a shortcut. Optionally, the instruction code can also be used to determine at least one of the shortcut's name, trigger method, etc. For example, the instruction code can be serialized in JSON or other lightweight data formats for easy storage and retrieval.

[0245] Taking the batch operation steps as an example, which include "checking A's birthday", "confirming that the birthday is in this month", and "sending 'Happy Birthday A'", the instruction code converted by the device into the batch operation steps can be shown in Table 3.

[0246] Table 3

[0247]

[0248]

[0249] S1102. The device displays a shortcut command editing interface. (Optional)

[0250] In some embodiments, the device may also display a shortcut editing interface. Optionally, this shortcut editing interface can be the runtime interface of a shortcut application. The shortcut application provides shortcut creation functionality. When the device recognizes the steps of a batch operation, it can launch the shortcut application to display the shortcut editing interface, thereby creating the recognized batch operation steps as shortcuts.

[0251] In some implementations, the shortcut editing interface can be used by users to edit the shortcut, including but not limited to setting the shortcut's name, setting the shortcut's trigger method (such as voice command trigger, specified gesture operation trigger, trigger via specified control, etc.), and changing the operation steps included in the shortcut (which can be used as an example of a fifth user operation). Different shortcuts can correspond to different trigger methods, which facilitates the execution of shortcuts.

[0252] In some implementations, the shortcut editing interface can also be used for users to confirm whether to create shortcuts for batch operations recognized by the device.

[0253] In this way, during the process of creating shortcuts based on the identified batch operation steps, the device can also present a shortcut editing interface, allowing users to make necessary supplementary inputs. This includes, but is not limited to, specifying, modifying, or setting the operation object in the shortcut as a variable to be input, modifying or setting the operation instruction in the shortcut as a variable to be input, supplementing the trigger condition of the operation instruction, and confirming the shortcut. This makes the generated shortcuts more accurate and better meet the actual needs of users.

[0254] For example, let's take a batch operation as an example, which includes the following three steps in sequence: "Check A's birthday", "Confirm that the birthday is in this month", and "Send 'Happy Birthday A'". Figure 12 A schematic diagram of a shortcut command editing interface provided in an embodiment of this application is shown.

[0255] like Figure 12 As shown, the editing interface 1200 displays a naming control 1201, a trigger method setting control 1202, a task editing control 1203, and a task adding control 1204. The naming control 1201 can be used to name shortcut commands, and the trigger method setting control 1203 can be used to set the trigger method for the shortcut command. Optionally, one or more trigger methods can be set. Figure 12 Two types are shown. The task editing control 1203 can be used to edit the steps in a shortcut. Optionally, during the execution of the shortcut, the main keywords (such as...) in the steps presented in the task editing control 1203... Figure 12 The "A" shown is variable, allowing the device to execute shortcuts on different objects by replacing the main keyword. The task addition control 1204 can be used to add more steps to a shortcut.

[0256] Optionally, steps S1101 and S1102 can be executed synchronously or sequentially. If they are executed sequentially, their order is not limited.

[0257] S1103, Device generates quick commands.

[0258] In some embodiments, the device can directly generate shortcut instructions based on the instruction code converted in step S1101.

[0259] In other embodiments, while the device is also performing step S1102, the device can also generate shortcuts based on the operations performed by the user in the shortcut editing interface and the instruction codes converted in step S1101.

[0260] In some embodiments, to facilitate the identification of different shortcuts, the device can assign a unique identifier to each shortcut. This ensures that different shortcuts can be used accurately subsequently. Optionally, this unique identifier can be user-defined, such as by the user using a method like... Figure 12 The name entered in the naming control 1201 shown. Of course, it can also be customized by the device.

[0261] Optionally, after a shortcut is created, the device can store the shortcut locally, or it can be stored in another location. This application embodiment does not impose any restrictions on this.

[0262] Understandable. Figure 12 The illustrated embodiment presents a shortcut editing interface when the device creates shortcuts from identified batch operation steps, allowing the user to create shortcuts. In other embodiments, users can also actively create shortcuts by activating the shortcut application. In response to the user activating the shortcut application, the device may also present, for example... Figure 12 The shortcut editing interface 1200 shown allows users to set one or more of the following in order to create a shortcut: the name of the shortcut, the trigger method, and the operation to be performed.

[0263] For example, Table 4 shows some examples of custom shortcuts provided in the embodiments of this application.

[0264] Table 4

[0265] Send message File Management Photo editing Social media updates Reading Mode Text processing Music playback Email processing Speed ​​Dial Weather forecast Travel Assistant Application fast launch Schedule reminders Shopping list Automation scripts navigation Currency conversion Data backup Health monitoring Language translation Equipment control Alarm clocks and timers

[0266] Optionally, these custom shortcuts, such as those shown in Table 4, can be created actively by the user through the shortcut application, or they can be created by the device based on the steps of a batch operation after it has recognized those steps. The different shortcuts shown in Table 4 can achieve different functions.

[0267] Furthermore, once a shortcut is created, users can activate it using the corresponding trigger method.

[0268] As a concrete example, when a user wants to save a generated screenshot to a specific location, they can create a "Save Screenshot to Specified Location" shortcut by performing the demonstration. Subsequently, any screenshots generated by the user can be saved to the specified location using this shortcut. The following example illustrates the process of creating a shortcut.

[0269] In some embodiments, a user can perform various operations to enable the device's shortcut creation function, and then begin performing a demonstration operation. Correspondingly, after enabling the shortcut creation function, the device can begin recording the user's demonstration operation. Specifically, the user can perform operations such as clicking the screenshot icon in the screenshot tool displayed on the device screen to enable the screenshot function; the device can recognize the operation step as "clicking the screenshot icon in the screenshot tool on the screen." Next, the user performs operations such as swiping to select a rectangular area to capture that area. The device can recognize the operation step as "defining a rectangular area (x1, y1)(x2, y2)." The user continues to perform operations such as clicking the copy button in the screenshot tool displayed on the screen to generate a screenshot and copy it. The device can recognize the operation step as "generating and copying the screenshot."

[0270] Next, the user navigates to path A and creates a new PowerPoint presentation within path A. The device recognizes this as "create a new PowerPoint presentation in path A." ​​Then, the user renames the PowerPoint presentation and inputs a voice command such as "rename PowerPoint." The device recognizes this as "rename PowerPoint." Following this, the user opens the renamed PowerPoint presentation and creates a new blank page (or slide) within it, inputting a voice command such as "create a new blank page here." The device recognizes this as "create a new blank page in the new PowerPoint presentation." Then, the user pastes a previously copied screenshot into the blank page and resizes the screenshot to match the blank page, inputting a voice command such as "resize them to the same size." The device recognizes this as "copy the screenshot to the blank page and resize the screenshot to match the blank page." This concludes the user's demonstration. Correspondingly, the device also recognizes all the steps of the demonstration operation performed by the user, such as: "click the screenshot icon in the screenshot tool on the screen", "draw a rectangular area (x1, y1)(x2, y2)", "generate and copy the screenshot", "create a new PPT in path A", "rename the PPT", "create a blank page in the new PPT", "copy the screenshot to the blank page and adjust the screenshot to the same size as the blank page", etc.

[0271] Understandably, for further information on the demonstration operations used to create shortcuts, please refer to the demonstration operations described above in the context of batch operations.

[0272] Subsequently, the device can create shortcuts based on the identified steps. Optionally, the device can present a shortcut editing interface, such as the one described above, based on these identified steps, for the user to perform necessary supplementary input. For example, the device can output the identified steps in the shortcut editing interface, and the user can specify the variables in the steps, supplement the conditions for executing the steps, etc. Other supplementary inputs can be found in the description of supplementary inputs above. For example, the user can specify the coordinates of the rectangular area "(x1, y1)(x2, y2)" in the step "define a rectangular area (x1, y1)(x2, y2)" as a variable. Optionally, in this embodiment, the default value of the variable can be the setting value when the user last executed the corresponding shortcut. For another example, the user can also specify "path A" in the step "create a new PPT in path A" as a variable. For example, users can set execution conditions for the step "Create a new PPT in path A," such as "If a PPT already exists in path A, then do not create a new PPT. If a PPT does not exist in path A, then create a new PPT," to help the device determine whether to execute the step "Create a new PPT in path A." ​​Finally, after the user completes the additional input, the user can perform operations such as saving as a shortcut to create a shortcut, thus completing the creation of the "Screenshot to specified storage location" shortcut.

[0273] Subsequently, the user can activate the "Screenshot Location Specify" shortcut by performing actions such as clicking the icon or other identifier of the shortcut. In response to this action, the device begins executing the "Screenshot Location Specify" shortcut. Specifically, referring to the example above where the user specifies the coordinates of the rectangular area "(x1, y1)(x2, y2)" and "path A" as variables, the device can first instruct the user to select the rectangular area when executing the shortcut. For example, the user can perform a swipe operation to redraw the rectangular area. Then, the device can execute the subsequent steps of "drawing the rectangular area (x1, y1)(x2, y2)" based on the redrawn rectangular area. Alternatively, the device can default to using the coordinates of the rectangular area previously drawn by the user as the coordinates of the rectangular area involved in the current shortcut execution. Next, the device executes the step "generate and copy screenshot" in the background based on the coordinates of the rectangular area to generate and copy the screenshot of the corresponding rectangular area. Then, the device instructs the user to enter "path A", or the device defaults to using the previous "path A". The device continues to execute the remaining steps based on the current "path A", such as "create a new PPT in path A", "rename the PPT", "create a blank page in the new PPT", and "copy a screenshot to the blank page and adjust the screenshot to the same size as the blank page". When the device completes all steps, this shortcut command ends.

[0274] Similarly, in other scenarios where users need to perform a series of operations multiple times, shortcuts can be created to improve operational efficiency and reduce the complexity of user operations. For example, if a user wants to export video files edited in the software in a specific MP4 format each time (which may include parameters such as encoding type and compression rate), the user can perform a demonstration operation. Based on this demonstration, the device can create a "Common Format Export" shortcut. Subsequently, when the user edits video files again in the software, they can use this "Common Format Export" shortcut to export the edited video files in the specific MP4 format without having to repeatedly set the various parameters required for that MP4 format.

[0275] For example, when a user registers for an appointment through a hospital's official WeChat account, they need to open their social media app every few minutes, access the hospital's WeChat account, and select the hospital district, department, doctor, and date to check for available appointments. The user can then perform a demonstration, and the device can create a "Check Appointment Availability" shortcut based on this demonstration. Subsequently, when the user wants to check if any appointments have been cancelled, they can use this "Check Appointment Availability" shortcut to access the designated interface on the device.

[0276] In some embodiments, the shortcuts and steps identified by the device based on a user-performed demonstration operation described in this application can be shared across devices. Optionally, the steps identified by the device based on a user-performed demonstration operation can be shared in the form of shortcuts or other forms. In this way, by performing a demonstration operation on one device and sharing the steps identified or shortcuts created based on that demonstration operation with other devices, the steps identified or shortcuts created based on that demonstration operation can be used directly on other devices. This can improve the operational efficiency of other devices and reduce the difficulty of operation. For example, when an elderly person needs to set a function on their mobile phone, their children can perform a demonstration operation on their own mobile phone, and the children's mobile phone can transmit the shortcuts created based on the demonstration operation to the elderly person's mobile phone. The elderly person can directly use the shortcuts to complete the corresponding function setting on their own mobile phone, greatly reducing the difficulty of operation for the elderly and improving operational efficiency.

[0277] For example, Figure 13 The following is a flowchart illustrating another object manipulation method provided in an embodiment of this application, such as... Figure 13 As shown, the method includes the following steps:

[0278] S1301, The device receives a first user operation for processing a first object in a batch of data.

[0279] The first user operation is used to process the first object. For example, the first user operation may include at least one of the following: operations on an input device equipped with the device (such as the physical input device described above) (such as various operations performed by the user on the physical input device to control the device, such as mouse clicks, mouse swipes, keyboard shortcuts, touch operations on a touchscreen / stylus / touchpad), and voice commands. For example, the first user operation may be as follows: Figure 2 Middle (3) to Figure 4 The various user operations included in the demonstration operation shown in (2) are as follows.

[0280] Optionally, the first object may include one or more objects, wherein the first object is a subset of the objects in the batch data. For example, the batch data may be as follows: Figure 2 The batch data 201 shown in (1) can be used as follows: Figure 2 In the example shown in (1), “Resume-Xiaohong.pdf”, the first object includes an object.

[0281] S1302, The device performs a batch operation on the second object in the batch data based on the first user operation.

[0282] The second object is any object other than the first object. The processing result of the device on the first object in response to the first user's operation is consistent with the processing result of the device on the second object using batch operations. Optionally, the second object may include one or more objects. For example, the second object may be as follows: Figure 2 The following are examples of the resumes: “Resume-Xiaolan.pdf”, “Resume-Xiaoming.pdf”, “Resume-Xiaoqiang.pdf”, and “Resume-Xiaotang.pdf” shown in (1).

[0283] In some embodiments, the device may output n processing steps corresponding to a batch operation (as described above for batch operations) based on a first user operation, where n is an integer greater than or equal to 1, and the device performs n processing steps on the second object. For example, these n processing steps may be as follows: Figure 8 Operation steps 800 to 830 are shown in (4). Optionally, the device may display or verbally play n processing steps.

[0284] In some implementations, the device can output n processing steps based on at least one of the following: operation of the input device equipped with the device, voice commands, or content displayed on the interface when the device receives a first user operation (such as the interactive interface information described above), i.e., the exemplary operation information described in step S502. Optionally, when a second user operation exists, the device can also output n processing steps based on the first user operation and the second user operation. The second user operation is a user operation used to indicate to the device whether the steps in the identified batch operation are accurate.

[0285] In some embodiments, the device performs n processing steps on the second object, which may include: the device determining the subject keyword to be replaced included in each of the n processing steps. Optionally, some processing steps may include the subject keyword to be replaced, and some processing steps may not include the subject keyword to be replaced. A description of this step can be found in the description of the device determining the subject keyword to be replaced in each step of the batch operation described above. The device determines the subject keyword of the second object, where the subject keyword of the second object has the same attributes as the subject keyword to be replaced. A description of this step can be found in the description of the device extracting the subject keyword of the batch processing object described above. The device replaces the subject keyword to be replaced included in each processing step with the subject keyword of the second object, and the device sequentially executes each processing step after the subject keyword replacement. A description of this step can be found in... Figure 10 The following is a description of step S1002.

[0286] In some embodiments, each of the n processing steps may include an operation instruction and an operation object corresponding to the operation instruction. Before the device outputs the n processing steps corresponding to the batch operation based on the first user operation, the device may identify the operation instruction and the operation object corresponding to the operation instruction in each processing step based on the first user operation. This implementation can be referred to Figure 5d The identification operation instructions shown Figure 6 The diagram shows the identification of the operation object and the relevant implementation of the operation steps included in the demonstration operation based on the identified operation instructions and operation object.

[0287] The above primarily describes the solutions provided by the embodiments of this application from a methodological perspective. It is understood that, in order to achieve the above functions, the device includes corresponding hardware structures and / or software modules for executing each function. By combining the units and algorithm steps of the various examples described in the embodiments disclosed in this application, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by a computer driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solutions of the embodiments of this application.

[0288] This application provides embodiments that can divide the device into functional modules based on the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional module. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0289] like Figure 14 The diagram shown is a structural schematic of another device provided in this application embodiment. This device 1400 can be used to implement the methods executed by the devices described in the above method embodiments. For example, the device 1400 may include a processing unit 1401 and a display unit 1402.

[0290] The processing unit 1401 is used to support the execution of the device 1400. Figures 1 to 13 The processing functions of the device described in any one of the above, and the display unit 1402, are used to support the device 1400 in performing its functions. Figures 1 to 13 The display function of the device described in any one of the following statements.

[0291] Optional, Figure 14The illustrated device 1400 may further include a storage unit 1403 that stores programs or instructions. When the processing unit 1401 executes the program or instructions, it causes... Figure 14 The device 1400 shown can perform the method described in the above-described method embodiments.

[0292] Figure 14 The technical effects of the device 1400 shown can be referred to the technical effects described in the above method embodiments, and will not be repeated here. Figure 14 The processing unit 1401 involved in the illustrated device 1400 can be implemented by a processor or processor-related circuit components, and can be a processor or processing module. The display unit 1402 can be implemented by display screen-related components.

[0293] This application also provides a chip system, such as... Figure 15 As shown, the chip system includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 and the interface circuit 1502 are interconnected via lines. For example, the interface circuit 1502 can be used to receive signals from other devices. As another example, the interface circuit 1502 can be used to send signals to other devices (e.g., the processor 1501). Exemplarily, the interface circuit 1502 can read instructions stored in memory and send those instructions to the processor 1501. When the instructions are executed by the processor 1501, the device can perform the various steps executed by the device in the above embodiments. Of course, the chip system may also include other discrete components, which are not specifically limited in this application embodiment.

[0294] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0295] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application does not specifically limit the type of memory or the arrangement of the memory and processor.

[0296] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0297] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0298] This application also provides a computer storage medium storing computer instructions, which, when executed on a device, cause the device to perform the methods described in the above-described method embodiments.

[0299] This application provides a computer program product, which includes a computer program or instructions that, when run on a computer, cause the computer to perform the methods described in the above-described method embodiments.

[0300] In addition, this application also provides an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the apparatus to perform the methods in the above-described method embodiments.

[0301] In this embodiment, the device, computer storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0302] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0303] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The embodiments can be combined with or referenced to each other without conflict. The apparatus embodiments described above are merely illustrative; for example, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0304] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0305] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0306] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0307] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An object operation method characterized by comprising: The method includes: The device receives a first user operation for processing a first object in a batch of objects; The device performs a batch operation on a second object in the batch objects based on the first user operation, wherein the second object is an object other than the first object, and the processing result of the device on the first object in response to the first user operation is consistent with the processing result of the device on the second object using the batch operation.

2. The method of claim 1, wherein, The device performs a batch operation on the second object in the batch objects based on the first user operation, including: The device outputs n processing steps corresponding to the batch operation based on the first user operation, where n is an integer greater than or equal to 1; The device performs the n processing steps on the second object.

3. The method of claim 2, wherein, The first user operation includes at least one of the following: operation of the input device equipped on the device, and voice commands; the device outputs n processing steps corresponding to the batch operation based on the first user operation, including: The device outputs the n processing steps based on at least one of the following: operation of the input device equipped on the device, the voice command, and the content of the interface displayed when the device receives the first user operation.

4. The method according to any one of claims 1-3, characterized in that, Before the device performs a batch operation on the second object in the batch objects based on the first user operation, the method further includes: Upon receiving the first operation from the first user operation, the device outputs a first reminder message. The first reminder message is used to indicate that the device has identified a first processing step, or the first reminder message is used to confirm with the user whether the identified first processing step is accurate. The first processing step is a processing step identified based on the first operation.

5. The method of claim 4, wherein, After the device outputs the first reminder message, the method further includes: The device receives a second user operation, which is used to indicate to the device whether the first processing step is accurate.

6. The method of claim 5, wherein, The first notification message includes the first processing step, and the second user operation includes editing the first processing step.

7. The method according to claim 5 or 6, characterized in that, The second user operation includes at least one of voice commands and operation of the input device equipped with the device.

8. The method according to any one of claims 5-7, characterized in that, The device outputs n processing steps corresponding to the batch operation based on the first user operation, including: The device outputs the n processing steps based on the first user operation and the second user operation.

9. The method according to any one of claims 1-8, characterized in that, Before the device performs a batch operation on the second object in the batch objects based on the first user operation, the method further includes: Receive a third user operation, the third user operation being used to instruct the device on the batch objects; and / or; Output a second reminder message, which is used to confirm the second object in the batch of objects with the user; A fourth user operation is received, the fourth user operation being used to indicate the second object to the device.

10. The method according to any one of claims 1-9, characterized in that, Before the device receives a first user operation for processing a first object in a batch of objects, the method further includes: the device receiving information for instructing the first user operation to begin execution; And / or, Before the device performs a batch operation on a second object in the batch of objects based on the first user operation, the method further includes: the device receiving information indicating that the first user operation has ended execution; The information indicating the start of the first user operation and the information indicating the end of the first user operation are used by the device to identify the first user operation.

11. The method according to any one of claims 2-10, characterized in that, After the device outputs n processing steps corresponding to the batch operation based on the first user operation, the method further includes: The device creates the n processing steps as shortcut instructions.

12. The method according to claim 11, characterized in that, Before the device creates the n processing steps as shortcuts, the method further includes: The device receives a fifth user operation, which is used to set the triggering method of the shortcut command, set the name of the shortcut command, and edit at least one of the n processing steps.

13. The method according to any one of claims 2-10, characterized in that, After the device receives a first user operation for processing a first object in a batch of objects, the method further includes: The device sends the n processing steps to the target device. The n processing steps are used by the target device to perform processing on objects in the target device. The processing result of the target device on the objects in the target device using the n processing steps is consistent with the processing result of the device on the first object in response to the first user operation. or, The device sends a shortcut instruction created based on the n processing steps to the target device, the shortcut instruction being used to trigger the target device to execute the n processing steps on an object in the target device.

14. The method according to any one of claims 1-13, characterized in that, The batch operation is performed on a single application, or the batch operation is performed on multiple applications.

15. An object manipulation method, characterized in that, The method includes: The device receives a first user operation input in the first time period for processing a first object; The device creates a shortcut based on the first user operation. The shortcut is used to trigger processing of a second object in a second time period, which is after the first time period. The processing result of the shortcut on the second object is consistent with the processing result of the device on the first object in response to the first user operation.

16. A device, characterized in that, include: A processor and a memory, the memory being coupled to the processor, the memory being used to store program code including instructions, the processor reading the instructions from the memory to cause the device to perform the method as claimed in any one of claims 1-14, or to cause the device to perform the method as claimed in claim 15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when executed on the device, causes the device to perform the method as claimed in any one of claims 1-14, or causes the device to perform the method as claimed in claim 15.

18. A computer program product, characterized in that, The computer program product includes: a computer program or instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1-14, or cause the computer to perform the method as claimed in claim 15.

Images