Data processing method and device and electronic equipment

Abstract

The invention provides a data processing method and device and electronic equipment. The method is applied to the electronic equipment and comprises the steps that a first arrangement result is received, and the first arrangement result corresponds to a first user instruction; determining at least one piece of equipment information from the multiple pieces of equipment information according to the first arrangement result; sending at least one piece of equipment information; receiving a second arrangement result, wherein the second arrangement result is determined by at least one piece of equipment information and the first arrangement result; and executing the first workflow corresponding to the second arrangement result. According to the data processing method and device and the electronic equipment, the electronic equipment can provide the at least one piece of equipment information related to the arrangement result for the server, so that the server optimizes the arrangement result according to the at least one piece of equipment information, the success rate of executing the user instruction by the electronic equipment is improved, and the user experience is improved.

Description

Technical Field

[0001] This application relates to electronic equipment technology, and more particularly to a data processing method, apparatus, and electronic equipment. Background Technology

[0002] The intelligentization of electronic devices has become a current development trend. For example, mobile phones, tablets, and personal computers can support voice assistant functions. On-device voice assistants can acquire user commands, understand and orchestrate the user commands based on a cloud-based big data model, and then execute the orchestrated results on the device. It's clear that currently, the execution of orchestrated results on the device relies on the cloud-based big data model's correct understanding and orchestration of user commands.

[0003] Therefore, optimizing the understanding and arrangement of user commands to improve the user experience is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a data processing method, apparatus, and electronic device. The method can optimize the understanding and arrangement of user instructions to improve the user experience.

[0005] In a first aspect, a data processing method is provided, applied to an electronic device, the method comprising: receiving a first orchestration result from a server, the first orchestration result corresponding to a first user instruction; determining at least one piece of device information from a plurality of device information based on the first orchestration result; sending the at least one piece of device information to the server; receiving a second orchestration result from the server, the second orchestration result being determined by the at least one piece of device information and the first orchestration result; and executing a first workflow corresponding to the second orchestration result.

[0006] For example, an electronic device may be pre-configured with multiple device information. The electronic device may select at least one device from the multiple device information based on the first arrangement result. The at least one device information may be related information associated with the first arrangement result, and the at least one device information may be used to optimize the original arrangement result.

[0007] Based on the above scheme, electronic devices can provide the server with at least one device information as association information, so that the server can optimize the orchestration results based on at least one device information, thereby improving the success rate of electronic devices executing user commands and enhancing user experience.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the plurality of device information includes at least one of the following: the status information of the electronic device, the information of the application, or the relationship between the status information of the electronic device and the information of the application, wherein the information of the application includes the type of the application, and the status information of the electronic device includes language information and / or battery information.

[0009] For example, multiple device information can be a device information relationship graph.

[0010] Based on the above scheme, at least one piece of device information related to the first orchestration result is selected by filtering multiple device information sets to obtain an optimized orchestration result. On the one hand, the electronic devices execute the workflow corresponding to the optimized orchestration result, which can ensure the success rate of user command execution. On the other hand, the optimized orchestration result involves a device information relationship graph, making the execution logic of the electronic devices more aligned with the device information and improving the execution effect of user commands.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the plurality of device information includes information about the application, wherein determining at least one device information from the plurality of device information according to the first orchestration result includes: when the first orchestration result is related to the first application, determining information about a first type of application related to the first application from the plurality of device information, wherein the information about the first type of application belongs to the application.

[0012] Based on the above scheme, when the first orchestration result is related to a certain application, the electronic device filters out other applications of the same type as the application as associated information, so as to facilitate the server to optimize the orchestration result.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the information of the first type of application is used to indicate that the number of applications belonging to the first type is N, where N is a positive integer greater than or equal to 2.

[0014] Based on the above scheme, the information of the first type of application can be the number of applications of the first type, or it can be the specific applications of the first type (e.g., the name or identifier of the application). The server can determine that there are multiple applications of the first type in the electronic device based on the information of the first type of application. The optimized orchestration result obtained through this information allows the electronic device to select a more suitable application from the same type of application to execute user instructions.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the plurality of device information includes the power information, wherein determining at least one device information from the plurality of device information according to the first arrangement result includes: determining the power information from the plurality of device information when the first arrangement result is related to the power information.

[0016] Based on the above scheme, the power information can represent the current power level of the electronic device. When the electronic device executes user commands, it will affect the power level of the electronic device.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, when the power information indicates that the power of the electronic device is below a power threshold, the first workflow includes displaying a first prompt message, which prompts the user to charge the electronic device.

[0018] Based on the above solution, when the battery is low, the workflow corresponding to the optimized orchestration result can remind the user to charge the electronic device, thus avoiding interruption of user command execution due to insufficient battery.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the plurality of device information includes the language information used to represent the system language, wherein determining at least one device information from the plurality of device information according to the first arrangement result includes: determining the language information from the plurality of device information when the first arrangement result is related to the language information.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, when the working language corresponding to the first workflow is inconsistent with the system language, the first workflow includes displaying a second prompt message, which prompts the user to switch the working language to the system language.

[0021] Based on the above scheme, when the first workflow involves an application, the working language refers to the language within that application. When the working language differs from the system language, the electronic device can prompt the user to switch the working language to the system language to improve the user experience.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, sending the at least one device information to the server includes: sending the at least one device information, the first orchestration result, and the first user instruction to the server.

[0023] Based on the above scheme, in order to facilitate the server's optimization of orchestration results, electronic devices can send the information needed to optimize the orchestration results to the server so that the server can efficiently optimize the orchestration results.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the electronic device includes a preset user instruction set and a preset workflow set, wherein the preset user instructions in the preset user instruction set correspond one-to-one with the preset workflows in the preset workflow set, wherein, before receiving the first orchestration result from the server, the method further includes: acquiring the first user instruction; and sending the first user instruction to the server when the first user instruction fails to match the preset user instructions in the preset user instruction set.

[0025] It should be noted that the set of preset user instructions and the set of preset workflows are in one-to-one correspondence. The set of preset user instructions can include multiple high-frequency, essential preset user instructions. In this context, user instructions can be used to reflect user intent, and preset user instructions can be understood as preset user intents.

[0026] Therefore, if there is no preset user instruction in the preset user instruction set of the electronic device that can be successfully matched with the user instruction, the user instruction can be sent to the server, and the server will generate the arrangement result.

[0027] Based on the above solution, to reduce the interaction between electronic devices and servers, a set of frequently used and essential pre-defined user commands and corresponding pre-defined workflows can be configured in the electronic devices. When a user command fails to match a pre-defined user command in the pre-defined user command set, the cloud-side large model needs to orchestrate and understand the user command.

[0028] In conjunction with the first aspect, in some implementations of the first aspect, before sending the first user instruction to the server, the method further includes: determining at least one sub-instruction based on the first user instruction; determining at least one first vector based on the at least one sub-instruction; determining a preset user instruction vector set based on the preset user instruction set, the preset user instruction vector set including at least one second vector, the at least one first vector corresponding one-to-one with the at least one second vector; and determining whether the first user instruction matches a preset user instruction in the preset user instruction set based on the similarity between the at least one first vector and the at least one second vector.

[0029] Based on the above scheme, using sentence segmentation technology, N sub-instructions (N is a positive integer greater than or equal to 1) are obtained, and these N sub-instructions are converted into N first vectors. The preset user instruction set is also converted into a preset user instruction vector set. These N first vectors are then matched with the N second vectors in the preset user instruction vector set for similarity. A match is considered successful when the similarity is greater than a similarity threshold.

[0030] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: recording the number of times a first type of user instruction fails to match with a preset user instruction in the preset user instruction set, wherein the first user instruction belongs to the first type of user instruction; when the number of matching failures meets a preset condition, adding a second preset user instruction to the preset user instruction set according to the first type of user instruction, and adding a second preset workflow to the preset workflow set, wherein the second preset workflow corresponds to the second preset user instruction.

[0031] Based on the above scheme, electronic devices can preset the number of times a similar user command will fail to match within a certain period (e.g., 10 days) to 20 times. When the number of failures exceeds a threshold (15 times), it indicates that the preset condition is met, and such user commands can be updated to the preset user command set, along with the corresponding workflow set. By continuously updating the preset user command set and the preset workflow set, the accuracy of executing user commands can be improved, while reducing interaction with the cloud-based large model, lowering communication latency, and enhancing the user experience.

[0032] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: obtaining a second user instruction; when the second user instruction successfully matches a first preset user instruction, determining a first preset workflow based on the first preset user instruction, wherein the first preset workflow belongs to a set of preset workflows and the first preset user instruction belongs to a set of preset user instructions; and executing the first preset workflow.

[0033] Based on the above scheme, when the second user instruction successfully matches the first preset user instruction in the preset user instruction set, the electronic device can execute the first preset workflow corresponding to the first preset user instruction.

[0034] In conjunction with the first aspect, in some implementations of the first aspect, executing the first preset workflow includes: determining parameter information according to the second user instruction; determining a second workflow according to the first preset workflow and the parameter information; and executing the second workflow.

[0035] Based on the above scheme, in order to further improve the accuracy of electronic devices in executing user instructions, the workflow can be optimized by combining at least one of the techniques of slot filling and substitution resolution.

[0036] In conjunction with the first aspect, in some implementations of the first aspect, the execution of the first workflow corresponding to the second orchestration result includes: determining the first workflow based on the second orchestration result and user information, wherein the user information includes at least one of the following: user habits, user interests, and user relationships; and executing the first workflow.

[0037] Based on the above solution, electronic devices can optimize the workflow corresponding to the orchestration results issued by the server according to user information, making the execution of electronic devices more in line with the user's personalized needs and improving the user experience.

[0038] In a second aspect, a data processing method is provided, applied to an electronic device, the method comprising: receiving a first orchestration result from a server, the first orchestration result corresponding to a first user instruction; determining a first workflow based on the first orchestration result and user information; and executing the first workflow.

[0039] Based on the above scheme, electronic devices can optimize the workflow corresponding to the orchestration results from the server based on user information, so that the execution of electronic devices can provide users with a personalized experience.

[0040] In conjunction with the second aspect, in some implementations of the second aspect, the user information includes at least one of the following: user habits, user interests, and user relationships.

[0041] For example, user information can be a personal knowledge graph (PKG), which can be used to describe personal entities, attributes, and their relationships. By leveraging user habits, interests, and characteristics represented by this information, the workflow corresponding to the orchestration results can be optimized, making the execution of user commands by electronic devices more aligned with the user's personality.

[0042] Thirdly, a data processing method is provided, which is applied to an electronic device. The electronic device includes a preset user instruction set and a preset workflow set. The preset user instructions in the preset user instruction set correspond one-to-one with the preset workflows in the preset workflow set. The method further includes: obtaining a first user instruction; determining whether the first user instruction matches the preset user instructions in the preset user instruction set; when the first user instruction matches the first preset user instruction, determining a first preset workflow based on the first preset user instruction, wherein the first preset user instruction belongs to the preset user instruction set and the first preset workflow belongs to the preset workflow set; and executing the first preset workflow.

[0043] It should be noted that the set of preset user instructions and the set of preset workflows are in one-to-one correspondence. The set of preset user instructions can include multiple high-frequency, essential preset user instructions. User instructions can be used to reflect user intent, and preset user instructions can be understood as preset user intents. Therefore, when a first user instruction successfully matches a first preset user instruction in the set of preset user instructions, the electronic device can execute the first preset workflow corresponding to the first preset user instruction.

[0044] Based on the above scheme, electronic devices are equipped with a set of frequently used and essential pre-set user commands and a set of pre-set workflows. When an electronic device receives a user command, it can first determine whether there is a matching pre-set user command. If the user command matches a pre-set user command, it can execute the pre-set workflow corresponding to the pre-set user command set, thereby reducing the interaction between the electronic device and the server, reducing communication latency, and improving the user experience.

[0045] In conjunction with the third aspect, in some implementations of the third aspect, the method further includes: when the first user instruction fails to match with a preset user instruction in the preset user instruction set, sending the first user instruction to the server; recording the number of times the first type of user instruction fails to match with a preset user instruction in the preset user instruction set, wherein the first user instruction belongs to the first type of user instruction; when the number of matching failures meets a preset condition, adding a second preset user instruction to the preset user instruction set according to the first type of user instruction, and adding a second preset workflow to the preset workflow set, wherein the second preset workflow corresponds to the second preset user instruction.

[0046] As can be seen, electronic devices can preset the number of times a similar user command will fail to match within a certain period (e.g., 10 days) to 20 times. When the number exceeds the threshold (15 times), such user commands can be updated to the preset user command set, and the corresponding workflow can also be updated to the preset workflow set. By continuously updating the preset user command set and the preset workflow set, the accuracy of executing user commands can be improved, while reducing interaction with the cloud-side large model, reducing communication latency, and improving user experience.

[0047] In conjunction with the third aspect, in some implementations of the third aspect, determining whether the first user instruction matches the preset user instructions in the preset user instruction set includes: determining at least one sub-instruction based on the first user instruction; determining at least one third vector based on the at least one sub-instruction; determining a preset user instruction vector set based on the preset user instruction set, the preset user instruction vectors including at least one fourth vector, the at least one third vector corresponding one-to-one with the at least one fourth vector; and determining whether the first user instruction matches the preset user instructions in the preset user instructions based on the similarity between the at least one third vector and the at least one fourth vector.

[0048] It should be noted that user commands and preset user commands in the preset user command set can be converted into vectors to determine whether a user command can be successfully matched with a preset user command in the preset user command set.

[0049] In conjunction with the third aspect, in some implementations of the third aspect, the execution of the first preset workflow includes: determining parameter information according to the first user instruction; determining a second workflow according to the first preset workflow and the parameter information; and executing the second workflow.

[0050] Based on the above scheme, in order to further improve the accuracy of electronic devices in executing user instructions, the workflow can be optimized by combining at least one of the techniques of slot filling and substitution resolution.

[0051] Fourthly, a data processing method is provided, applied to a server, the method comprising: determining a first arrangement result according to a first user instruction; sending the first arrangement result to an electronic device; receiving at least one device information from the electronic device, the at least one device information being determined by the first arrangement result and multiple device information; determining a second arrangement result according to the first arrangement result and the at least one device information; and sending the second arrangement result to the electronic device.

[0052] Based on the above scheme, the server can optimize the original orchestration result according to at least one device information, so that the electronic device can execute the workflow corresponding to the optimized orchestration result, thereby improving the success rate of the electronic device in executing user instructions and enhancing the user experience.

[0053] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the plurality of device information includes at least one of the following: the status information of the electronic device, the information of the application, or the relationship between the status information of the electronic device and the information of the application, wherein the information of the application includes the type of the application, and the status information of the electronic device includes language information and / or battery information.

[0054] For example, multiple device information can be a device information relationship graph.

[0055] In conjunction with the fourth aspect, in some implementations of the fourth aspect, receiving at least one device information from the electronic device includes: receiving the at least one device information from the electronic device, the first arrangement result, and the first user instruction.

[0056] Based on the above scheme, in order to facilitate the server's optimization of orchestration results, the information required for optimizing the orchestration results can be received so that the server can efficiently optimize the orchestration results.

[0057] Fifthly, a data processing apparatus is provided, the method being applied to the apparatus, the apparatus including a transceiver unit for receiving a first orchestration result from a server, the first orchestration result corresponding to a first user instruction; further including a processing unit for determining at least one piece of device information from a plurality of device information based on the first orchestration result; the transceiver unit is further configured to send the at least one piece of device information to the server; further configured to receive a second orchestration result from the server, the second orchestration result being determined by the at least one piece of device information and the first orchestration result; the processing unit is further configured to execute a first workflow corresponding to the second orchestration result.

[0058] In conjunction with the fifth aspect, in some implementations of the fifth aspect, the plurality of device information includes at least one of the following: device status information, application information, or the relationship between device status information and application information, wherein the application information includes the application type, and the device status information includes language information and / or battery level information.

[0059] In conjunction with the fifth aspect, in some implementations of the fifth aspect, the plurality of device information includes information about the application, and the processing unit is specifically configured to, when the first orchestration result is related to the first application, determine from the plurality of device information information information about a first type of application related to the first application, wherein the information about the first type of application belongs to the information of the application.

[0060] In conjunction with the fifth aspect, in some implementations of the fifth aspect, the information of the first type of application is used to indicate that the number of applications belonging to the first type is N, where N is a positive integer greater than or equal to 2.

[0061] In conjunction with the fifth aspect, in some implementations of the fifth aspect, the plurality of device information includes the power information, and the processing unit is specifically used to determine the power information from the plurality of device information when the first arrangement result is related to the power information.

[0062] In conjunction with the fifth aspect, in some implementations of the fifth aspect, when the power information indicates that the device's power is below a power threshold, the first workflow includes displaying a first prompt message to prompt the user to charge the device.

[0063] In conjunction with the fifth aspect, in some implementations of the fifth aspect, the plurality of device information includes the language information, which is used to represent the system language, and the processing unit is specifically used to determine the language information from the plurality of device information when the first arrangement result is related to the language information.

[0064] In conjunction with the fifth aspect, in some implementations of the fifth aspect, when the working language corresponding to the first workflow is inconsistent with the system language, the first workflow includes displaying a second prompt message, which prompts the user to switch the working language to the system language.

[0065] In conjunction with the fifth aspect, in some implementations of the fifth aspect, the transceiver unit is specifically used to send the at least one device information, the first arrangement result, and the first user instruction to the server.

[0066] In conjunction with the fifth aspect, in some implementations of the fifth aspect, the device includes a preset user instruction set and a preset workflow set, wherein the preset user instructions in the preset user instruction set correspond one-to-one with the preset workflows in the preset workflow set, and the transceiver unit is further configured to acquire the first user instruction; and further configured to send the first user instruction to the server when the first user instruction fails to match the preset user instructions in the preset user instruction set.

[0067] In conjunction with the fifth aspect, in some implementations of the fifth aspect, the processing unit is further configured to: determine at least one sub-instruction based on the first user instruction; determine at least one first vector based on the at least one sub-instruction; determine a preset user instruction vector set based on the preset user instruction set, the preset user instruction vector set including at least one second vector, the at least one first vector corresponding one-to-one with the at least one second vector; and determine whether the first user instruction matches a preset user instruction in the preset user instruction set based on the similarity between the at least one first vector and the at least one second vector.

[0068] In conjunction with the fifth aspect, in some implementations of the fifth aspect, the processing unit is further configured to record the number of times a first type of user instruction fails to match with a preset user instruction in the preset user instruction set, wherein the first user instruction belongs to the first type of user instruction; and is further configured to, when the number of matching failures meets a preset condition, add a second preset user instruction to the preset user instruction set according to the first type of user instruction, and add a second preset workflow to the preset workflow set, wherein the second preset workflow corresponds to the second preset user instruction.

[0069] In conjunction with the fifth aspect, in some implementations of the fifth aspect, the transceiver unit is further configured to acquire a second user instruction; the processing unit is further configured to, when the second user instruction successfully matches the first preset user instruction, determine a first preset workflow based on the first preset user instruction, wherein the first preset workflow belongs to the preset workflow set and the first preset user instruction belongs to the preset user instruction set; and execute the first preset workflow.

[0070] In conjunction with the fifth aspect, in some implementations of the fifth aspect, the processing unit is specifically used to determine parameter information according to the second user instruction; to determine a second workflow according to the first preset workflow and the parameter information; and to execute the second workflow.

[0071] In conjunction with the fifth aspect, in some implementations of the fifth aspect, the processing unit is specifically used to determine the first workflow based on the second arrangement result and user information, wherein the user information includes at least one of the following: user habits, user interests, and user relationships; and execute the first workflow.

[0072] In a sixth aspect, a data processing apparatus is provided, the apparatus including a transceiver unit for receiving a first arrangement result from a server, the first arrangement result corresponding to a first user instruction; and a processing unit for determining a first workflow based on the first arrangement result and user information; and for executing the first workflow.

[0073] In conjunction with the sixth aspect, in some implementations of the sixth aspect, the user information includes at least one of the following: user habits, user interests, and user relationships.

[0074] In a seventh aspect, a data processing apparatus is provided. The method is applied to the apparatus, which includes a preset user instruction set and a preset workflow set. The preset user instructions in the preset user instruction set correspond one-to-one with the preset workflows in the preset workflow set. The apparatus includes a transceiver unit for acquiring a first user instruction; and a processing unit for determining whether the first user instruction matches a preset user instruction in the preset user instruction set; further for determining a first preset workflow based on the first preset user instruction when the first user instruction matches the first preset user instruction, wherein the first preset user instruction belongs to the preset user instruction set and the first preset workflow belongs to the preset workflow set; and further for executing the first preset workflow.

[0075] In conjunction with the seventh aspect, in some implementations of the seventh aspect, the transceiver unit is further configured to send the first user instruction to the server when the first user instruction fails to match with the preset user instructions in the preset user instruction set; the processing unit is further configured to record the number of times the first type of user instruction fails to match with the preset user instructions in the preset user instruction set, wherein the first user instruction belongs to the first type of user instruction; and is further configured to, when the number of matching failures meets a preset condition, add a second preset user instruction to the preset user instruction set according to the first type of user instruction, and add a second preset workflow to the preset workflow set, wherein the second preset workflow corresponds to the second preset user instruction.

[0076] In conjunction with the seventh aspect, in some implementations of the seventh aspect, the processing unit is specifically configured to: determine at least one sub-instruction based on the first user instruction; determine at least one third vector based on the at least one sub-instruction; determine a preset user instruction vector set based on the preset user instruction set, the preset user instruction vector set including at least one fourth vector, the at least one third vector corresponding one-to-one with the at least one fourth vector; and determine whether the first user instruction matches the preset user instruction in the preset user instruction based on the similarity between the at least one third vector and the at least one fourth vector.

[0077] In conjunction with the seventh aspect, in some implementations of the seventh aspect, the processing unit is specifically used to determine parameter information according to the first user instruction; to determine a second workflow according to the first preset workflow and the parameter information; and to execute the second workflow.

[0078] Eighthly, a data processing apparatus is provided, the apparatus including a processing unit for determining a first arrangement result according to a first user instruction; a transceiver unit for sending the first arrangement result to an electronic device; and for receiving at least one device information from the electronic device, the at least one device information being determined by the first arrangement result and a plurality of device information; the processing unit is further configured to determine a second arrangement result based on the first arrangement result and the at least one device information; and the transceiver unit is further configured to send the second arrangement result to the electronic device.

[0079] In conjunction with the eighth aspect, in some implementations of the eighth aspect, the plurality of device information includes at least one of the following: status information of the electronic device, application information, or the relationship between the status information of the electronic device and the application information, wherein the application information includes the type of the application, and the status information of the electronic device includes language information and / or battery information.

[0080] In conjunction with the eighth aspect, in some implementations of the eighth aspect, the transceiver unit is specifically configured to receive the at least one device information, the first arrangement result, and the first user instruction from the electronic device.

[0081] A ninth aspect provides an electronic device comprising: one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the methods described in the first aspect, the second aspect, the third aspect, and any possible implementation thereof.

[0082] In a tenth aspect, a server is provided, the server including units / modules for performing methods as described in the fourth aspect above and any possible implementation thereof.

[0083] Eleventhly, a data processing apparatus is provided, comprising: a processor coupled to a memory for storing a computer program, the processor for running the computer program, such that the data processing apparatus performs the method of the first aspect and any possible implementation thereof, or performs the method of the second aspect and any possible implementation thereof, or performs the method of the third aspect and any possible implementation thereof.

[0084] In a twelfth aspect, a data processing apparatus is provided, comprising: a processor coupled to a memory for storing a computer program, the processor for running the computer program such that the data processing apparatus performs the methods described in the fourth aspect and any possible implementation thereof.

[0085] In a thirteenth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a computer, causes the computer to implement the method as described in the first aspect and any possible implementation thereof, or the method as described in the second aspect and any possible implementation thereof, or the method as described in the third aspect and any possible implementation thereof, or the method as described in the fourth aspect and any possible implementation thereof.

[0086] In a fourteenth aspect, a computer program product comprising instructions is provided, which, when run on a computer, causes the computer to perform the method as described in the first aspect and any possible implementation thereof, or to perform the method as described in the second aspect and any possible implementation thereof, or to perform the method as described in the third aspect and any possible implementation thereof, or to perform the method as described in the fourth aspect and any possible implementation thereof.

[0087] In a fifteenth aspect, a chip is provided, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface to execute the method as described in the first aspect and any possible implementation thereof, or to execute the method as described in the second aspect and any possible implementation thereof, or to execute the method as described in the third aspect and any possible implementation thereof, or to execute the method as described in the fourth aspect and any possible implementation thereof.

[0088] In conjunction with aspect fifteen, in one possible implementation, the processor is coupled to the memory via an interface.

[0089] In conjunction with aspect fifteen, in one possible implementation, the chip system further includes a memory in which computer programs or computer instructions are stored. Attached Figure Description

[0090] Figure 1 This is a schematic diagram of the structure of an electronic device.

[0091] Figure 2 This is a software structure block diagram of the electronic device provided in the embodiments of this application.

[0092] Figure 3 This is a schematic diagram of a system architecture provided in an embodiment of this application.

[0093] Figure 4 This is a schematic flowchart illustrating a data processing method provided in an embodiment of this application.

[0094] Figure 5 This is a schematic diagram of a device information relationship map provided in an embodiment of this application.

[0095] Figure 6 This is a set of data processing comparison diagrams provided in the embodiments of this application.

[0096] Figure 7 This is a set of data processing comparison diagrams provided in the embodiments of this application.

[0097] Figure 8 This is a schematic flowchart illustrating a data processing method provided in an embodiment of this application.

[0098] Figure 9 This is a schematic diagram of a Personal Knowledge Graph (PKG) provided in an embodiment of this application.

[0099] Figure 10 This is a set of data processing comparison diagrams provided in the embodiments of this application.

[0100] Figure 11 This is a set of data processing comparison diagrams provided in the embodiments of this application.

[0101] Figure 12 This is a schematic flowchart illustrating a data processing method provided in an embodiment of this application.

[0102] Figure 13 This is a flowchart illustrating a sentence splitting process provided in an embodiment of this application.

[0103] Figure 14 This is a schematic diagram of a preset workflow optimized using slot filling technology, provided in an embodiment of this application.

[0104] Figure 15This is a schematic diagram of a data sorting process provided in an embodiment of this application.

[0105] Figure 16 This is a schematic flowchart illustrating a data processing method provided in an embodiment of this application.

[0106] Figure 17 This is a schematic flowchart illustrating a data processing method provided in an embodiment of this application.

[0107] Figure 18 This is a schematic flowchart illustrating a data processing method provided in an embodiment of this application.

[0108] Figure 19 This is an example diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0109] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0110] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one, two, or more than two. The term “and / or” is used to describe the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0111] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0112] The following describes an electronic device, a user interface for such an electronic device, and embodiments for using such an electronic device. In some embodiments, the electronic device may be a portable electronic device that also includes other functions such as a personal digital assistant and / or music player, such as a mobile phone, tablet computer, wearable electronic device with wireless communication capabilities (such as a smartwatch), etc. Exemplary embodiments of the portable electronic device include, but are not limited to, carrying... Alternatively, it could be a portable electronic device with another operating system. The aforementioned portable electronic device could also be other portable electronic devices, such as laptops. It should also be understood that in some other embodiments, the aforementioned electronic device may not be a portable electronic device, but rather a desktop computer.

[0113] For example, Figure 1 A schematic diagram of the structure of electronic device 100 is shown. Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0114] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0115] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0116] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0117] The processor 110 may also include a memory for storing instructions and data.

[0118] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0119] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0120] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0121] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.

[0122] A modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal.

[0123] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc.

[0124] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-CDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0125] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0126] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0127] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0128] The ISP is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye.

[0129] Camera 193 is used to capture still images or videos.

[0130] A digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals.

[0131] Video codecs are used to compress or decompress digital video.

[0132] NPU stands for Neural-Network (NN) Computing Processor. By drawing inspiration from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can quickly process input information and continuously learn on its own.

[0133] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the electronic device 100.

[0134] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area.

[0135] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0136] Audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Audio module 170 can also be used for encoding and decoding audio signals.

[0137] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0138] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0139] Microphone 170C is used to convert sound signals into electrical signals.

[0140] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals.

[0141] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0142] The 180K touch sensor, also known as a "touch panel," is used to detect touch operations applied to or near it.

[0143] Figure 2 This is a software structure block diagram of an electronic device 100 according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the system library layer, and the kernel layer. The application layer may include a series of application packages.

[0144] like Figure 2 As shown, the application layer can include camera, settings, skin modules, user interface (UI), third-party applications, etc. Third-party applications can include gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, SMS, etc.

[0145] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer may include some predefined functions.

[0146] like Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0147] The window manager is used to manage windowed applications. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture screenshots. The content provider stores and retrieves data, making this data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0148] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views.

[0149] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0150] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0151] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0152] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0153] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0154] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0155] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0156] A 2D graphics engine is a graphics engine for 2D drawing.

[0157] In addition, the system library may also include status monitoring service modules, such as a physical status recognition module for analyzing and recognizing user gestures; and a sensor service module for monitoring sensor data uploaded by various sensors at the hardware layer to determine the physical status of the electronic device 100.

[0158] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0159] The hardware layer can include various types of sensors, such as Figure 1 The various sensors introduced in the text.

[0160] In summary Figure 1 and Figure 2 The electronic devices described in this application embodiment include devices that support voice assistant functions, such as mobile terminals, tablet computers, personal computers, smart screens, and smartwatches.

[0161] Currently, most electronic devices' voice assistants rely on large-scale cloud-based algorithms to execute user commands. For example, such as... Figure 3 The diagram illustrates a system architecture provided in an embodiment of this application. This system architecture mainly includes an edge device and a cloud side. The actuator of the edge device acquires user instructions and provides them to the cloud side. The cloud-side big model performs intent understanding and orchestration on the user instructions and sends the orchestration results to the actuator of the edge device. The edge device parses the orchestration results, calls the corresponding functions, sets parameters according to the instructions of the big model, and finally executes the system on the edge device.

[0162] It is evident that current intelligent assistants, when executing user commands, rely solely on analyzing the user's basic intent based on the voice commands, resulting in a low success rate and an inability to provide a personalized experience. Furthermore, the need for edge devices to pass user commands to a large cloud-based model for orchestration introduces communication latency, thereby impacting the user experience.

[0163] Based on this, this application provides a data processing method, apparatus, and electronic device. The edge device provides the cloud-side large model with association information related to the device information relationship graph, enabling the cloud-side large model to optimize the orchestration results based on the association information. The edge device can optimize the workflow corresponding to the orchestration results based on the personal knowledge graph (PKG) to improve the success rate of executing user commands. Simultaneously, the edge device can be configured with a set of high-frequency, essential pre-set user commands and a set of pre-set workflows, allowing certain high-frequency user commands to be executed on the edge device without cloud-side large model orchestration, reducing communication latency and improving user experience.

[0164] The data processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0165] In scenario one, the edge device provides the cloud-side big model with association information related to the device information relationship graph, enabling the cloud-side big model to optimize the orchestration results based on the association information, improve the success rate of executing user commands, and thus enhance the user experience.

[0166] like Figure 4 The diagram illustrates a schematic flowchart of a data processing method provided in an embodiment of this application. This method can construct a device information relationship graph based on attributes such as device status, application purpose, category, and logical relationships. It then determines associated information based on the device information relationship graph. The cloud-based large model optimizes the orchestration results based on the associated information, ensuring the rationality and success rate of execution and improving user experience. The method is described in detail below.

[0167] S401, the end-side device obtains the original orchestration results from the cloud side.

[0168] For example, the edge device can upload user commands to the cloud, where the cloud-based big model processes the user commands to obtain the raw orchestration results. The raw orchestration results are used to represent the workflow corresponding to the user commands.

[0169] S402, the end-side equipment determines the associated information based on the original arrangement results and the equipment information relationship diagram.

[0170] It should be noted that a device information relationship graph can be pre-configured on the endpoint device. This device information relationship graph is a graphical representation of the relationships between relevant information on the endpoint device. This information may include device status, the purpose, category, and logical relationships of applications on the endpoint device. This application does not limit the specific method of representing the relationships between related endpoint devices.

[0171] For example, such as Figure 5The diagram illustrates a device information relationship graph. This graph includes device status information and various categories of applications. Device status information indicates the current state of the device. This information may include language, battery level, and other details. Users can modify certain device information through settings; for example, they can set the language information in the device status information. Furthermore, multiple applications on the device can be categorized by function, such as entertainment applications (Apps), utility applications, communication applications, media applications, and educational applications. Each category can contain multiple applications; for example, entertainment applications include Game 1 and Game 2, and media applications include Music 1 and Music 2. The device status information and the various applications are interconnected, allowing for optimization of the original arrangement using the device information relationship graph.

[0172] For example, when a user commands "play a song," the device uploads the command to the cloud-based big data model. The raw orchestration result obtained from the cloud-based big data model indicates that the workflow is to open the music application and play music. The associated information determined by the device based on this raw orchestration result and the device information relationship graph can be power information, because playing music consumes device power.

[0173] For example, when a user issues the command "play a song," the edge device uploads the user's command to the cloud-based big data model. The original orchestration result obtained from the cloud-based big data model indicates that the workflow is to open the music application and play music. The association information determined by the edge device based on this original orchestration result and the device information relationship graph may include music application 1 and music application 2 on the edge device, because the edge device can play music in both music application 1 and music application 2.

[0174] S403, the terminal device sends association information to the cloud side.

[0175] In other words, in step S402 above, the end-side device obtains the associated information from the device information relationship graph based on the original arrangement results, and synchronizes the associated information to the cloud side.

[0176] It should be noted that the terminal device must send at least the associated information to the cloud side.

[0177] In other words, after the end-side device determines the association information, it can send the association information to the cloud; or send the association information and user instructions to the cloud; or send the association information and the original orchestration result to the cloud; or send the association information, user instructions, and the original orchestration result to the cloud. When the end-side device sends multiple types of information to the cloud, it can package these multiple types of information and send them to the cloud. For example, the end-side device can package the association information, user instructions, and the original orchestration result and send them to the cloud.

[0178] S404, the cloud side optimizes the original orchestration result based on the associated information to obtain the optimized orchestration result.

[0179] For example, the cloud-based big model can optimize the execution logic of the workflow based on the associated information to obtain an optimized orchestration result.

[0180] It should be noted that the optimized orchestration results can have two levels of meaning. First, the edge devices can execute the workflow corresponding to the cloud-side optimized orchestration results, ensuring the success rate of user command execution. Second, the optimized orchestration results involve the device information relationship graph, making the execution logic of the edge devices more aligned with the device information, thereby improving the execution effect of user commands.

[0181] S405, the end-side device determines the workflow based on the optimized orchestration results from the cloud side.

[0182] S406, End-side device executes workflow.

[0183] It should be noted that the orchestration result is used to represent the workflow corresponding to the user command. The execution and optimization of the workflow corresponding to the orchestration result by the end device can improve the success rate of executing user commands and bring a better user experience to the execution process.

[0184] For example, when a user commands "play a song," the device uploads the command to the cloud-based big data model. The initial arrangement result obtained from the cloud-based big data model indicates a workflow of opening the music application and playing music. The associated information determined by the device based on this initial arrangement result and the device information relationship graph can be battery information, since playing music consumes the device's power. When the battery is low, the cloud-based big data model, based on the battery information, can optimize the workflow by displaying a prompt message or playing a notification sound to remind the user to charge.

[0185] For ease of understanding, such as Figure 6 As shown, taking a mobile terminal as an example, a set of data processing comparison diagrams are presented, illustrating the difference in the execution workflow of the terminal device before and after the orchestration results are optimized.

[0186] like Figure 6As shown in (a), the interface of the recording application before optimization is displayed. This interface shows multiple controls and the playback progress of the recording file. The multiple controls include a text-to-speech control, an editing control, a skip-mute control, and a speed-up control, etc. Users can trigger different functions through different controls.

[0187] like Figure 6 As shown in (b), the optimized recording application interface is displayed. This interface shows multiple controls, the playback progress of the recording file, and a prompt message 601. Prompt message 601 informs the user that the recording language is different from the system language, and the user can choose to translate the recording language into the system language.

[0188] It's easy to see that the optimized recording application's interface can prompt users to translate the recording language into the system language. If users choose to translate the recording language into the system language, it will improve the efficiency of users obtaining information from the recording file.

[0189] For example, the edge device sends the user command "play recording file 1" to the cloud side. The cloud-side big data model, based on the user command, obtains the original arrangement result, which includes opening the recording application and playing recording file 1. The edge device, based on the original arrangement result and the device information relationship graph, determines that the language information in the device status information is the associated information, and sends this associated information, the user command, and the original arrangement result to the cloud side. The cloud-side big data model determines that the system's language information is inconsistent with the recording language, and can optimize the original arrangement result to obtain an optimized arrangement result, which includes opening the recording file, asking the user if they need the recording language translated into the system language, and playing the recording file. The edge device executes the corresponding workflow based on the optimized arrangement result, and the edge device can then display... Figure 6 The interface shown in (b) is shown in the image.

[0190] like Figure 7 As shown, a set of data processing comparison diagrams are presented, illustrating the difference in the workflow executed by the end-side device before and after the orchestration results are optimized.

[0191] like Figure 7 Figure (a) illustrates a data processing flow diagram before optimization. The actuator of the edge device receives the user instruction "Send the contract to Xiao Wang" and sends the user instruction to the cloud side. The workflow corresponding to the original orchestration result obtained by the cloud-side big model based on the user instruction includes locating the contact Xiao Wang and sending the contract to Xiao Wang. During the execution of the workflow corresponding to the original orchestration result, the edge device first matches the Communication 2 application, but the contacts in Communication 2 application do not include Xiao Wang, although file sharing permissions are supported. Therefore, contact location fails, contract sending fails, and user instruction execution fails.

[0192] like Figure 7 Figure (b) illustrates an optimized data processing flow. The actuator of the edge device receives the user instruction "Send the contract to Xiao Wang" and sends the user instruction to the cloud side. The cloud-side big model's workflow corresponding to the original orchestration result obtained from the user instruction includes locating the contact person Xiao Wang and sending the contract to Xiao Wang. The edge device determines that there are multiple communication applications based on the original orchestration result and the device information relationship graph, including Communication 1 application, Communication 2 application, and Communication 3 application. Among them, the contact person in Communication 1 application includes Xiao Wang and supports file sharing permissions; the contact person in Communication 2 application does not include Xiao Wang, but supports file sharing permissions; the contact person in Communication 3 application includes Xiao Wang, but does not support file sharing permissions. The edge device uses the above three applications belonging to the communication category as association information and sends the association information, user instruction, and original orchestration result to the cloud side. The cloud-side big model optimizes the original orchestration result based on the association information to obtain an optimized orchestration result and sends the optimized orchestration result to the edge device. The optimized orchestration process includes selecting a target application from the three applications mentioned above, which includes Xiao Wang as a contact and supports file sharing permissions; opening the target application; and sending the contract to Xiao Wang. Based on the optimized orchestration, the terminal device selects Communication 1, which includes Xiao Wang as a contact and supports file sharing permissions, as the target application from the three applications. It then opens Communication 1, and after user confirmation, sends the contract to Xiao Wang, thus successfully executing the user's command.

[0193] It's easy to understand that in the example above, the contacts of the three communication applications, as well as whether file sharing permissions are supported, are information held by the end device.

[0194] It is evident that optimizing the orchestration results through the cloud-based large model can improve the success rate of executing user commands.

[0195] Scenario 2: The edge device optimizes the workflow corresponding to the orchestration results based on the personal knowledge graph PKG, so that the execution of the edge device can provide users with a personalized experience.

[0196] like Figure 8 The diagram illustrates a schematic flowchart of a data processing method provided in an embodiment of this application. This method optimizes the workflow corresponding to the orchestration results based on a Personal Knowledge Graph (PKG), improving the success rate of user command execution while providing a personalized user experience. The method is described in detail below.

[0197] S801, the end-side device obtains the orchestration results from the cloud side.

[0198] For example, the edge device can upload user instructions to the cloud, where the cloud-based big model processes the user instructions to obtain an orchestration result, which is used to represent the workflow corresponding to the user instructions.

[0199] S802, the end-side device determines whether there is a missing match based on the arrangement result.

[0200] For example, when the end-side device determines that there is no missing match, the following step S805 can be performed; when the end-side device determines that there is a missing match, the following step S803 can be performed.

[0201] It should be noted that the orchestration results have corresponding workflows. The process by which the end device determines whether the orchestration results match the missing parts can be understood as determining whether the workflow corresponding to the orchestration results can be successfully executed.

[0202] For example, if the workflow corresponding to the orchestration result includes step one and step two, and the end device determines that step one cannot be executed successfully, it indicates that the orchestration result is missing, and the user command will not be executed successfully.

[0203] S803, the end-side device determines the matching information based on the arrangement results and the personal knowledge graph PKG.

[0204] It should be noted that when the end device determines that a match is missing based on the orchestration result, it can combine the Personal Knowledge Graph (PKG) to determine the matching information. The matching information is related to the missing information in step S802. The matching information can be used to supplement the missing information so that the end device can execute user instructions.

[0205] like Figure 9 The diagram illustrates a Personal Knowledge Graph (PKG). A PKG is a semantic network used to describe personal entities, attributes, and their relationships. It is not merely a graphical representation of data, but a semantic description of personal knowledge information, capable of being understood and processed by computers. A PKG typically includes key components such as entities, attributes, and relationships. Entities represent basic concepts in personal knowledge, such as people. Attributes describe the characteristics of entities, such as preferences and interests. Relationships represent the relationships between entities, such as the relationships between people.

[0206] by Figure 9 Taking PKG as an example, the first layer includes the central user entity; the second layer includes various attributes, such as interests, social interactions, and events; the third layer includes information refined from the attributes, such as interests including movies, music, etc.; social interactions including family, colleagues, and friends, etc.; and events including time and location, etc.

[0207] In simple terms, personal knowledge graphs (PKGs) represent user habits, characteristics, relationships, and other information. Using this information can optimize the workflow corresponding to the orchestration results, making the execution of user commands by the edge devices more in line with the user's personality.

[0208] S804, the end-side device determines the workflow based on the arrangement results and matching information.

[0209] It should be noted that the end-side device can optimize the orchestration results based on the matching information to obtain the final workflow.

[0210] S805, the end-side device executes the workflow.

[0211] It should be noted that if the end-side device determines that there are no missing matches in the orchestration result, the workflow executed is the workflow corresponding to the orchestration result. If the end-side device determines that there are missing matches in the orchestration result, the workflow executed is the workflow corresponding to the orchestration result after optimizing the matching information.

[0212] For example, the workflow corresponding to the orchestration result includes steps one and two. If the end device determines that there are no missing matches based on the orchestration result, it can directly execute steps one and two. If the end device determines that step one cannot be executed, it indicates that there are missing matches. The end device can determine that step three can replace step one based on the Personal Knowledge Graph (PKG) and execute steps three and two.

[0213] For ease of understanding, such as Figure 10 As shown, a set of data processing comparison diagrams are presented, illustrating the difference in the execution workflow of the edge device before and after optimization based on the Personal Knowledge Graph (PKG).

[0214] like Figure 10 Figure (a) shows a schematic diagram of the data processing flow before optimization. The edge device obtains the user instruction "send a message to my cousin" and sends the user instruction to the cloud side. The workflow corresponding to the orchestration result obtained by the cloud side big model based on the user instruction includes retrieving the contact "cousin" and sending a text message to "cousin". During the process of executing the workflow corresponding to the orchestration result, the edge device fails to find the contact "cousin", therefore, the contact location fails and the user instruction execution fails.

[0215] like Figure 10Figure (b) illustrates an optimized data processing flow diagram. The edge device obtains the user instruction "send a message to my cousin" and sends the user instruction to the cloud side. The workflow corresponding to the orchestration result obtained by the cloud-side big model based on the user instruction includes locating the contact "cousin" and sending a text message to "cousin". If the edge device fails to find the contact "cousin" during the execution of the workflow corresponding to the orchestration result, that is, after the edge device determines that the match is missing, it can convert "cousin" into the actual name "Zhang San" based on the personal knowledge graph PKG. The actual name "Zhang San" can be understood as the matching information in step S803. The workflow determined by the edge device based on the orchestration result and the matching information includes retrieving the contact "Zhang San" and sending a text message to "Zhang San". This process can use workflow parameter personalization filling technology, that is, in the workflow, the parameters in the workflow are dynamically adjusted and optimized according to the actual situation to achieve more personalized and accurate results. The edge device can send a text message to "Zhang San" after retrieving the contact "Zhang San" and obtaining user confirmation, so that the user instruction is successfully executed.

[0216] like Figure 11 As shown, a set of data processing comparison diagrams are presented, illustrating the difference in the execution workflow of the edge device before and after optimization based on the Personal Knowledge Graph (PKG).

[0217] like Figure 11 Figure (a) shows a schematic diagram of the data processing flow before optimization. The edge device receives the user instruction "Make my image look better" and sends the user instruction to the cloud side. The cloud side's large model obtains the arrangement result based on the user instruction. However, the edge device cannot recognize the image editing operation corresponding to "make my image look better", therefore, the edge device fails to execute the user instruction.

[0218] like Figure 11 Figure (b) illustrates an optimized data processing flow. The edge device receives the user's instruction "Make me look better" and sends it to the cloud. The cloud-side big model generates an arrangement result based on the user's instruction. However, the edge device cannot recognize the image editing operation corresponding to "look better". The edge device determines the user's image editing habits based on the Personal Knowledge Graph (PKG), confirms the editing operation for the specific image according to the user's image editing habits, and completes the image editing after user confirmation. The edge device successfully executes the user's instruction.

[0219] It is evident that optimizing the workflow corresponding to the orchestration results through end-side devices can improve the success rate of executing user commands.

[0220] Scenario 3: Configure high-frequency, essential pre-set user command sets and corresponding pre-set workflow sets on the edge devices to reduce interaction with the cloud-side large model, thereby reducing communication latency and improving user experience.

[0221] like Figure 12 The diagram illustrates a schematic flowchart of a data processing method provided in an embodiment of this application. This method allows for the configuration of a set of frequently used and essential pre-defined user commands and corresponding pre-defined workflows on the endpoint device, reducing interaction between the endpoint device and the cloud-side large model, thereby reducing communication latency and improving user experience. The method is described in detail below.

[0222] S1201 is configured with a set of preset user instructions and a set of preset workflows in the end-side device.

[0223] It should be noted that the set of preset user instructions and the set of preset workflows are in one-to-one correspondence. The set of preset user instructions can include multiple high-frequency, essential preset user instructions. In this context, user instructions can be used to reflect user intent, and preset user instructions can be understood as preset user intents.

[0224] S1202, the end device generates an embedding vector representation of the user instruction based on this user instruction.

[0225] It should be noted that embedding vector representation is a method of capturing the meaning or features of data as digital representations of points in a multi-dimensional space, enabling machines to efficiently process and compare this data. In natural language processing, embedding vector representation typically refers to converting words, sentences, or documents into fixed-dimensional vectors that capture the semantic and syntactic relationships between words. In the embodiments of this application, embedding vector representation technology can be used to process user commands in order to determine the similarity between the user command and preset user commands.

[0226] For example, before generating the embedded vector representation of the user instruction based on the current user instruction, the edge device can use sentence splitting technology to split the current user instruction into multiple sub-instructions to represent multiple user sub-intents.

[0227] Sentence splitting is a task in natural language processing that breaks down a continuous text into individual sentences. This can be achieved by identifying specific end markers (e.g., periods, question marks, or exclamation marks) or by using machine learning models to predict sentence boundaries. Sentence splitting is crucial for applications such as text analysis, information extraction, and dialogue systems. In this embodiment, user instructions may be complex long sentences. Sentence splitting techniques can break down a complex user instruction into multiple easily understood short sentences to identify the user's intent.

[0228] like Figure 13The diagram illustrates a process for determining the similarity between a user instruction and a pre-set set of user instructions using sentence segmentation technology.

[0229] First, complex long user commands can be broken down into short, easily understandable sentences (q). User command q can be understood as a short, easily understood sentence obtained after the sentence breakdown. User command q can be used to express user intent. The preset user command set configured on the terminal device includes multiple preset user commands p1, p2, p3… and the preset workflow set includes multiple preset workflows W1, W2, W3…

[0230] The edge device vectorizes the user command q (the embedding vector representation of the user command) to obtain E. Q (q) Vectorize p from the preset user instruction set. i (The embedded vector representation of the pre-set user instructions) yields E P (p i ). For E Q (q) and E P (p i Calculate the vector similarity SIM(q,p) i ).

[0231] Specifically, the similarity between the embedding vector representation of the user instruction and the embedding vector representation of each preset user instruction in the preset user instruction set can be calculated, and the target preset user instruction p can be selected from the preset user instruction set based on the similarity threshold. n User command q and target preset user command p n If the similarity is greater than the similarity threshold, the user command q and the target preset user command p... n A successful match indicates that the target has a pre-set user command p. n This can represent the user intent corresponding to the user command q. The edge device pre-configures the user command p according to the target. n Filter the target preset workflow W from the preset workflow set. n End-side devices can execute target-preset workflows.

[0232] It's easy to understand that if a user command is a complex long sentence, sentence splitting technology can be used to obtain multiple short sentences, thereby identifying multiple sub-intents of the user; if the user command itself is a simple short sentence, sentence splitting can ultimately identify a single user intent.

[0233] S1203, the edge device matches the embedded vector representation of the user instruction with the preset user instructions in the preset user instruction set to determine the similarity.

[0234] It should be noted that similarity is a key feature of embedded vector representation. The similarity between two vectors can be measured by calculating the distance between them. In the embodiments of this application, the method for determining similarity is not limited.

[0235] For example, based on sentence segmentation technology, the edge device splits the user command into multiple sub-commands and generates embedding vector representations of each sub-command. The embedding vector representations of each sub-command are then matched with preset user commands in a preset user command set to determine the similarity.

[0236] When the similarity is greater than the similarity threshold, step S1204 can be executed; when the similarity is less than or equal to the similarity threshold, step S1207 can be executed.

[0237] S1204, when the similarity is greater than the similarity threshold, determine the preset workflow.

[0238] It should be noted that when the similarity is greater than the similarity threshold, it means that the user command successfully matches any preset user command in the preset user command set, and the match has a corresponding preset workflow.

[0239] S1205, the end-side equipment optimizes the preset workflow based on at least one of the technologies of slot filling and substitution digestion to obtain an optimized workflow.

[0240] For example, parameter information in the user instruction can be extracted through sentence splitting and referential resolution techniques, and slots can be filled in the preset workflow to obtain an optimized workflow.

[0241] The following section provides a detailed introduction to the technologies used to optimize pre-built workflows.

[0242] Coreference resolution is a task in natural language processing that aims to identify the specific objects or entities referred to by pronouns and noun phrases in text. For example, in the sentence "book a meeting with Zhang San at 3 pm and send him the meeting schedule," the pronoun "he" refers to the object "Zhang San." Coreference resolution techniques require analyzing the contextual and semantic information of the text to determine the correct referential relationship. In the embodiments of this application, coreference resolution techniques can be used to improve the depth and accuracy of understanding user instructions.

[0243] Slot filling is a task in natural language processing. For example, in task-oriented dialogue systems, it involves extracting specific information from user input and mapping that information into predefined slots. These slots represent key information the dialogue system needs from the user to complete a specific task or perform a specific operation. Slot filling often works in conjunction with intent recognition. Intent recognition is used to determine the user's overall goal or intent, while slot filling further refines these intents, extracting the specific intents required to perform them.

[0244] like Figure 14 As shown, a schematic diagram illustrates a process for optimizing a pre-set workflow using slot filling technology.

[0245] It should be noted that user commands, while representing user intent, also carry specific parameter information. User commands are variable, and the specific parameter information in each user command can differ. Therefore, the preset user commands in the preset user command set of the edge device can represent user intent, but may not include specific parameter information. To ensure the correct execution of user commands, before executing the final workflow, techniques such as slot filling are needed to optimize the preset workflow using specific parameter information.

[0246] For example, Figure 14 User instruction q and target preset user instruction p in n Match successful. The user command 'q' and the target preset user command 'p' can be used as a reference. n Identify the slots that need to be filled, and these slots will be populated with specific parameter information. Target preset user command p n Corresponding to the pre-defined target workflow W n The parameter information is obtained from the user command q, and the slots are filled into the target preset workflow. In other words, the specific parameter information is filled into the corresponding slots to obtain the workflow that the end device finally needs to execute.

[0247] For example, a user instruction might be "Play Li Si's songs through the Music 1 application," and a target preset user instruction matching this instruction might be "Play XXX's (singer's name)'s songs through XXX (application)." The slots corresponding to the application and the singer's name in the target preset user instruction need to be filled with specific parameter information. The target preset workflow corresponding to this user instruction includes, firstly, opening XXX (application), and secondly, playing XXX (singer's name)'s songs. Slot identification is performed on the user instruction "Play Li Si's songs through the Music 1 application," obtaining parameter information including "Music 1 application" and "Li Si." This parameter information is then filled into the target preset workflow, resulting in a workflow that includes, firstly, opening the Music 1 application, and secondly, playing Li Si's songs.

[0248] It should be noted that the pre-set workflow can be optimized through slot filling and substitution digestion techniques.

[0249] For example, a user instruction is "Play Li Si's song through the Music 1 application and share his song to a social application." Using sentence segmentation technology, this is divided into two short sentences: "Play Li Si's song through the Music 1 application" and "Share his song to a social application." The target preset user instructions that successfully match these two short sentences are target preset user instruction 1 "Play XXX's (singer's name)'s song through XXX (application)" and target preset user instruction 2 "Share XXX's (singer's name)'s song to XXX (application)." In target preset user instruction 1, the slots corresponding to the application for playing the song and the singer's name need to be filled with specific parameter information; in target preset user instruction 2, the slots corresponding to the application for sharing the song and the singer's name need to be filled with specific parameter information. Target preset workflow 1 corresponding to target preset user instruction 1 includes the first step of opening XXX (the application for playing the song) and the second step of playing XXX's (singer's name) song. Target preset workflow 2 corresponding to target preset user instruction 2 includes the first step of opening XXX (the application for sharing the song) and the second step of sharing XXX's (singer's name) song. Slot identification is performed on the short phrase "Play Li Si's songs through the Music 1 application" to obtain parameter information including "Music 1 application" and "Li Si". This parameter information is then filled into the target preset workflow 1, resulting in workflow 1 that includes the first step of opening the Music 1 application and the second step of playing Li Si's songs. Slot identification is also performed on the short phrase "Share his songs to a social application" to obtain parameter information including "him" and "social application". Using a referential resolution technique, it is determined that "him" in the user command refers to "Li Si". The parameter information "Li Si" and "social application" is then filled into the target preset workflow 2, resulting in workflow 2 that includes the first step of opening the social application and the second step of sharing Li Si's songs.

[0250] S1206, End-side devices execute optimized workflows.

[0251] For example, after the above-mentioned slot filling, or slot filling and reference resolution techniques, are used to supplement the target preset workflow with specific parameter information, the resulting workflow can be understood as an optimized workflow.

[0252] It should be noted that in step S1205 above, the end-side device optimizes the preset workflow based on at least one of the techniques of slot filling and reference resolution, which can improve the accuracy and effectiveness of executing user instructions. It is easy to understand that if the user instruction is a simple instruction, after successfully matching with a preset user instruction in the preset user instruction set, the corresponding preset workflow can be executed directly.

[0253] S1207: When the similarity is less than or equal to the similarity threshold, the matching is determined to be unsuccessful, and the user instruction is sent to the cloud side.

[0254] It should be noted that when the similarity is less than or equal to the similarity threshold, it means that the user command fails to match any of the preset user commands in the preset command set and there is no corresponding preset workflow. In this case, the user command can be sent to the cloud side, where the cloud side's large model will process it to obtain the orchestration result. The end device can then execute the workflow corresponding to the orchestration result.

[0255] S1208, the terminal device records the number of times a user instruction of the same type fails to match. When the number of times exceeds the threshold, the user instruction of this type is updated to the preset user instruction set.

[0256] For example, the edge device can preset a limit of 20 failed matches for the same type of user command within a certain period (e.g., 10 days). When the number of failed matches exceeds a threshold (15 times), the user command can be updated to a preset user command set, and the corresponding workflow can also be updated to a preset workflow set. By continuously updating the preset user command set and preset workflow set, the accuracy of executing user commands can be improved, while reducing interaction with the cloud-side large model, reducing communication latency, and improving user experience.

[0257] For ease of understanding, such as Figure 15 As shown, a schematic diagram of a data processing flow is presented, which demonstrates how, in a scenario where the terminal device is configured with a set of pre-set user instructions and a set of pre-set workflows, the interaction with the large cloud model is reduced during the execution of user instructions.

[0258] like Figure 15 As shown, the edge device receives the user instruction "Schedule a meeting with Zhang San at 3 PM and share the schedule with him via SMS". The edge device uses techniques such as sentence segmentation, referential resolution, and slot filling to perform similarity matching between the user's complex long sentence instruction and various preset user instructions in a preset user instruction set. A successful match yields the first target preset user instruction "Schedule a meeting with Zhang San at 3 PM" and the second target preset user instruction "Send the 3 PM schedule content to Zhang San via SMS". Based on the first target preset user instruction, the edge device determines the workflow for setting the schedule from the preset workflow set, including adding the schedule. Based on the second target preset user instruction, the edge device determines the workflow for sharing the schedule via SMS from the preset workflow set, including retrieving the schedule and sending an SMS message about the schedule content to Zhang San. The edge device executes both of these workflows to correctly execute the user instruction.

[0259] It should be noted that the above three scenarios introduced three aspects of improving user experience during the execution of user commands by intelligent assistants. It's easy to understand that in practical applications, at least one of these methods can be used to improve the user experience. The following section will combine... Figure 16 This diagram illustrates a data processing method that employs solutions from the three scenarios described above to enhance user experience. The method will be described in detail below.

[0260] S1601 is configured with a set of preset user instructions and a set of preset workflows in the end-side device.

[0261] It should be noted that the set of preset user instructions and the set of preset workflows are in one-to-one correspondence. The set of preset user instructions can include multiple high-frequency, essential preset user instructions. In this context, user instructions can be used to reflect user intent, and preset user instructions can be understood as preset user intents.

[0262] S1602, the edge device matches the embedded vector representation of the user instruction with the preset user instructions in the preset user instruction set to determine the similarity.

[0263] S1603, when the similarity is less than or equal to the similarity threshold, the end device sends the user command to the cloud side.

[0264] S1604, the cloud side determines the original arrangement result based on the user's instructions.

[0265] It should be noted that when the similarity is less than or equal to the similarity threshold, it means that the user command fails to match all the preset user commands in the preset command set. The terminal device does not have a preset workflow that can be executed. The user command can be uploaded to the cloud side, where the cloud side's large model will orchestrate the user command to obtain the original orchestration result. The original orchestration result is used to represent the workflow corresponding to the user command.

[0266] S1605, the end-side device determines the associated information based on the original orchestration results and device information relationship map obtained from the cloud side.

[0267] S1606, the cloud side optimizes the original orchestration result based on the association information obtained from the end-side device to obtain the optimized orchestration result.

[0268] It should be noted that after the end-side device determines the association information, it can send the association information to the cloud; or send the association information and user instructions to the cloud; or send the association information and the original orchestration result to the cloud; or send the association information, user instructions, and the original orchestration result to the cloud. The cloud-side large model can optimize the execution logic of the workflow based on the association information to obtain an optimized orchestration result.

[0269] S1607, the end-side device determines the missing match based on the optimized orchestration results from the cloud side.

[0270] S1608, the end-side device determines matching information based on the optimized orchestration results and the personal knowledge graph PKG.

[0271] It should be noted that if the edge device determines that the workflow corresponding to the original orchestration result cannot be executed successfully, it indicates that the optimized orchestration result is missing a match. The edge device can combine the Personal Knowledge Graph (PKG) to determine the matching information. The matching information is related to the missing information, and the matching information can be used to supplement the missing information so that the edge device can execute user commands.

[0272] S1609, the end-side device determines the workflow based on the optimized orchestration results and matching information.

[0273] S1610, End-side device executes workflow.

[0274] It should be noted that the end-side device ultimately executes a workflow optimized based on the matching information.

[0275] During the aforementioned data processing, the edge device prioritizes matching pre-set user instructions based on the user's commands. If a match is successful, the pre-set workflow is executed directly. If a match fails, the cloud-side big data model processes the user instructions to obtain the orchestration result. The edge device can determine the associated information by combining the device information relationship graph, and the cloud-side big data model optimizes the orchestration result. Simultaneously, the edge device can further optimize the orchestration result issued by the cloud by combining it with the user's knowledge graph (PKG). The edge device then executes the final optimized workflow. This process, while minimizing communication and interaction with the cloud, improves the accuracy of executing user commands, making the execution process more personalized and enhancing the user experience.

[0276] like Figure 17 The diagram shows a schematic flowchart of a data processing method, which adopts the methods of Scenario 1 and Scenario 2 mentioned above to improve user experience. The method will be described in detail below.

[0277] S1701, the cloud side determines the original arrangement result based on the user's instructions.

[0278] S1702, the end-side device determines the associated information based on the original orchestration results and device information relationship map obtained from the cloud side.

[0279] S1703, the cloud side optimizes the original orchestration result based on the association information obtained from the end-side device to obtain the optimized orchestration result.

[0280] It should be noted that after the end-side device determines the association information, it can send the association information to the cloud; or send the association information and user instructions to the cloud; or send the association information and the original orchestration result to the cloud; or send the association information, user instructions, and the original orchestration result to the cloud. The cloud-side large model can optimize the execution logic of the workflow based on the association information to obtain an optimized orchestration result.

[0281] S1704, the end-side device determines the missing match based on the optimized orchestration results from the cloud side.

[0282] S1705, the end-side device determines matching information based on the optimized orchestration results and the personal knowledge graph PKG.

[0283] It should be noted that if the edge device determines that the workflow corresponding to the original orchestration result cannot be executed successfully, it indicates that the optimized orchestration result is missing a match. The edge device can combine the Personal Knowledge Graph (PKG) to determine the matching information. The matching information is related to the missing information, and the matching information can be used to supplement the missing information so that the edge device can execute user commands.

[0284] S1706, the end-side device determines the workflow based on the optimized orchestration results and matching information.

[0285] S1707, End-side device executes workflow.

[0286] It should be noted that the end-side device ultimately executes a workflow optimized based on the matching information.

[0287] During the aforementioned data processing, the edge device uploads user commands to the cloud, where a large cloud model processes them to produce orchestration results. The edge device can then use a device information relationship graph to determine associated information, and the cloud model further optimizes the orchestration results. Simultaneously, the edge device can further optimize the orchestration results from the cloud by combining them with a personal knowledge graph (PKG). The edge device then executes the final optimized workflow. This process improves the accuracy of executing user commands, making the execution process more personalized and enhancing the user experience.

[0288] like Figure 18 The diagram illustrates a schematic flowchart of a data processing method provided in an embodiment of this application. This method can be applied to, for example... Figure 3 In the system architecture shown, where, Figure 18 The server shown is a cloud-side device, and the electronic device is an edge-side device. The method will be described in detail below.

[0289] S1801, the server determines the first arrangement result based on the first user instruction.

[0290] For example, an electronic device acquires a first user instruction and uploads it to a server. The server's large model can process the first user instruction to obtain a first orchestration result, which can be understood as the original orchestration result obtained by the cloud-side large model from processing the first user instruction.

[0291] It should be noted that the arrangement results can be used to represent the workflow corresponding to user instructions.

[0292] S1802, the server sends the first arrangement result, and the electronic device accordingly obtains the first arrangement result.

[0293] It should be noted that the first arrangement result corresponds to the first user instruction.

[0294] S1803, the electronic device determines at least one piece of device information from multiple pieces of device information based on the first arrangement result.

[0295] It is understandable that at least one device information is the related information mentioned above.

[0296] For example, multiple device information includes the status information of the electronic device, application information, or the relationship between the status information of the electronic device and the application information, wherein the application information includes the type of the application, and the status information of the electronic device includes language information and / or battery information.

[0297] It is understandable that information from multiple devices can be as follows: Figure 5 The diagram shows the device information relationships. Application information can include application type, purpose, logical relationships, etc. The electronic device's status information indicates its current state, and this status information can also be set by the user. The diagram shows the current state of the electronic device and its relationships with various applications on the device.

[0298] It should be noted that electronic devices can be pre-configured with device information. Based on the first arrangement result, the electronic device can filter out at least one piece of device information related to the first arrangement result from multiple pieces of device information; the filtered at least one piece of device information can be used to optimize the original arrangement result.

[0299] Optionally, the multiple device information includes application information; when the first arrangement result is related to the first application, information of a first type of application related to the first application is determined from the multiple device information, and the information of the first type of application belongs to the application information.

[0300] In other words, when the first orchestration result is related to a certain application, the electronic device can filter out other applications of the same type as the application as associated information, so as to facilitate the server to optimize the orchestration result.

[0301] For example, the information of the first type of application is used to indicate that the number of applications belonging to the first type is N, where N is a positive integer greater than or equal to 2.

[0302] It should be noted that the information about the first type of applications can be the number of applications of the first type, or it can be the specific applications of the first type (e.g., the name or identifier of the application). Based on the information about the first type of applications, the server can determine that there are multiple applications of the first type in the electronic device. The optimized orchestration result obtained through this information allows the electronic device to select a more suitable application from the same type of applications to execute user instructions.

[0303] Optionally, the multiple device information includes power information; when the first arrangement result is related to power information, the power information is determined from the multiple device information.

[0304] It should be noted that battery information indicates the current battery level of an electronic device. The battery level of an electronic device will be affected when the device executes user commands.

[0305] For example, when the power information indicates that the power of the electronic device is below the power threshold, the first workflow includes a first prompt message, which prompts the user to charge the electronic device.

[0306] In other words, when the battery is low, the optimized orchestration result can remind the user to charge the electronic device, thus avoiding interruption of user command execution due to insufficient battery power.

[0307] Optionally, the multiple device information includes language information, which is used to represent the system language; when the first arrangement result is related to the language information, the language information is determined from the multiple device information.

[0308] For example, when the working language corresponding to the first workflow is inconsistent with the system language, the first workflow includes displaying a second prompt message, which prompts the user to switch the working language to the system language.

[0309] For example, when the first workflow involves an application, the working language refers to the language used within that application. Figure 6As shown in (b), the application involved in the first workflow is a recording application, and the recording language is the working language. When the recording language is different from the system language, a prompt message 601 can be displayed. The prompt message 601 is used to inform the user that the recording language is different from the system language, and the user can choose to translate the recording language into the system language.

[0310] S1804, the electronic device sends at least one device information, and correspondingly, the server receives at least one device information.

[0311] For example, an electronic device may send at least one device information to a server; it may also send at least one device information and a first arrangement result to a server; or it may send at least one device information, a first arrangement result, and a first user instruction to a server. When an electronic device sends multiple types of information to a server, it may package the multiple types of information and send them to the server.

[0312] S1805, the server determines the second arrangement result based on at least one device information and the first arrangement result.

[0313] It should be noted that the server can optimize the first orchestration result based on at least one device information, and the resulting second orchestration result is the optimized orchestration result.

[0314] For example, the second arrangement result can guarantee the success rate of executing user instructions; it can also make the execution of user instructions more in line with the current state of the electronic device, thereby improving the execution effect of user instructions.

[0315] S1806, the server sends the second arrangement result, and the electronic device receives the second arrangement result accordingly.

[0316] S1807, the electronic device executes the first workflow corresponding to the second arrangement result.

[0317] It should be noted that the orchestration result is used to represent the workflow corresponding to the user command. The execution and optimization of the workflow corresponding to the orchestration result by the end device can improve the success rate of executing user commands and bring a better user experience to the execution process.

[0318] To reduce the interaction between electronic devices and servers, a set of frequently used and essential pre-defined user commands and corresponding pre-defined workflows can be configured in the electronic devices. When a user command can successfully match a pre-defined user command in the pre-defined user command set, the electronic device can execute a pre-defined workflow in the pre-defined workflow set; when a user command fails to match a pre-defined user command in the pre-defined user command set, the cloud-side large model needs to orchestrate and understand the user command.

[0319] For example, the electronic device includes a preset user instruction set and a preset workflow set, wherein the preset user instructions in the preset user instruction set correspond one-to-one with the preset workflows in the preset workflow set; before receiving the first orchestration result from the server, a first user instruction is obtained; when the first user instruction fails to match the preset user instructions in the preset user instruction set, the first user instruction is sent to the server.

[0320] It should be noted that the set of preset user instructions and the set of preset workflows are in one-to-one correspondence. The set of preset user instructions can include multiple high-frequency, essential preset user instructions. In this context, user instructions can be used to reflect user intent, and preset user instructions can be understood as preset user intents.

[0321] Therefore, if there is no preset user instruction in the preset user instruction set of the electronic device that can be successfully matched with the user instruction, the user instruction can be sent to the server, and the server will generate the arrangement result.

[0322] For example, sentence segmentation techniques can be used to determine whether the first user instruction matches a preset user instruction in a preset user instruction set.

[0323] Specifically, before the electronic device sends the first user instruction to the server, the electronic device determines at least one sub-instruction based on the first user instruction; determines at least one first vector based on the at least one sub-instruction; determines a preset user instruction vector set based on a preset user instruction set, the preset user instruction vector set including at least one second vector, the at least one first vector and the at least one second vector corresponding one-to-one; and determines whether the first user instruction matches the preset user instructions in the preset user instruction set based on the similarity between the at least one first vector and the at least one second vector.

[0324] It should be noted that, based on sentence segmentation technology, N sub-instructions are obtained (N is a positive integer greater than or equal to 1), and these N sub-instructions are converted into N first vectors. The preset user instruction set is also converted into a preset user instruction vector set. These N first vectors are then matched with the N second vectors in the preset user instruction vector set for similarity. A match is considered successful when the similarity is greater than a similarity threshold.

[0325] For example, the number of failed matches can be accumulated to optimize the set of preset user instructions and the set of preset workflows.

[0326] Specifically, the electronic device records the number of times the first type of user instruction fails to match with the preset user instructions in the preset user instruction set. The first user instruction belongs to the first type of user instruction. When the number of matching failures meets the preset conditions, a second preset user instruction is added to the preset user instruction set according to the first type of user instruction, and a second preset workflow is added to the preset workflow set. The second preset workflow corresponds to the second preset user instruction.

[0327] As can be seen, electronic devices can be preset to have a limit of 20 failed matches for the same type of user command within a certain period (e.g., 10 days). When the number of failed matches exceeds a threshold (15 times), it indicates that the preset condition is met, and such user commands can be updated to the preset user command set, along with the corresponding workflow set. By continuously updating the preset user command set and the preset workflow set, the accuracy of executing user commands can be improved while reducing interaction with the cloud-based large model, lowering communication latency, and enhancing the user experience.

[0328] For example, the electronic device acquires a second user instruction; when the second user instruction successfully matches a first preset user instruction, it determines a first preset workflow based on the first preset user instruction, wherein the first preset workflow belongs to a set of preset user workflows and the first preset user instruction belongs to a set of preset user instructions; and executes the first preset workflow.

[0329] It can be seen that when the second user instruction successfully matches the first preset user instruction in the preset user instruction set, the electronic device can execute the first preset workflow corresponding to the first preset user instruction.

[0330] To further improve the accuracy of electronic devices in executing user commands, the workflow can be optimized by combining at least one of the techniques of slot filling and substitution resolution.

[0331] For example, parameter information is determined according to the second user instruction; a second workflow is determined according to the first preset workflow and the parameter information; and the second workflow is executed.

[0332] It is understandable that this approach of configuring electronic devices with pre-set user instruction sets and pre-set workflows can be independent of the aforementioned approach of optimizing orchestration results, by reducing the interaction between electronic devices and servers.

[0333] For example, an electronic device includes a set of preset user instructions and a set of preset workflows. The preset user instructions in the set of preset user instructions correspond one-to-one with the preset workflows in the set of preset workflows. The electronic device obtains a first user instruction; determines whether the first user instruction matches the preset user instructions in the set of preset user instructions; when the first user instruction matches the first preset user instruction, determines a first preset workflow based on the first preset user instruction, wherein the first preset user instruction belongs to the set of preset user instructions and the first preset workflow belongs to the set of preset workflows; and executes the first preset workflow.

[0334] For example, when the first user instruction fails to match the preset user instructions in the preset user instruction set, the electronic device sends the first user instruction to the server; the number of times the first type of user instruction fails to match the preset user instructions in the preset user instruction set is recorded, and the first user instruction belongs to the first type of user instruction; when the number of matching failures meets the preset conditions, a second preset user instruction is added to the preset user instruction set according to the first type of user instruction, and a second preset workflow is added to the preset workflow set, and the second preset workflow corresponds to the second preset user instruction.

[0335] For example, the electronic device determines at least one sub-instruction based on a first user instruction; determines at least one third vector based on the at least one sub-instruction; determines a preset user instruction vector set based on a preset user instruction set, the preset user instruction vector set including at least one fourth vector, and at least one third vector corresponding one-to-one with at least one fourth vector; and determines whether the first user instruction matches a preset user instruction in the preset user instructions based on the similarity between at least one third vector and at least one fourth vector.

[0336] For example, the electronic device determines parameter information based on a first user instruction; determines a second workflow based on a first preset workflow and the parameter information; and executes the second workflow.

[0337] In addition, electronic devices can also optimize the orchestration results sent by the server.

[0338] For example, the electronic device determines a first workflow based on the second arrangement result and user information; and executes the first workflow.

[0339] It is evident that the workflow can be further optimized based on user information before the final workflow is executed.

[0340] For example, user information includes at least one of the following: user habits, user interests, and user relationships.

[0341] It should be noted that user information can be as follows: Figure 9The Personal Knowledge Graph (PKG) shown can be used to describe a semantic network of personal entities, attributes, and their relationships. By leveraging user information such as user habits, interests, and characteristics, the workflow corresponding to the orchestration results can be optimized, making the execution of user commands by electronic devices more personalized to the user.

[0342] Of course, it is understandable that electronic devices can also optimize the original orchestration results, that is, optimize the orchestration results that have not been optimized by the cloud-side big model through correlation information.

[0343] For example, an electronic device receives a first orchestration result from a server, the first orchestration result corresponding to a first user instruction; determines a first workflow based on the first orchestration result and user information; and executes the first workflow. The user information includes at least one of user habits, user interests, and user relationships.

[0344] It is evident that workflows can be optimized based on user information before electronic devices execute their processes, thereby enhancing the user experience.

[0345] Specifically, before determining the workflow based on the orchestration results and user information, the electronic device can determine whether there are any missing matches based on the orchestration results, that is, whether the workflow corresponding to the current orchestration result can be executed successfully. If there are missing matches, it means that the workflow cannot be executed successfully. Matching information can be determined through the orchestration results and user information, and the missing information can be supplemented to improve the execution effect of user instructions.

[0346] In the data processing method provided in this application, the electronic device prioritizes matching preset user instructions based on user commands. If a match is successful, the preset workflow can be executed directly; if a match fails, the server processes the user commands to obtain the orchestration result. The electronic device can determine the associated information by combining the device information relationship graph, and the server optimizes the orchestration result. Simultaneously, the electronic device can further optimize the orchestration result issued by the server by combining it with a personal knowledge graph (PKG), and then execute the final optimized workflow. This process improves the accuracy of executing user commands while minimizing communication and interaction with the server, making the execution process more in line with the user's personalized needs and enhancing the user experience.

[0347] The above describes a data processing method according to embodiments of this application. The following, in conjunction with... Figure 19 This application describes an apparatus according to embodiments thereof. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be referred to the foregoing method embodiments.

[0348] Figure 19This is a structural example diagram of a data processing apparatus 1900 provided in an embodiment of this application. The data processing apparatus 1900 may possess the functions of the electronic device (end-side device) or server (cloud-side) in the above method embodiments, and can be used to execute the steps performed by the functions of the sub-device (end-side device) or server (cloud-side) in the above method embodiments. This function can be implemented in hardware, or in software, or in software executed by hardware. The hardware or software includes one or more modules corresponding to the above functions.

[0349] In one possible implementation, the data processing apparatus 1900 includes a transceiver unit 1910 and a processing unit 1920. The transceiver unit 1910 and the processing unit 1920 are coupled to each other.

[0350] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the aforementioned method steps to implement the data processing method described in the above embodiments.

[0351] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the data processing method described in the above embodiment.

[0352] Furthermore, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. When the apparatus is a chip, the chip may include a connected processor and a memory. The memory stores computer execution instructions, and when the chip is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to perform the data processing methods described in the above-described method embodiments.

[0353] In one possible implementation, the chip implementing the data processing method provided in this application includes a transceiver module and a processing module. The transceiver module can be an input / output circuit or a communication interface; the processing module can be a processor, microprocessor, or integrated circuit integrated on the chip.

[0354] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0355] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0356] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0357] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or split into more modules, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0358] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0359] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0360] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0361] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method is applied to an electronic device, and the method includes: Receive a first arrangement result from the server, the first arrangement result corresponding to a first user instruction; Based on the first arrangement result, at least one piece of device information is determined from multiple pieces of device information; Send the at least one device information to the server; Receive a second orchestration result from the server, the second orchestration result being determined by the at least one device information and the first orchestration result; Execute the first workflow corresponding to the second arrangement result.

2. The method according to claim 1, characterized in that, The plurality of device information includes at least one of the following: the status information of the electronic device, the information of the application, or the relationship between the status information of the electronic device and the information of the application, wherein the application information includes the type of the application, and the status information of the electronic device includes language information and / or battery information.

3. The method according to claim 2, characterized in that, The multiple device information includes information about the application. The step of determining at least one device information from multiple device information based on the first arrangement result includes: When the first arrangement result is related to the first application, information of a first type of application related to the first application is determined from the plurality of device information, and the information of the first type of application belongs to the information of the application.

4. The method according to claim 3, characterized in that, The information for the first type of application is used to indicate that the number of applications belonging to the first type is N, where N is a positive integer greater than or equal to 2.

5. The method according to any one of claims 2 to 4, characterized in that, The multiple device information includes the power information. The step of determining at least one device information from multiple device information based on the first arrangement result includes: When the first arrangement result is related to the power information, the power information is determined from the plurality of device information.

6. The method according to claim 5, characterized in that, When the power information indicates that the power of the electronic device is below a power threshold, the first workflow includes displaying a first prompt message, which prompts the user to charge the electronic device.

7. The method according to any one of claims 2 to 6, characterized in that, The multiple device information includes the language information, which is used to represent the system language. The step of determining at least one device information from multiple device information based on the first arrangement result includes: When the first arrangement result is related to the language information, the language information is determined from the plurality of device information.

8. The method according to claim 7, characterized in that, When the working language corresponding to the first workflow is inconsistent with the system language, the first workflow includes displaying a second prompt message, which prompts the user to switch the working language to the system language.

9. The method according to any one of claims 1 to 8, characterized in that, Sending the at least one device information to the server includes: Send the at least one device information, the first orchestration result, and the first user instruction to the server.

10. The method according to any one of claims 1 to 9, characterized in that, The electronic device includes a set of preset user instructions and a set of preset workflows, wherein the preset user instructions in the set of preset user instructions correspond one-to-one with the preset workflows in the set of preset workflows. The method further includes, prior to receiving the first arrangement result from the server: Obtain the first user instruction; When the first user instruction fails to match the preset user instruction in the preset user instruction set, the first user instruction is sent to the server.

11. The method according to claim 10, characterized in that, Before sending the first user instruction to the server, the method further includes: At least one sub-instruction is determined based on the first user instruction; Determine at least one first vector according to the at least one sub-instruction; A preset user instruction vector set is determined based on the preset user instruction set, wherein the preset user instruction vector set includes at least one second vector, and the at least one first vector corresponds one-to-one with the at least one second vector; Based on the similarity between the at least one first vector and the at least one second vector, it is determined whether the first user instruction matches a preset user instruction in the preset user instruction set.

12. The method according to claim 10 or 11, characterized in that, The method further includes: Record the number of times the first type of user instruction fails to match the preset user instructions in the preset user instruction set, wherein the first user instruction belongs to the first type of user instruction; When the number of matching failures meets the preset conditions, a second preset user instruction is added to the preset user instruction set according to the first type of user instruction, and a second preset workflow is added to the preset workflow set. The second preset workflow corresponds to the second preset user instruction.

13. The method according to any one of claims 10 to 12, characterized in that, The method further includes: Obtain second user instructions; When the second user instruction and the first preset user instruction are successfully matched, a first preset workflow is determined according to the first preset user instruction. The first preset workflow belongs to the preset workflow set, and the first preset user instruction belongs to the preset user instruction set. Execute the first preset workflow.

14. The method according to claim 13, characterized in that, The execution of the first preset workflow includes: The parameter information is determined according to the second user instruction; The second workflow is determined based on the first preset workflow and the parameter information; Perform the second workflow.

15. The method according to any one of claims 1 to 14, characterized in that, The first workflow corresponding to the second arrangement result includes: The first workflow is determined based on the second arrangement result and user information, wherein the user information includes at least one of the following: User habits, user interests, and user relationships; Perform the first workflow.

16. A data processing method, characterized in that, The method is applied to an electronic device, and the method includes: Receive a first arrangement result from the server, the first arrangement result corresponding to a first user instruction; The first workflow is determined based on the first arrangement result and user information; Perform the first workflow.

17. The method according to claim 16, characterized in that, The user information includes at least one of the following: User habits, user interests, and user relationships.

18. A data processing method, characterized in that, The method is applied to an electronic device, which includes a preset user instruction set and a preset workflow set, wherein the preset user instructions in the preset user instruction set correspond one-to-one with the preset workflows in the preset workflow set, and the method further includes: Obtain the first user's instructions; Determine whether the first user instruction matches a preset user instruction in the preset user instruction set; When the first user instruction and the first preset user instruction are successfully matched, the first preset workflow is determined according to the first preset user instruction. The first preset user instruction belongs to the preset user instruction set, and the first preset workflow belongs to the preset workflow set. Execute the first preset workflow.

19. The method according to claim 18, characterized in that, The method further includes: When the first user instruction fails to match the preset user instruction in the preset user instruction set, the first user instruction is sent to the server. Record the number of times the first type of user instruction fails to match the preset user instructions in the preset user instruction set, wherein the first user instruction belongs to the first type of user instruction; When the number of matching failures meets the preset conditions, a second preset user instruction is added to the preset user instruction set according to the first type of user instruction, and a second preset workflow is added to the preset workflow set. The second preset workflow corresponds to the second preset user instruction.

20. The method according to claim 18 or 19, characterized in that, Determining whether the first user instruction matches a preset user instruction in the preset user instruction set includes: At least one sub-instruction is determined based on the first user instruction; Determine at least one third vector according to the at least one sub-instruction; A preset user instruction vector set is determined based on the preset user instruction set, wherein the preset user instruction vector includes at least one fourth vector, and the at least one third vector corresponds one-to-one with the at least one fourth vector; Based on the similarity between the at least one third vector and the at least one fourth vector, it is determined whether the first user instruction matches the preset user instruction in the preset user instructions.

21. The method according to any one of claims 18 to 20, characterized in that, The execution of the first preset workflow includes: The parameter information is determined according to the first user instruction; The second workflow is determined based on the first preset workflow and the parameter information; Perform the second workflow.

22. A data processing method, characterized in that, The method is applied to a server, and the method includes: The first arrangement result is determined according to the first user instruction; Send the first arrangement result to the electronic device; Receive at least one device information from the electronic device, the at least one device information being determined by the first arrangement result and multiple device information; A second arrangement result is determined based on the first arrangement result and the information of at least one device. The second arrangement result is sent to the electronic device.

23. The method according to claim 22, characterized in that, The plurality of device information includes at least one of the following: electronic device status information, application information, or the relationship between the electronic device status information and the application information, wherein the application information includes the application type, and the electronic device status information includes language information and / or battery information.

24. The method according to claim 22 or 23, characterized in that, Receiving at least one device information from the electronic device includes: Receive at least one device information, the first arrangement result, and the first user instruction from the electronic device.

25. An electronic device, characterized in that, include: One or more processors; One or more memory units; The one or more memories store one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method as claimed in any one of claims 1 to 15, or the method as claimed in claim 16 or 17, or the method as claimed in any one of claims 18 to 21.

26. A server, characterized in that, The server includes a unit for performing any one of claims 22 to 24.

27. A data processing apparatus, characterized in that, include: A processor coupled to a memory for storing a computer program, the processor for running the computer program such that the data processing apparatus performs the method as claimed in any one of claims 1 to 15, or performs the method as claimed in claims 16 or 17, or performs the method as claimed in any one of claims 18 to 21, or performs the method as claimed in any one of claims 22 to 24.

28. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a computer, causes the computer to perform the method as described in any one of claims 1 to 15, 16 or 17, 18 to 21, 22 to 24.

29. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 15, 16 or 17, 18 to 21, 22 to 24.

30. A chip, characterized in that, The chip includes a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method as described in any one of claims 1 to 15, 16 or 17, 18 to 21, 22 to 24.

Images