Operation guiding method, operation guiding device and electronic equipment

Abstract

The invention discloses an operation guiding method, an operation guiding device and electronic equipment, and belongs to the technical field of information processing. The method comprises the steps of predicting an operation intention of equipment control operation according to reference information under the condition that the equipment control operation is received; the reference information comprises equipment operation environment information, equipment operation information and a user portrait; outputting operation guide information corresponding to the operation intention, wherein the operation guide information comprises equipment response processing information used for prompting whether to execute equipment response processing corresponding to the operation intention; wherein the equipment operation environment information comprises at least one of the following items: equipment state information, equipment time information, foreground application information, background application information, notification information and media playing state information; the equipment operation information comprises operation characteristic information of the equipment control operation.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an operation guidance method, an operation guidance device and an electronic device. Background Art

[0002] When using a terminal, a user often needs to perform multiple or multiple rounds of operations in order to achieve the operation intention. For example, when a user needs to play a video, the operation steps involved may include unlocking the screen, searching for a video playback application (Application, APP), opening the APP, browsing history or hobbies, and playing the video.

[0003] In related technologies, voice assistants can be used to assist users in operations. However, in this process, in order to realize the operation intention, users also need to input multiple rounds of requests to the voice assistant. For example, in the process of using a voice assistant to write a complete weekly report, the user needs to perform the following operations through the voice assistant in sequence: wake up the assistant, input "write a weekly report", output a general reply, such as the general reply is "OK", after outputting the weekly report, if it does not meet expectations, it is also necessary to supplement personal occupation, supplement personal experience, modify the details, and finally copy the final weekly report content and export the data.

[0004] From the above examples, we can see that in the process of using voice assistants to assist users in operations, the input operations or dialogues often have multiple and multiple rounds of demands, that is, users generally have to input content multiple times to clearly express their questions and demands. At the same time, users have different understandings of vocabulary and expressions, and voice assistants cannot accurately predict users' operation intentions based on their voice input. It can be seen that in the user operation method in the related art, users need to interact multiple times before the terminal can know the user's operation intentions, and the operation efficiency is low. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an operation guidance method, an operation guidance device and an electronic device, which can predict the user's operation intention based on the device operating environment information, device operation information, and user portrait, and then implement operation guidance based on the predicted operation intention, thereby improving operation efficiency.

[0006] In a first aspect, an embodiment of the present application provides an operation guidance method, the method comprising:

[0007] When receiving a device control operation, predicting the operation intention of the device control operation according to reference information; the reference information includes device operating environment information, device operation information and user portrait;

[0008] Outputting operation guidance information corresponding to the operation intention, the operation guidance information including device response processing information for prompting whether to execute the device response processing corresponding to the operation intention;

[0009] Among them, the device operating environment information includes at least one of the following: device status information, device time information, foreground application information, background application information, notification information, and media playback status information; the device operation information includes operation feature information of the device control operation.

[0010] In a second aspect, an embodiment of the present application provides an operation guidance device, the device comprising:

[0011] A prediction module, configured to predict the operation intention of the device control operation based on reference information when receiving the device control operation; the reference information includes device operating environment information, device operation information and user portrait;

[0012] an output module, configured to output operation guidance information corresponding to the operation intention, wherein the operation guidance information includes device response processing information for prompting whether to execute the device response processing corresponding to the operation intention;

[0013] Among them, the device operating environment information includes at least one of the following: device status information, device time information, foreground application information, background application information, notification information, and media playback status information; the device operation information includes operation feature information of the device control operation.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0017] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions, and when the computer instructions are executed by a processor, the method described in the first aspect is implemented.

[0018] In the embodiment of the present application, when receiving the device control operation, the user's operation intention to perform the device control operation can be predicted based on the device operating environment information, device operation information and user portrait. The principle is: based on the fact that most of the functional operations performed by the user on the device have relevant periodicity and event logic, on the one hand, by mining the user portrait, the user's operation preferences can be recorded; on the other hand, based on the device operating environment information and device operation information, more comprehensive information can be obtained, such as information within and across APPs, all local files, and historical operations of users under different device operating environments; finally, using the above information as reference information to predict the operation intention of the device control operation, it is possible to achieve the prediction of the operation intention that matches the user's preferences and the current device-side information. After the operation intention is predicted, the operation guidance information corresponding to the operation intention is output, and the operation guidance information includes the device response processing information for prompting whether to execute the device response processing corresponding to the operation intention. In this way, through the user's positive feedback operation on the operation guidance information, the setting can be controlled according to the device response processing information in the operation guidance information, and the device response processing corresponding to the operation intention is directly executed, so that the user can be provided with relevant recommendations and guidance for directly reaching the operation intention based on the operation guidance information, thereby improving the operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of an operation guidance method provided by some embodiments of the present application;

[0020] Figure 2 is a schematic diagram of a process for constructing a user portrait in some embodiments of the present application;

[0021] Figure 3 is a schematic diagram of an interface of application scenario 1 in some embodiments of the present application;

[0022] Figure 4 is a schematic diagram of an interface of application scenario 2 in some embodiments of the present application;

[0023] Figure 5 yes Figure 3 A schematic diagram of the prompt information in the application scenario 1 is shown;

[0024] Figure 6 yes Figure 4 A schematic diagram of the prompt information in the second application scenario is shown;

[0025] Figure 7 is a flow chart of an operation guidance method provided by some embodiments of the present application;

[0026] Figure 8 is a flow chart of an operation guidance method provided by some embodiments of the present application;

[0027] Fig. 9 is a schematic diagram of the structure of an operation guidance device provided in some embodiments of the present application;

[0028] Fig.10 is a schematic diagram of the structure of an electronic device provided by some embodiments of the present application;

[0029] Fig.11 It is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0031] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "or" in the specification and claims represents at least one of the connected objects, such as including A or B, can be including only A but not including B, or, only including B but not including A, or including A and B.

[0032] It should be noted that the operation guidance method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, and vehicle-mounted electronic devices. In some embodiments of the present application, the operation guidance method is executed by an electronic device as an example to illustrate the operation guidance method provided in the embodiments of the present application.

[0033] In order to facilitate understanding of the operation guidance method provided in the embodiment of the present application, the following nouns or terms involved in the embodiment of the present application are also explained:

[0034] The terminal side refers to the device side that executes the operation guidance method provided in the embodiment of the present application, including but not limited to portable terminal devices such as mobile phones and computers, such as the terminal side large language model (LLM), which means a large language model deployed on the device side.

[0035] LLM refers to a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of language text. The large language model in the embodiment of the present application can adopt a Transformer-based text conversion large language model, such as Transformer-based bidirectional encoder representations (Bidirectional Encoder Representations from Transformers, BERT), or, other structural types of large language models can be adopted, such as the LLaVA-UHD model. The structural type of the large language model used in the embodiment of the present application is not specifically limited here.

[0036] Prompt is a control instruction of LLM. It is a piece of text or question input into LLM to guide LLM to generate specific answers or content.

[0037] Guidance is used to help users directly access common operations when operating electronic devices.

[0038] Recommendations are mainly based on user behavior and voice input, and are combined with the device operating environment and operation history to recommend or execute related functions.

[0039] The operation intention is used to indicate the ultimate purpose of a specific device control operation when the user performs the device control operation. For example, when an electronic device receives an unread message from a social application, if the user unlocks the screen, it can be inferred that the user's operation intention of unlocking the screen is to open the chat interface corresponding to the unread message, based on the fact that in the past, when there were unread messages, the user usually opened the chat interface corresponding to the unread message after unlocking the screen.

[0040] Reference information refers to information that needs to be referenced in the process of predicting the operation intention of the device control operation.

[0041] Device control operations refer to device control operations that trigger the prediction of operation intentions. For example, when a user turns on the screen, it is necessary to predict whether the user's operation intention of turning on the screen is to check unread messages, search for songs, or perform device control operations such as searching for songs. Device control operations can include operations of the user using the device's applications, such as the use of applications installed on the electronic device, or basic control operations of the user on the device, such as power-on control, power-off control, and lock screen control.

[0042] The operation purpose information indicates the operation purpose achieved by the user through controlling the device. For example, the user performs a series of device control operations such as turning on the screen, unlocking the screen, opening the notification bar, and clicking on the unread messages of friend A, so that the electronic device responds to and processes the series of operations and finally displays the conversation interface of friend A in the social application. The operation purpose information of the device control operation can be summarized as "viewing the unread messages of friend A", where the device control operation type is "view", and the object of the device control operation is "the unread messages of friend A".

[0043] The operation purpose information indicates the characteristic information of the user's control operation on the device, which is used to describe the characteristic information of the user's control operation on the device, and the operation purpose information can be a task-based short sentence in a natural language format. The operation purpose information includes the operation type and the operation object, wherein the operation type is specifically the type of device response processing triggered by the electronic device according to the user operation, such as the opening operation of the application, the sharing operation of the data, etc., and the operation object is specifically the application program, multimedia content and other objects targeted by the device response processing triggered by the electronic device according to the user operation, such as the object opened is a certain application program, and the multimedia content shared is a certain video, etc. For example: if a certain operation purpose information is "opening a social application", then the operation type of the operation purpose information is "open", and the operation object is "social application". For another example, "opening a social application", "viewing the messages of friend A in the social application" and other operation types belong to the "view" category, and "searching for music", "searching for goods" and other operation types belong to the "search" category, wherein "viewing" and "searching" are only examples and do not limit the specific types of operation types.

[0044] Historical device control operations refer to device control operations performed by users who operate electronic devices in historical time periods. For example, historical device control operations may include operations performed by users using applications on electronic devices in historical time periods, such as operations on applications installed on electronic devices, or basic control operations performed by users on devices in historical time periods, such as power-on control, power-off control, and screen lock control. The specific operation type and user operation method of the historical device control operation are similar to those of the aforementioned device control operations, and will not be described in detail here.

[0045] Information entities refer to entities in media information such as text and images, for example, they may include names of people, places, song titles, etc.

[0046] The operation guidance method provided in the embodiment of the present application can be applied to scenarios where a user performs device control operations on an electronic device, such as clicking, long pressing, sliding the touch screen, etc., to predict the user's next operation intention and provide operation guidance corresponding to the predicted operation intention.

[0047] Among them, a specific application scenario is: when an electronic device receives an unread communication message from a social friend A, when the user lights up the screen of the electronic device by pressing a power control or sliding the screen or other device control operations, it is predicted that the user's operation intention of lighting up the screen is to check the unread communication messages from social friend A, or to reply to the unread communication messages from social friend A, or to open a game application, or to open an audio or video application, etc. Based on the predicted operation intention, relevant recommendations and guidance for directly reaching the operation intention are provided.

[0048] Another specific application scenario is: when a user copies or selects a piece of text information or a picture, such as copying the text information "Singer B's new song "Dao Xiang" is very good", at this time, based on the text information, it can be predicted that the user's intention of copying or selecting the above text information is to search for singer B's song "Dao Xiang", share singer B's song "Dao Xiang" to the social circle, follow singer B, etc. Based on the predicted operational intention, relevant recommendations and guidance are provided to directly reach the operational intention.

[0049] The user's operation intention on the device side is often related to the device side events and time, that is, the device side events, time and user information will affect the user's operation intention, and the user's operation intention is also related to the device operating environment and the user's operation on the device side. For example: when a user sees a favorite clothing on an electronic device, he will search on a shopping website; when a user sees a favorite article on an electronic device, he will try to get the full text; when a user sees a favorite picture on an electronic device, he will search for similar photos or write a copy.

[0050] In the embodiment of the present application, in the process of predicting the user's operation intention, the operation feature information of the device control operation that triggers the prediction of the operation intention, the device operating environment when the device control operation is received, the user portrait and other information are comprehensively considered, which can more accurately and personally predict the user's next operation intention, and provide operation guidance corresponding to the predicted operation intention to improve the operation efficiency of the device.

[0051] The operation guidance method, operation guidance device and electronic device provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0052] See also Figure 1 The operation guidance method provided in the embodiment of the present application includes the following steps:

[0053] Step 101: When a device control operation is received, predict the operation intention of the device control operation based on reference information; the reference information includes device operating environment information, device operation information and user portrait.

[0054] Among them, the device operating environment information includes at least one of the following: device status information, device time information, foreground application information, background application information, notification information, and media playback status information; the device operation information includes operation feature information of the device control operation.

[0055] For example: when an electronic device receives an unread communication message from social friend A, and the user turns on the screen of the electronic device by pressing a power control or sliding the screen, in an application scenario where the user's intention to turn on the screen is predicted, the reference information may include reference information such as the user's previous operating habits, the presence of unread communication messages from social friend A on the current electronic device, whether the foreground application includes a social application, whether the background application includes a social application, and the reception time of the unread communication message from social friend A.

[0056] As another example, when a user copies or selects a piece of text information or a picture, such as copying the text information "Singer B's new song Daoxiang is very good", the reference information is predicted based on the text information to predict the user's intention to copy or select the text information. The reference information may include the user's previous operating habits, the existence of copied or selected text information on the current electronic device, the information entity contained in the copied or selected text information, whether the foreground application includes audio or social applications, whether the background application includes audio or social applications, and other reference information.

[0057] In some embodiments of the present application, the above-mentioned device control operation may be an input for triggering prediction of operation intention, such as a user's click touch input, long press touch input, touch input for opening a specified APP, pressing a power control, etc. on the electronic device.

[0058] In some embodiments of the present application, the above-mentioned device control operation specifically refers to user input, such as unlock input, click, long press, circle search and other user inputs, and the type of device control operation is not specifically limited here. The operation characteristic information of the device control operation may include the operation type of the device control operation, for example: the operation information of the device control operation may indicate that the device control operation includes at least one operation of any operation type such as click, long press, slide, etc.; in addition, the operation characteristic information may also include the operation object of the device control operation, such as whether the click is the screen, the application icon or the application interface, etc.

[0059] In some embodiments of the present application, the device status information includes network status information of the electronic device, such as whether it is connected to WiFi, whether it is connected to Bluetooth, etc., and may also include time information of the device, such as obtaining the current time of the device when receiving a device control operation.

[0060] In some embodiments of the present application, the foreground application information is used to indicate an application program that is in a foreground running state on the electronic device, and the background application information is used to indicate an application program that is in a background running state on the electronic device.

[0061] In some embodiments of the present application, the notification information is used to indicate relevant information of unread messages on the electronic device, such as the application corresponding to the unread message, the reception time of the unread message, the message content of the unread message, the information entity of the unread message, etc.

[0062] In some implementations of the present application, the media playback status information is used to indicate the playback status of a media file such as audio or video by a media playback application, such as play, pause, etc.

[0063] In some embodiments of the present application, in addition to the above-mentioned device status information, current time information, foreground application information, background application information, notification bar information, and media playback status information, the device operating environment information may also include other information related to the device-side operating environment, such as global positioning system (GPS) positioning information, user input before receiving device control operations, etc.

[0064] In some embodiments of the present application, the user portrait may include information that can reflect the user's operation preference. For example, when there are unread messages, the user often opens the notification bar and clicks on the unread message to open the corresponding application to read the unread message immediately after unlocking the screen. When there are unread messages and a device control operation for unlocking the screen is received, it can be predicted that the user's intention to unlock the screen is to open the corresponding application to read the unread message.

[0065] In some implementations of the present application, the content of the user portrait may also include information such as the user's age, gender, occupation, group, commonly used apps, hobbies, etc. For example, a user portrait is: male, literary youth, music, travel, sports.

[0066] In some embodiments of the present application, the user portrait can record the user's operating behavior, preferences, etc. on the electronic device; based on the device operating environment information and device-side operation information, more comprehensive device-side information can be obtained, such as information within and across APPs, all local files, and the user's operation history in different device-side environments.

[0067] It is worth noting that compared with the related technologies, in which product recommendations, song recommendations, and multimedia content recommendations are made within a single APP, the products, songs, or articles that users like are recommended mainly based on the user's usage history of the APP, and the technical solutions that cannot use cross-APP information for recommendation are used. This application can predict operation intentions based on device-side information across APPs. Based on the fact that most of the user's operation behaviors on electronic devices have related periodicity and event logic, the operation intentions of device control operations are predicted based on the above reference information, which can achieve operation intention predictions that match user preferences, device control operations, and the current device operating environment.

[0068] In some embodiments of the present application, the method for determining the operational intent of the device control operation based on reference information may be to pre-train a large language model to input the reference information into the trained large language model, so as to utilize the large language model's understanding and analysis of the reference information to infer the operational intent of the device control operation.

[0069] In other embodiments, the method for determining the operational intent of the device control operation based on reference information may be to pre-store sample feature information corresponding to various historical device control operations, and extract feature information of the reference information, and match the feature information of the reference information with the sample feature information corresponding to the historical device control operations, and use the known operational purpose of the historical device control operation corresponding to the sample feature information that matches the feature information of the reference information as the operational intent of the device control operation.

[0070] Step 102: output operation guidance information corresponding to the operation intention, wherein the operation guidance information includes device response processing information for prompting whether to execute the device response processing corresponding to the operation intention.

[0071] In some embodiments of the present application, the operation guidance information corresponding to the operation intention may be outputted by at least one output method such as display and voice broadcast. For example, assuming that the predicted operation intention is to open a social application, the operation guidance information "Do you want to open the social application?" may be displayed on the display screen, or the operation guidance information "Do you want to open the social application?" may be voice broadcasted.

[0072] It is worth mentioning that after the operation guidance information is output, since the operation guidance information includes device response processing information for prompting whether to execute the device response processing corresponding to the operation intention, when the user performs a feedback operation on the operation guidance information, it can be determined based on the type of feedback operation whether to execute the device response processing corresponding to the operation intention in the operation guidance information.

[0073] For example: when an electronic device receives an unread communication message from a social friend A, and the user turns on the screen of the electronic device by pressing a power control or sliding the screen, in an application scenario where the user's intention to turn on the screen is predicted, it is assumed that it is predicted that the user's intention to turn on the screen of the electronic device is to open a social application to view the notification message from friend A, and an operation guidance message of "Do you want to open the social application to view the notification message from friend A?" is displayed on the display screen. If the user confirms the operation guidance message, such as clicking or long pressing the operation guidance message, the device response processing of opening the social application and displaying the conversation interface of friend A is performed.

[0074] As another example: in an application scenario where a user copies or selects a piece of text information or a picture, such as copying the text information "Singer B's new song "Dao Xiang" is very good", the user's intention to copy or select the text information is predicted based on the text information. Assuming that it is predicted that the user's intention to copy or select the text information is to search for singer B's song "Dao Xiang", an operation guidance message "Do you want to search for singer B's song "Dao Xiang"? " is displayed on the display screen. If the user confirms the operation guidance message, such as clicking or long pressing the operation guidance message, the device response processing of opening the audio application and entering singer B's song "Dao Xiang" in the search box of the audio application to search for the song is executed.

[0075] In this way, through the user's feedback operation on the operation guidance information, the setting can be controlled according to the device response processing information in the operation guidance information, and the device response processing corresponding to the operation intention can be directly executed without the user having to unlock the screen, search for application icons, click on application icons, etc., so that based on the operation guidance information, the user can be provided with relevant recommendations and guidance directly to the operation intention, thereby improving the operating efficiency of the electronic device.

[0076] As an optional implementation, the method further includes:

[0077] Obtain system log information and system status information of the device;

[0078] The system log information and the system status information are input into a first large language model, and the system log information and the system status information are structured through the first large language model to output device operating environment information and device operation information in a natural language format.

[0079] In some embodiments, the first large language model is any large language model capable of semantic understanding, such as BERT, RoBERTa, etc. Through the semantic understanding of system log information and system status information by the first large language model, the system log information and system status information can be converted into natural language information with a fixed structure.

[0080] In some embodiments of the present application, system log information and system status information of the device can be obtained in response to the device control operation, for example: system log information and system status information when the device control operation is received, or system log information within a preset time period before the device control operation can be obtained, such as system log information for the past 10 seconds or 1 minute, or system log information of a specified type can be obtained, such as system log information for recording foreground applications, background applications, notification bar information, operation information of the device control operation, and other information.

[0081] In some implementations of the present application, the system log information may be a process identifier (PID) system log, such as: ```07-15 19:54:04.638 32084 32147D_V_UsageStatsManager:queryEvents,sEmptyResults:android.app.usage.UsageEvents@4303e72```.

[0082] In some implementations of the present application, device status information such as current time and network connection status can be directly extracted from system status information, while information related to interaction or behavior needs to be extracted based on system logs.

[0083] In some embodiments of the present application, the first large language model is used to perform structured processing on the system log information and the system status information, and the first large language model can be used to convert the system log information and the system status information into natural language information with a fixed structure.

[0084] In some embodiments of the present application, a prompt word (prompt) may be generated to prompt the first language model to convert system log information and system status information into structured natural language information, and the format of the prompt may be [role] + [example (optional)] + [requirement]. In this way, based on the prompt, the first language model may be controlled to convert system log information and system status information into natural language information with a fixed structure.

[0085] For example, the [Role] in the prompt is: You are now a log understanding expert. Now you will understand and extract the device log information and convert it into a structured language.

[0086] The requirements in the prompt include:

[0087] 1. Pay attention only to key system logs and device status information;

[0088] 2. System logs and device status information that have little impact on behavior may not be processed;

[0089] 3. The structured language needs to be converted into json format, such as {front-end application: P; back-end application: Q; current operation: W; notification information: Z; ...};

[0090] The input of the first language model is system log information and system status information, and the output is structured language.

[0091] For another example, the following Table 1 shows the system logs corresponding to two operation examples and the structured data generated based on the system logs:

[0092] Table 1

[0093]

[0094]

[0095] In some embodiments of the present application, the first large language model can be used to process system logs and system status information into device operating environment information and device operation information in a natural language format. In this way, the program language and messy system status information can be organized into information in a natural language format, so that users or other large language models can read or understand the information in a natural language format.

[0096] It is worth mentioning that in some embodiments, when the second largest language model is used to predict the operation intention of the device control operation according to the reference information, the first largest language model and the second largest language model are different large language models. In this way, after the system log information and the system status information are processed into the device operation environment information and the device operation information in the natural language format by using the information understanding and generation ability of the first largest language model, the second largest language model is used to read or understand the device operation environment information and the device operation information in the natural language format and the user portrait to predict the operation intention of the device control operation, the performance requirements for the second largest language model can be reduced, so that the second largest language model can be applied to various device ends. However, since it is necessary to first use the first largest language model to process the system log information and the system status information into the device operation environment information and the device operation information in the natural language format, additional space storage is required to store the device operation environment information and the device operation information in the natural language format. At this time, the device operation environment information and the device operation information in the reference information specifically refer to the device operation environment information and the device operation information in the natural language format.

[0097] In some embodiments, the second largest language model is any large language model capable of semantic understanding, such as BERT, RoBERTa, etc. Through the semantic understanding of the reference information by the second largest language model, the operational intention of the device control operation can be predicted.

[0098] Of course, in other embodiments of the present application, when the second largest language model is used to predict the operational intention of the device control operation based on the reference information, the first largest language model and the second largest language model are the same large language model. In this way, the system log information and the system status information can be directly used as part of the reference information, so that the system log information, the system status information and the user portrait can be directly input into the second largest language model, so that the second largest language model can be used to predict the operational intention of the device control operation based on the system log information, the system status information and the user portrait. At this time, the second largest language model is required to have the reasoning function based on the system log, which increases the effect and performance requirements of the second largest language model. In addition, the model parameters of the second largest language model are strongly related to the system log on the device side. If the device is changed, the relevant parameters of the second largest language model may not be universal.

[0099] As an optional implementation, the device operation information further includes copied or selected information entities;

[0100] The predicting, based on the reference information, the operation intention of the device control operation includes:

[0101] An operation intention of the device control operation matching the information entity is predicted according to the reference information.

[0102] The copied or selected information entity may include information entities in at least one multimedia content item among text, link, and picture, such as information entities such as names of people, places, and songs located in at least one of the text, link, and picture.

[0103] In some embodiments of the present application, the extraction of plain text named entity recognition (NER) can be extracted using Transformer-based bidirectional encoder representations (Bidirectional Encoder Representations from Transformers, BERT).

[0104] In some embodiments of the present application, information entities in images may be extracted using a multimodal image understanding model, such as LLaVA-UHD or other models.

[0105] When there is multimedia content in a copied or selected state, the operational intention of the device control operation is likely to tend to share, edit, search, and other processes on the information entity in the copied or selected state. Based on this principle, in this embodiment, the information entity that has been copied or selected when the device control operation is received is also obtained to infer the operational intention of the device control operation based on the information entity, so that the predicted operational intention is related to the information entity. For example: assuming that the copied text information includes the information entity of song G, it can be predicted that the operational intention of copying the text information may be an operation related to song G, such as searching for song G, playing song G, sharing song G, etc. In this way, the predicted operational intention is related to the information entity in the copied text information, which can improve the matching degree between the inferred operational intention and the information entity in the content when there is content in a copied or selected state.

[0106] As an optional implementation, the user portrait includes historical operation information;

[0107] The historical operation information includes an operation type of at least one historical device control operation within a historical time period and an operation execution count and a weight coefficient of the historical device control operation of each operation type.

[0108] It should be noted that, for historical device operations, the operation purpose ultimately achieved is determined by the user's actual operation, and the operation purpose is known.

[0109] In some embodiments of the present application, the operation type of at least one historical device control operation may include at least one pre-set operation type label, such as: locking the screen to view social application notifications, reading friend A's notifications, searching for music, sharing music, searching for singer B's music, etc.

[0110] In some implementations of the present application, the operation type may indicate the operation type of the operation purpose actually occurring in the corresponding historical device control operation, such as a viewing operation purpose or a shopping operation purpose.

[0111] In some embodiments of the present application, the number of operation executions of a historical device control operation of an operation type indicates the number of times a user has performed the historical device control operation to achieve a certain type of operation purpose. The greater the number of operation executions, the higher the preference of the user to perform the historical device control operation to achieve this type of operation purpose.

[0112] In some embodiments of the present application, the weight coefficient of each operation type of historical device control operation may be positively correlated with the number of operation executions of the historical device control operation of the operation type, or may be determined based on other user portrait information such as the user's gender, age, hobbies, etc. For example, for young women, the weight coefficient of the operation purpose of shopping is relatively large; for music lovers, the weight coefficient of the operation purpose of audio is relatively large.

[0113] By classifying the operation purposes of historical device control operations into different operation types and recording the operation execution times and weight coefficients of historical device control operations of each operation type, the bias of users in performing historical device control operations of various types of operation purposes can be reflected based on historical operation information.

[0114] For example, if the number of operation executions of historical device control operations of the social application class is greater than the number of operation executions of historical device control operations of the audio class, it can be reflected that the user is more inclined to execute device control operations of the social application class.

[0115] In this embodiment, the historical operation information can reflect the regularity of the operation purpose achieved by the user's historical device control operation, the frequency of historical device control operations for various types of operation purposes, the preference degree of historical device control operations for various types of operation purposes, etc. For example, when the historical operation information reflects that there are unread messages of social applications, the frequency of unlocking the screen, opening social applications, and checking unread messages of social applications after the user turns on the screen is high, then when predicting the user's operation intention of turning on the screen in the scenario where there are unread messages of social applications, it can be predicted that the user is likely to want to check the unread messages of social applications. In this way, when the operation intention of the device control operation is inferred based on the historical operation information, the matching degree between the operation intention and the operation purpose of the historical device control operation can be improved, and the personalization degree of the operation intention prediction can be improved.

[0116] In some embodiments of the present application, before predicting the operation intention of the device control operation according to the reference information, the method further includes:

[0117] In the detection time period, when a device control operation or a device response process is detected, device operation information is collected, wherein the device operation information includes at least one of the following: device operation data, application jump data, application usage data, copied or selected information entities;

[0118] Performing dimensionality reduction processing on the device operation information to obtain operation purpose information corresponding to the device control operation or device response processing, wherein the operation purpose information includes an operation type and an operation object;

[0119] Statistics are collected on all operation purpose information collected during the detection time period to obtain historical operation information, and the user portrait includes the historical operation information.

[0120] In some embodiments of the present application, dimensionality reduction processing of the device operation information refers to deleting information irrelevant to the operation intention in the complex device operation information, and retaining information related to the operation intention, so as to obtain more concise operation purpose information, such as summarizing the device operation information into an execution short sentence, such as checking the notification message of social application friend A, or sharing the song "Rice Fragrance" by singer B. For example: the device operation information can be input into a large language model, so as to use the large language model to understand and summarize the device operation information, and obtain more concise operation purpose information. For another example: a neural network model can be used to extract feature information related to the operation intention in the device operation information, and determine the operation purpose information based on the feature information.

[0121] In some embodiments of the present application, the collection of device operation information is triggered based on device control operations or device response processing. In this way, the device control operation or device response processing can be detected in real time within the detection time period, and the device operation information is collected once each time a device control operation or device response processing is detected.

[0122] In some embodiments of the present application, the device control operation or device response processing that triggers the collection of device operation information can be set in advance. For example, the device control operation may include clicking, long pressing, sliding, etc., and the device response processing may include the response processing of opening a preset application, the response processing of receiving an unread notification message, the response processing of waking up the display screen, etc.

[0123] In other embodiments, the device control operation or device response process that triggers the collection of device operation information may be a device control operation or device response process mined based on a large language model. For example, a large language model is used to mine potential device control operations or device response processes in device operation environment information, device operation information, and user portraits.

[0124] In some embodiments of the present application, the device operation data may include data related to device control behaviors such as long pressing, selecting, copying, sharing, editing, and operating performed by a user on the electronic device.

[0125] Among them, sharing-type device control behaviors can be specifically divided into categories such as sharing operations such as text, images, links, music, and videos; operation-type device control behaviors can include image search, product search, optical character recognition (OCR), translation, and other device control behaviors used to implement specific functions and algorithm capabilities.

[0126] In some embodiments of the present application, application jump data may include jumps between different types of applications, such as an image application jumping to a shopping application. Application jump data may also include jumps from entities to applications, for example, when copying the text content of an information entity containing a song, jumping to an audio playback application for music retrieval.

[0127] In some implementations of the present application, application usage data may include the installation status, usage time period, click rate, time period in the foreground, common location tags, etc. of various types of applications.

[0128] In some embodiments of the present application, the device operation information is subjected to dimensionality reduction processing to obtain the operation purpose information corresponding to the device control operation or device response processing, which may be an abstraction processing of the device operation information to summarize the operation purpose information corresponding to the device operation information, and the operation purpose information may be a task-type short sentence, in which case the operation purpose information includes an operation type and an operation object. For example, if the operation purpose information is: open a social application, the operation type is "open", and the operation object is "social application".

[0129] In some embodiments of the present application, all operation purpose information collected during the detection time period is counted to obtain historical operation information. The operation purpose information may be classified, and each type of operation purpose information is clustered to count the number of operation executions and weight coefficients of the device control operation of each operation type collected during the detection time period.

[0130] For example: Figure 2 As shown, the construction of user portraits can include the following steps:

[0131] Step 201: triggering based on device control operation or device response processing;

[0132] Step 202: extract the selected or copied information entity;

[0133] Step 203: Perform dimensionality reduction processing on the device operation information to obtain operation purpose information of the device control operation or the device response processing;

[0134] Step 204: Count the operation purpose information to obtain historical operation information.

[0135] In step 201, the collection of device operation information can be triggered based on a preset device control operation or device response processing, for example: when the user unlocks the screen, the device operation information is collected, wherein the device operation information can reflect the relevant device control operations and device response processing performed after the user unlocks the screen.

[0136] In step 202, when it comes to selected or copied content, it is necessary to extract information entities in the selected or copied content. For example, the extraction of plain text named entity recognition (NER) can be done using a Transformer-based bidirectional encoder representation (BERT); pictures can use a multi-modal image understanding model, such as LLaVA-UHD and other models.

[0137] In step 203, the operation purpose information corresponding to the preset input or the preset device-side event is determined according to the device operation information. In some embodiments of the present application, this step may be to summarize the device operation information, so as to reduce the device operation information collected based on each device control operation or device response processing trigger to the operation purpose information of the device control operation or device response processing, and finally obtain the operation purpose information in a natural language format.

[0138] This implementation can trigger the collection of device operation information based on device control operations or device response processing, so that the collected device operation information can reflect the device operation status of the device when receiving the device control operation or device response processing, as well as the subsequent response processing to the device control operation or device response processing. In this way, the operation purpose information obtained by downgrading the device operation information can reflect the operation type and operation object triggered by the device control operation or device response processing; finally, combined with the statistical results of all operation purpose information collected within the detection time period, the historical operation information obtained can reflect the user's operation habits, operation preferences, etc.

[0139] In some embodiments of the present application, similar to the reference information, the device operation information may include information obtained by structured processing of system logs and system status information, and the device operation information is abstracted and extracted into operation purpose information based on the third language model.

[0140] For example, the prompt for entering the third largest language model is as follows:

[0141] [Role]: You are now an action understanding expert. The user's behavior is interpreted as an execution phrase, which requires the action type and the object of the action.

[0142] [Requirements]: 1. Only retain the most core content; 2. The output word count should not exceed 50 words.

[0143] The corresponding relationship between the equipment operation information and the operation purpose information is shown in Table 2 below:

[0144] Table 2

[0145]

[0146]

[0147] In the above step 204, a large amount of operation purpose information can be clustered, such as using the BERT clustering method, to cluster the same type of operation purposes together and count them to calculate the weight coefficient of each type of operation purpose. For example: for operation purposes such as viewing, copying, sharing, editing, and searching, the number of executions and weight coefficients of the corresponding device control operations are counted respectively.

[0148] At this point, the complete user profile may include the following information in Table 3:

[0149] Table 3

[0150]

[0151] Among them, the operation purpose information in Table 3 above can be classified into the following three categories:

[0152] 1. View. The specific purpose of this type of view operation is to view social application notifications on the lock screen and read A's friend notifications;

[0153] 2. Search: the purpose of this search type is to search for music;

[0154] 3. Sharing: The specific purpose of this type of sharing operation is to share music.

[0155] In some implementations of the present application, the following formula may be used to calculate the weight coefficients of various operation purpose information:

[0156] w=

[0157] (Total number of single-category operation purpose information ÷ Total number of operation purpose information + 1) ×

[0158] (Number of operation purpose information A ÷ Total number of single type of operation purpose information);

[0159] Wherein, w represents the weight coefficient of the operation purpose information A, and the single type of operation purpose information represents the total number of operation purpose information of the operation type including the operation purpose information A.

[0160] For example, for the operation purpose information shown in Table 3 above, the weight coefficient for viewing social application notifications on the lock screen = (30+20) ÷ (30+20+50+30) + 1) × (20 ÷ (20+30).

[0161] It is worth mentioning that before using the second largest language model to predict the operational intention of the device control operation, the initial large language model needs to be trained based on the training data to obtain the second largest language model that can predict the operational intention of the device control operation based on the reference information.

[0162] In some embodiments of the present application, the training data may include: device operating environment information, device operation information, user portraits, and operation purpose information, wherein the device operation information may include the operation information input as the first step in the user operation performed to implement the operation purpose information, and the operation purpose information may be understood as information about the operation purpose finally achieved. During the training process, the device operating environment information, device operation information, and user portraits are input into the initial large language model, and the parameters of the initial large language model are adjusted with the purpose of outputting information as the operation purpose information to obtain a second large language model, wherein the user portraits may include pre-configured portrait information of various types of users.

[0163] Among them, the device operating environment information, device operation information and user portrait in the training data can be the same type of information as the device operating environment information, device operation information and user portrait in the reference information.

[0164] Optionally, the device operation information may include a selected or copied information entity.

[0165] In some embodiments of the present application, the operation purpose information may be a task-based short sentence in a natural language format, such as opening a social application to read messages, opening a music application to search for songs by singer B, and then playing and sharing them.

[0166] For example: As shown in Table 4 below, they are Figure 3 and Figure 4 Two complete training data corresponding to the application scenarios shown:

[0167] Table 4

[0168]

[0169]

[0170]

[0171] In some embodiments of the present application, the use of the second largest language model to predict the operation intention of the device control operation according to the reference information includes:

[0172] Generate operation intention prediction prompt words based on reference information;

[0173] Inputting the operation intention prediction prompt word into a second language model, and outputting the operation intention of the device control operation through the second language model;

[0174] The operation intention prediction prompt word includes at least one of the following: the reference information, the first indication information, and the second indication information;

[0175] The first indication information is used to indicate that the task performed by the second large language model is to predict the operation intention based on the reference information;

[0176] The second indication information is used to indicate the prediction task requirements of the second largest language model, wherein the prediction task requirements include at least one of the following: the operation intention predicted by the second largest language model matches the user portrait; the device operating environment information is used as a variable for the second largest language model to predict the operation intention; the device operation information is used as a variable for the second largest language model to predict the operation intention; the prediction result output by the second largest language model is a task-type short sentence, and the task-type short sentence includes an operation type and an operation object.

[0177] In some embodiments of the present application, the operation intention prediction prompt word may be a Prompt input into the second largest language model. The first indication information may be the [Role] information in the Prompt; and the second indication information may be the [Requirement] information in the Prompt.

[0178] For example, the training process for the second largest language model can be based on the training and reasoning prompt implementation, which includes the following information:

[0179] [Role]: You are a user behavior prediction master. Please predict the user's behavior based on the user information and the current device information.

[0180] Requirements, including:

[0181] Requirement 1: The predicted behavior needs to be consistent with the user profile and combined with the current device operating environment information and device operation information as much as possible;

[0182] Requirement 2: For information entities in the selected or copied state, the most appropriate operation intention needs to be given according to the information entity;

[0183] Requirement 3: The output information should be in the form of task-based short sentences, such as [Task 1, Task 2, Task 3];

[0184]

Input

[0185]

Output

[0186] For example, when an electronic device receives an unread communication message from social friend A, and the user turns on the screen of the electronic device by pressing a power control or sliding the screen, in an application scenario where the user's intention to turn on the screen is predicted, the inference Prompt input into the second largest language model includes the following information:

[0187] [Role]: You are a user behavior prediction master. Please predict the user's behavior based on the reference information;

[0188] Requirements, including:

[0189] Requirement 1: The predicted behavior needs to be consistent with the user profile and combined with the current device operating environment information and device operation information as much as possible;

[0190] Requirement 3: The output information should be in the form of task-based short sentences, such as [Task 1, Task 2, Task 3];

[0191]

Input

[0192] [Output]: Operation intention: Open the social application to view the notification message from friend A.

[0193] For another example, when a user copies or selects a piece of text information or a picture, such as copying the text information "Singer B's new song Daoxiang is very good", based on the text information, the user's intention to copy or select the above text information is predicted, and the inference prompt input to the second largest language model includes the following information:

[0194] [Role]: You are a user behavior prediction master. Please predict the user's behavior based on the reference information;

[0195] Requirements, including:

[0196] Requirement 1: The predicted behavior needs to be consistent with the user profile and combined with the current device operating environment information and device operation information as much as possible;

[0197] Requirement 2: For the information entities "B" and "Dao Xiang" in the selected or copied state, the most appropriate operation intention needs to be given according to the information entity;

[0198] Requirement 3: The output information should be in the form of task-based short sentences, such as [Task 1, Task 2, Task 3];

[0199]

Input

[0200] [Output]: Operation intention: Search for singer B’s song “Dao Xiang”.

[0201] It should be noted that the above requirement 1 is used to limit the large language model to predict the operational intention of device control operations based on user portraits, and the predicted behavior cannot be unrelated to the user portrait. At the same time, the predicted behavior needs to be combined with the current device operating environment information and device operation information.

[0202] The above requirement 2 is used to indicate that when there is currently a selected or copied information entity, the predicted behavior needs to refer to the selected or copied information entity so that the predicted operation intention information is related to the information entity.

[0203] The above requirement 3 is used to limit the relevant tasks corresponding to the output results of the large language model to task-type short sentences that can be executed by an intelligent assistant or a voice assistant.

[0204] For example: Figure 3 In the application scenario one shown, when an unread notification message 31 from friend A is received, in order to view the messages of friend A in the social application when the screen is locked, the user in the related technology needs to perform the following operations: wake up the screen - unlock the screen - jump to the social application. In the embodiment of the present application, when only the screen is woken up, it is possible to predict the user's operation intention of viewing the messages of friend A in the social application based on the user portrait, the current device operating environment information and the device operation information, thereby providing relevant recommendations and guidance for directly reaching the operation intention and improving operation efficiency.

[0205] In this implementation, the prediction behavior of the second largest language model can be constrained based on the operation intention prediction prompt word, so as to achieve the purpose of using the second largest language model to predict the operation intention of the device control operation according to the reference information.

[0206] In some embodiments of the present application, the device control operation that triggers the prediction of the operation intention and the device control operation that triggers the collection of device operation information can be the same type of operation. For example, after collecting the device operation information based on the triggering of certain device control operations and obtaining the operation purpose information of these device control operations based on the dimensionality reduction processing of the device operation information, when a user operation similar to these device control operations is detected, the prediction process of the operation intention is triggered.

[0207] For example: For Figure 3 In the application scenario shown in FIG. 1 , when the user wakes up the screen, it is predicted that the operation intention of waking up the screen is "locking the screen to view messages from friend A in the social application"; Figure 4 In the application scenario shown, when the user performs the operation of selecting the text "Singer B's new song Daoxiang is good" 41, it is predicted that the operation intention for selecting the text is "search for and share singer B's Daoxiang to social applications."

[0208] As an optional implementation manner, after outputting the operation guidance information corresponding to the operation intention, the method further includes:

[0209] receiving a feedback operation on the operation guidance information;

[0210] The device response process is executed according to the device response process information.

[0211] In some embodiments of the present application, since the operation guidance information includes device response processing information for prompting whether to execute the device response processing corresponding to the operation intention, the device response processing is executed according to the device response processing information, so as to realize the operation intention. Figure 1 to achieve the purpose of the operation.

[0212] In some embodiments of the present application, the operation guidance information may be displayed on a touch screen. In this case, the feedback operation on the operation guidance information may be a touch input such as clicking or long pressing the operation guidance information.

[0213] In some embodiments of the present application, the operation guidance information may be associated with the execution interface of the above-mentioned device response processing, so that when the user clicks or long presses the operation guidance information, the device response processing may be directly executed.

[0214] In other embodiments of the present application, when a feedback operation on the operation guidance information is received, the interface of the intelligent assistant or the voice assistant can be called to utilize the intelligent assistant or the voice assistant to perform device response processing corresponding to the operation intention.

[0215] In other embodiments of the present application, the operation guidance information can be outputted by voice broadcast or the like. In this case, the feedback operation on the operation guidance information can be a confirmation operation on the broadcasted operation guidance information, such as clicking a confirmation control, or inputting voice confirmation information such as "confirm" or "OK" through voice interaction. In this way, after receiving the feedback operation on the operation guidance information, the device response processing corresponding to the operation intention can be directly executed.

[0216] For example: Figure 5 As shown, if the prediction is Figure 3 The operation intention in the application scenario shown is "lock the screen to view notifications from friend A in the social application", then Figure 3A bubble 51 pops up on the side or bottom of the interface shown, and the operation guidance information of "whether to execute the operation of locking the screen to view the notification of social application friend A" is displayed in the bubble 51. If the user clicks on the bubble 51, or calls up the voice assistant to input voice confirmation information, the device directly executes the device response processing of locking the screen to view the notification of social application friend A, so as to directly reach the chat interface of social application friend A. In this way, the user operation can be simplified, which is especially suitable for simplified commands when the hands are not convenient to operate while driving.

[0217] Another example: Figure 6 As shown, if the user is in Figure 4 In the application scenario shown, the text "Singer B's new song Daoxiang is good" 60 is selected, and the predicted operation intention is "Search and share singer B's Daoxiang to social applications", and an operation guidance information 62 pops up below the operation option 61 of the selected text. The operation guidance information 62 may include operation guidance options that match the operation intention. Clicking the operation guidance option can trigger the device response processing corresponding to the operation guidance option, such as adding singer B's Daoxiang to social applications, or searching for singer B's Daoxiang.

[0218] In this embodiment, after inferring the operation intention of the device control operation, the operation guidance information can be output to directly execute the device response processing corresponding to the operation intention based on the user's feedback operation on the operation guidance information, thereby realizing relevant recommendations and guidance directly reaching the operation intention and improving operation efficiency.

[0219] As an optional implementation manner, when the user portrait includes historical operation information, after the receiving of the feedback operation on the operation guidance information, the method further includes:

[0220] Feedback correction is performed on the historical operation information according to the operation type of the feedback operation.

[0221] In some embodiments of the present application, the operation type of the feedback operation may include positive feedback and negative feedback.

[0222] When the operation type of the feedback operation is positive feedback, it indicates that the user approves the predicted operation intention; when the operation type of the feedback operation is negative feedback, it indicates that the user does not approve the predicted operation intention.

[0223] In this embodiment, by performing feedback correction on the historical operation information according to the operation type of the feedback operation, the weight of the device control operation information corresponding to the operation intention recognized by the user in the historical operation information can be increased, or the weight of the device control operation information corresponding to the operation intention not recognized by the user in the historical operation information can be reduced. In this way, closed-loop correction of the historical operation information can be achieved, and the next time the corrected historical operation information is used to predict the operation intention of the device control operation, the corrected historical operation information can be made more consistent with the user's behavior habits or preferences, thereby making the predicted operation intention more accurate.

[0224] In some implementations of the present application, the step of performing feedback correction on the historical operation information according to the operation type of the feedback operation includes:

[0225] In the case where the operation type of the feedback operation is a positive feedback operation for controlling the device to perform a device response process corresponding to the operation intention, adding Figure 1 the number of operation executions and weight coefficients of historical device control operations of the corresponding operation type;

[0226] In the case where the operation type of the feedback operation is a negative feedback operation for controlling the device not to perform the device response processing corresponding to the operation intention, reducing the negative feedback operation corresponding to the operation intention Figure 1 The number of operation executions and weight coefficients of historical device control operations of the corresponding operation type.

[0227] For example, assuming that the adjustment factor is a, if the operation type of the feedback operation is a positive feedback operation for controlling the device to perform a device response process corresponding to the operation intention, then Figure 1 The number of operation executions of the historical device control operation of the corresponding operation type is (1+a)*the number of executions of the operation intention Figure 1 If the operation type of the feedback operation is a negative feedback operation for controlling the device not to perform the device response processing corresponding to the operation intention, then the number of original operation executions of the historical device control operation of the corresponding operation type is Figure 1 The number of operation executions of the historical device control operation of the corresponding operation type is (1-a)*the number of executions of the operation intention Figure 1 The number of executions of the original operation of the historical device control operation of the corresponding operation type, where a is a constant greater than 0 and less than 1.

[0228] For another example: Assume a is 0.1, for Figure 3In the application scenario one shown, if the user finally performs an operation related to viewing social application messages on the lock screen, such as jumping to the message viewing interface of the social application based on the prompt information, the weight coefficient of the operation purpose information of viewing social application messages on the lock screen can be determined as = ((30+20(1+0.1)) / (30+20(1+0.1)+50+30)+1)*(20(1+0.1) / (20(1+0.1)+30))=0.16; if the user does not perform an operation related to viewing social application messages on the lock screen, such as closing the prompt information for jumping to the message viewing interface of the social application, the weight coefficient of the operation purpose information of viewing social application messages on the lock screen can be determined as = ((30+20(1-0.1)) / (30+20(1-0.1)+50+30)+1)*(20(1-0.1) / (20(1-0.1)+30))=0.14.

[0229] It is worth mentioning that when the second largest language model is used for the first time to predict the operation intention, the user profile input to the second largest language model lacks historical operation information or is the default historical operation information. And by using the second largest language model to predict the operation intention and according to the user's feedback on the operation intention, the historical operation information is gradually improved, so that the historical operation information can more and more accurately reflect the user's behavior habits or preferences, and then the operation intention predicted based on the historical operation information that can accurately reflect the user's behavior habits or preferences can also be more and more in line with the user's behavior habits or preferences.

[0230] In this implementation, the weight coefficient and quantity of the operation purpose information corresponding to the operation intention in the user portrait can be updated accordingly according to whether the user's feedback type for the prediction result is positive feedback or negative feedback. By increasing the weight coefficient and quantity of the operation purpose information corresponding to the operation intention recognized by the user, the influence of the operation purpose information corresponding to the operation intention recognized by the user in the subsequent operation intention prediction process can be improved; by reducing the weight coefficient and quantity of the operation purpose information corresponding to the operation intention not recognized by the user, the influence of the operation purpose information corresponding to the operation intention recognized by the user in the subsequent operation intention prediction process can be reduced. In this way, the subsequent operation intention prediction results can be made more and more in line with the user's behavioral habits or preferences.

[0231] In some embodiments of the present application, historical operation information containing operation purpose information of device control operations can be stored locally on the device side in the form of a database, etc. Each time the historical operation information is called for operation intention prediction, corresponding to the same type of operation purpose, M historical device control operations and their operation purpose information with the largest weight coefficient or generated based on the latest device control operation can be selected as reference information for operation intention prediction, where M can be an integer greater than 1, such as 30.

[0232] In this way, the weights and usage frequencies of various types of operation purpose information are updated as the frequency with which the user performs the device control operation for the operation purpose. Thus, when the frequency with which the user performs a certain type of device control operation for an operation purpose is high, the number and weight coefficient of this type of operation purpose information will increase accordingly, thereby increasing the influence of this type of operation purpose information used to predict the operation intention information, thereby making the operation intention prediction results more in line with the user's operating habits.

[0233] In some implementations of the present application, historical operation information stored locally on the device side may be cleared. For example, for an operation type that has not had any new device control operations for a long time, the operation purpose data of that type may be cleared.

[0234] This way, you can avoid wasting memory space on data that has not been used for a long time.

[0235] In order to facilitate understanding of the embodiment of the present application, the second language model is used to predict the specific process of the operation intention of the device control operation according to the reference information, as shown in the following example. Figure 7 Taking the process shown in the figure as an example, the specific process of using the second largest language model to predict the operation intention of the device control operation according to the reference information is illustrated. Figure 7 As shown, in order to use the second largest language model to predict the operation intention of the device control operation according to the reference information, the following steps may be performed:

[0236] Step 701: Generate device operating environment information and device operation information in natural language format.

[0237] In this step, if there is no reference information matching the reference information to be predicted currently in the reference information corresponding to the stored historical operation purpose prediction results, after executing step 701, step 702 is directly executed. For example, when the operation intention prediction is performed for the first time, step 702 is executed after step 701.

[0238] In addition, when there is reference information matching the reference information currently to be predicted in the reference information corresponding to the stored historical operation purpose prediction results, the historical operation purpose prediction results corresponding to the reference information matching the reference information currently to be predicted can be used as the operation purpose corresponding to the reference information currently to be predicted. In this case, after executing step 701, step 705 can be directly executed.

[0239] Step 702: Build a user portrait.

[0240] In this step, based on the trigger of device control operation or device response processing, device operation information such as device operation data, application jump data, application usage data, copied or selected information entities can be collected, and then the device operation information can be input into a large language model to convert it into operation purpose information in a natural language format, and historical operation information can be obtained by clustering and statistics of a large amount of operation purpose information.

[0241] Step 703: construct training data.

[0242] In this step, device control operations or device response processing can be tracked to collect device operating environment information, device operation information, user portraits, and operation purpose information, and generate complete training data based on this.

[0243] Step 704: Predict the operation intention based on the device control operation trigger.

[0244] In this step, operation intention prediction can be triggered when a device control operation that matches or is similar to the device control operation used in the process of building a user portrait is detected, or when a device response processing that matches or is similar to the device response processing used in the process of building a user portrait is detected, such as when a specified APP is run.

[0245] Step 705: output operation guidance information according to the predicted operation intention.

[0246] In this step, the operation guidance information includes device response processing information of the device response processing corresponding to the direct operation intention.

[0247] Step 706: Correct the user portrait based on the feedback of the predicted operation intention.

[0248] In this step, the quantity or weight of the operation purpose information corresponding to the operation intention in the historical operation information can be updated based on whether the user accepts the predicted operation intention, such as whether a confirmation operation is performed on the operation guidance information corresponding to the operation intention.

[0249] Step 707: Record historical operation information.

[0250] In this step, the historical operation information may be stored locally on the device in the form of a database or the like, and each time an operation intention prediction is performed, the historical operation information may be read from the local database.

[0251] In addition, after this step, the user portrait can be updated based on the historical operation information recorded in this step, that is, after step 707, step 702 is executed.

[0252] In an embodiment of the present application, the device operating environment information, device operation information, and user operation information can be labeled through a large language model. When the user performs a device control operation, the reasoning ability of the large language model is combined to predict the user's operation intention, and the device response processing information corresponding to the operation intention is directly recommended to the user, which can realize relevant recommendations and guidance directly reaching the operation intention and improve operation efficiency.

[0253] As an optional implementation manner, when the operation intention of the device control operation includes N items of intention information, the operation guidance information includes N operation guidance options corresponding to the N items of intention information one by one; one operation guidance option is used to indicate whether a prompt is to execute device response processing information corresponding to one item of intention information, wherein N is an integer greater than 1;

[0254] After outputting the operation guidance information corresponding to the operation intention, the method further includes:

[0255] When a feedback operation is received for any one of the N operation guidance options, a device response process is performed according to the device response process information indicated by the operation guidance option selected by the feedback operation.

[0256] For example: Assuming that the device control operation is used to copy the text content "Singer B's song "Rice Fragrance" is good", and it is predicted that the operation intention of the device control operation may be to share singer B's song "Rice Fragrance" or to search for singer B's song "Rice Fragrance", at this time, two operation guidance options may be displayed, one operation guidance option is used to prompt whether to execute the device response processing information of sharing singer B's song "Rice Fragrance"; the other operation guidance option is used to prompt whether to execute the device response processing information of searching for singer B's song "Rice Fragrance". If the user clicks on the operation guidance option for prompting whether to execute the device response processing information of sharing singer B's song "Rice Fragrance", the device executes the device response processing of sharing singer B's song "Rice Fragrance", such as directly sharing singer B's song "Rice Fragrance" to a social application; if the user clicks on the operation guidance option for prompting whether to execute the device response processing information of searching for singer B's song "Rice Fragrance", the device executes the device response processing of searching for singer B's song "Rice Fragrance", such as directly searching for singer B's song "Rice Fragrance" on an audio application.

[0257] In this embodiment, corresponding to the same device control operation, one or more intention information can be predicted, and the output operation guidance information includes operation guidance options corresponding to the intention information one by one, so that the user can select any operation guidance option therein, so as to realize the recommendation and guidance of the device response processing corresponding to the selected operation guidance option, thereby making the operation intention prediction results more flexible and rich.

[0258] As an optional implementation, predicting the operation intention of the device control operation according to the reference information includes:

[0259] The reference information is encoded into a device operation vector, and the operation purpose information associated with the device operation vector matching the device operation vector in the vector library is determined as the operation intention of the device control operation; wherein different device operation vectors in the vector library are associated with different operation purpose information.

[0260] In some implementations of the present application, BERT encoding may be used to encode reference information into a device operation vector.

[0261] In some embodiments of the present application, the device operation vector in the vector library may be a vector obtained by collecting device operation information based on the trigger of historical device control operations or historical device response processing, such as device operation data, application jump data, application usage data, copied or selected information entities, and BERT encoding the device operation information, and each device operation vector in the vector library is associated with the operation purpose information obtained by reducing the device operation information to processing.

[0262] In some embodiments of the present application, when there is an information entity that has been selected or copied when receiving a device control operation, the device operation vector in the vector library can be replaced with the same entity, which can achieve simpler prediction of operation intention. For example: a device operation vector in the vector library is generated based on the copy operation of the text content "search for AAA singer's song a", and the device control operation that currently needs to predict the operation intention is to copy "search for song b", then the information entity "AAA singer's song a" in the operation purpose information corresponding to the above device operation vector in the vector library can be replaced with "song b" to obtain the predicted operation intention information of the device control operation for copying "search for song b".

[0263] In this embodiment, feature vectors corresponding to device operation information of various operation purposes are pre-stored, and feature vectors of reference information when receiving the current device control operation are extracted. By matching the feature vectors of the reference information with the feature vectors of device operation information of various operation purposes, the operation purpose corresponding to the feature vector of the device operation information that matches the feature vector of the reference information is used as the operation intention of the device control operation. In this way, the operation intention prediction can be realized without training and calling the second language model, but only the operation intention of the device operation vector in the constructed vector library can be matched.

[0264] It is worth mentioning that in the embodiment of the present application, the reference information, the operation purpose associated with the device operation vector in the vector library, the predicted operation intention, the operation intention recommendation or guidance, respectively, have the same meaning as the reference information, the operation purpose associated with the device operation vector in the vector library, the predicted operation intention, the operation intention recommendation or guidance in the aforementioned implementation method of using the second largest language model to predict the operation intention of the device control operation, and they are not repeated here.

[0265] For example: Figure 8 As shown, in order to predict the operation intention of the device control operation based on the device operation vector matching, the following steps may be performed:

[0266] Step 801: Generate device operating environment information and device operation information in natural language format.

[0267] In this step, if there is no reference information matching the reference information to be predicted currently in the reference information corresponding to the stored historical operation purpose prediction results, after executing step 801, step 802 is directly executed. For example, when the operation purpose prediction is performed for the first time, step 802 is executed after step 801.

[0268] In addition, when there is reference information matching the reference information currently to be predicted in the reference information corresponding to the stored historical operation purpose prediction results, the historical operation purpose prediction results corresponding to the reference information matching the reference information currently to be predicted can be used as the operation purpose corresponding to the reference information currently to be predicted. In this case, after executing step 801, step 804 can be directly executed.

[0269] Step 802: Build a user portrait.

[0270] Step 803: Predict the operation intention based on the device control operation trigger.

[0271] Steps 801 to 803 in this embodiment are similar to those in Figure 7 Therefore, step 701, step 702 and step 704 in the embodiment are the same; the difference is that the training data in step 703 is used to train the second largest language model, and in step 803, BERT encoding can be used to encode the reference information into an operation feature vector, and the operation feature vector is matched with the operation feature vector in the vector library generated based on step 803, so as to use the operation purpose information associated with the operation feature vector in the vector library that matches the operation feature vector as the predicted operation intention of the device control operation.

[0272] Step 804: output operation guidance information according to the predicted operation intention.

[0273] Step 805: Correct the user portrait based on the feedback of the predicted operation intention.

[0274] Step 806: Record historical operation information.

[0275] Steps 805 to 806 in this embodiment are similar to those in Figure 7 Therefore, step 705 to step 706 in the embodiment are similar and will not be described again.

[0276] In addition, after this step, the user portrait can be updated based on the historical operation information recorded in this step, that is, after step 806, step 802 is executed.

[0277] The operation guidance method in this embodiment is simpler. It matches the operation feature vector of the device control operation with the operation feature vector in the vector library to determine the operation intention of the device control operation based on the operation purpose information associated with the operation feature vector that matches the operation feature vector of the device control operation in the vector library, thereby reducing the resource overhead of the related large language model.

[0278] It is worth mentioning that the operation guidance method provided in the embodiment of the present application can be used to infer the operation intention of the user operation on an electronic device. In addition, it can also be used to infer the operation intention of the user operation between multiple electronic devices. For example, when a user has a mobile phone and a smart watch, and triggers the execution of a specified device response process on the mobile phone through the device control operation on the smart watch, the operation intention implemented on the mobile phone can be predicted based on the device control operation on the smart watch.

[0279] The operation guidance method provided in the embodiment of the present application can be executed by an operation guidance device. In the embodiment of the present application, the operation guidance device provided in the embodiment of the present application is described by taking the operation guidance method executed by the operation guidance device as an example.

[0280] like Fig. 9 As shown, the operation guidance device 900 provided in the embodiment of the present application includes the following modules:

[0281] A prediction module 901 is used to predict the operation intention of the device control operation according to reference information when receiving the device control operation; the reference information includes device operating environment information, device operation information and user portrait;

[0282] An output module 902 is used to output operation guidance information corresponding to the operation intention, wherein the operation guidance information includes device response processing information for prompting whether to execute the device response processing corresponding to the operation intention;

[0283] Among them, the device operating environment information includes at least one of the following: device status information, device time information, foreground application information, background application information, notification information, and media playback status information; the device operation information includes operation feature information of the device control operation.

[0284] In some embodiments of the present application, the operation guidance device 900 further includes:

[0285] The acquisition module is used to obtain the system log information and system status information of the device;

[0286] The structured processing module is used to input the system log information and the system status information into a first large language model, perform structured processing on the system log information and the system status information through the first large language model, and output the device operating environment information and device operation information in a natural language format.

[0287] In some embodiments of the present application, the device operation information also includes a copied or selected information entity;

[0288] The prediction module 901 is specifically used for:

[0289] An operation intention of the device control operation matching the information entity is predicted according to the reference information.

[0290] In some implementations of the present application, the user portrait includes historical operation information;

[0291] The historical operation information includes an operation type of at least one historical device control operation within a historical time period and an operation execution count and a weight coefficient of the historical device control operation of each operation type.

[0292] In some embodiments of the present application, the operation guidance device 900 further includes:

[0293] A collection module, used to collect device operation information when a device control operation or device response processing is detected within a detection time period, wherein the device operation information includes at least one of the following: device operation data, application jump data, application usage data, copied or selected information entities;

[0294] An information processing module, used for performing dimensionality reduction processing on the device operation information to obtain operation purpose information corresponding to the device control operation or device response processing, wherein the operation purpose information includes an operation type and an operation object;

[0295] The statistical module is used to collect statistics on all operation purpose information collected during the detection time period to obtain historical operation information, and the user portrait includes the historical operation information.

[0296] In some embodiments of the present application, the operation guidance device 900 further includes:

[0297] An operation receiving module, used for receiving a feedback operation on the operation guidance information;

[0298] The response module is used to perform device response processing according to the device response processing information.

[0299] In some implementations of the present application, when the user portrait includes historical operation information, the operation guidance device 900 further includes:

[0300] A feedback correction module is used to perform feedback correction on the historical operation information according to the operation type of the feedback operation.

[0301] In some embodiments of the present application, the feedback correction module is specifically used to:

[0302] In the case where the operation type of the feedback operation is a positive feedback operation for controlling the device to perform a device response process corresponding to the operation intention, adding Figure 1 the number of operation executions and weight coefficients of historical device control operations of the corresponding operation type;

[0303] In the case where the operation type of the feedback operation is a negative feedback operation for controlling the device not to perform the device response processing corresponding to the operation intention, reducing the negative feedback operation corresponding to the operation intention Figure 1 The number of operation executions and weight coefficients of historical device control operations of the corresponding operation type.

[0304] In some embodiments of the present application, when the operation intention of the device control operation includes N items of intention information, the operation guidance information includes N operation guidance options corresponding to the N items of intention information one by one; one operation guidance option is used to indicate whether a prompt executes device response processing information corresponding to one item of intention information, wherein N is an integer greater than 1;

[0305] The operation guiding device 900 further includes:

[0306] The response module is used to perform device response processing according to the device response processing information indicated by the operation guidance option selected by the feedback operation when receiving a feedback operation on any one of the N operation guidance options.

[0307] In some implementations of the present application, the prediction module 901 is specifically used to:

[0308] Using the second language model, based on the reference information, predict the operation intention of the device control operation;

[0309] Alternatively, the reference information is encoded into a device operation vector, and the operation purpose information associated with the device operation vector matching the device operation vector in the vector library is determined as the operation intention of the device control operation;

[0310] Different device operation vectors in the vector library are associated with different operation purpose information.

[0311] In some implementations of the present application, the prediction module 901 is specifically used to:

[0312] Generate operation intention prediction prompt words based on reference information;

[0313] Inputting the operation intention prediction prompt word into a second language model, and outputting the operation intention of the device control operation through the second language model;

[0314] The operation intention prediction prompt word includes at least one of the following: the reference information, the first indication information, and the second indication information;

[0315] The first indication information is used to indicate that the task performed by the second large language model is to predict the operation intention based on the reference information;

[0316] The second indication information is used to indicate the prediction task requirements of the second largest language model, wherein the prediction task requirements include at least one of the following: the operation intention predicted by the second largest language model matches the user portrait; the device operating environment information is used as a variable for the second largest language model to predict the operation intention; the device operation information is used as a variable for the second largest language model to predict the operation intention; the prediction result output by the second largest language model is a task-type short sentence, and the task-type short sentence includes an operation type and an operation object.

[0317] The operation guidance device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0318] The operation guidance device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0319] The operation guidance device provided in the embodiment of the present application can achieve Figure 1 , Figure 2 , Figure 7 or Figure 8 The various processes implemented by the method embodiment shown can achieve the same or similar beneficial effects as the aforementioned method embodiment, and will not be described again here to avoid repetition.

[0320] Alternatively, if Fig.10 As shown, an embodiment of the present application further provides an electronic device 1000, including a processor 1001 and a memory 1002, wherein the memory 1002 stores programs or instructions that can be executed on the processor 1001, and when the program or instructions are executed by the processor M01, the various steps of the aforementioned operation guidance method embodiment of the present application are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0321] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0322] Fig.11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0323] The electronic device 1100 includes but is not limited to: a radio frequency unit 1101, a network module 1102, an audio output unit 1103, an input unit 1104, a sensor 1105, a display unit 1106, a user input unit 1107, an interface unit 1108, a memory 1109, and a processor 1110 and other components.

[0324] Those skilled in the art will appreciate that the electronic device 1100 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0325] The processor 1110 is configured to predict the operation intention of the device control operation according to the reference information when receiving the device control operation through the user input unit 1107; the reference information includes the device operating environment information, the device operation information and the user portrait;

[0326] The processor 1110 is further configured to output operation guidance information corresponding to the operation intention through at least one of the audio output unit 1103 and the display unit 1106, wherein the operation guidance information includes device response processing information for prompting whether to execute the device response processing corresponding to the operation intention;

[0327] Among them, the device operating environment information includes at least one of the following: device status information, device time information, foreground application information, background application information, notification information, and media playback status information; the device operation information includes operation feature information of the device control operation.

[0328] In some implementations of the present application, the processor 1110 is further configured to:

[0329] Obtain system log information and system status information of the device;

[0330] The system log information and the system status information are input into a first large language model, and the system log information and the system status information are structured through the first large language model to output device operating environment information and device operation information in a natural language format.

[0331] In some embodiments of the present application, the device operation information also includes a copied or selected information entity;

[0332] The processor 1110 predicts the operation intention of the device control operation according to the reference information, including:

[0333] An operation intention of the device control operation matching the information entity is predicted according to the reference information.

[0334] In some implementations of the present application, the user portrait includes historical operation information;

[0335] The historical operation information includes an operation type of at least one historical device control operation within a historical time period and an operation execution count and a weight coefficient of the historical device control operation of each operation type.

[0336] In some implementations of the present application, before executing the operation intention of predicting the device control operation according to the reference information, the processor 1110 is further configured to:

[0337] In the detection time period, when a device control operation or a device response process is detected, device operation information is collected, wherein the device operation information includes at least one of the following: device operation data, application jump data, application usage data, copied or selected information entities;

[0338] Performing dimensionality reduction processing on the device operation information to obtain operation purpose information corresponding to the device control operation or device response processing, wherein the operation purpose information includes an operation type and an operation object;

[0339] Statistics are collected on all operation purpose information collected during the detection time period to obtain historical operation information, and the user portrait includes the historical operation information.

[0340] In some implementations of the present application, after executing the outputting of the operation guidance information corresponding to the operation intention, the processor 1110 is further configured to:

[0341] receiving a feedback operation on the operation guidance information through the user input unit 1107;

[0342] The device response process is executed according to the device response process information.

[0343] In some implementations of the present application, when the processor 1110 executes the user portrait including historical operation information, after the feedback operation of receiving the operation guidance information is further configured to:

[0344] Feedback correction is performed on the historical operation information according to the operation type of the feedback operation.

[0345] In some implementations of the present application, the processor 1110 performs feedback correction on the historical operation information according to the operation type of the feedback operation, including:

[0346] In the case where the operation type of the feedback operation is a positive feedback operation for controlling the device to perform a device response process corresponding to the operation intention, adding Figure 1 the number of operation executions and weight coefficients of historical device control operations of the corresponding operation type;

[0347] In the case where the operation type of the feedback operation is a negative feedback operation for controlling the device not to perform the device response processing corresponding to the operation intention, reducing the Figure 1 The number of operation executions and weight coefficients of historical device control operations of the corresponding operation type.

[0348] In some embodiments of the present application, when the operation intention of the device control operation includes N items of intention information, the operation guidance information includes N operation guidance options corresponding to the N items of intention information one by one; one operation guidance option is used to indicate whether a prompt executes device response processing information corresponding to one item of intention information, wherein N is an integer greater than 1;

[0349] After executing the outputting of the operation guidance information corresponding to the operation intention, the processor 1110 is further configured to:

[0350] When a feedback operation on any one of the N operation guidance options is received through the user input unit 1107, a device response process is performed according to the device response process information indicated by the operation guidance option selected by the feedback operation.

[0351] In some implementations of the present application, the processor 1110 performs the step of predicting the operation intention of the device control operation according to the reference information, including:

[0352] Using the second language model, based on the reference information, predict the operation intention of the device control operation;

[0353] Alternatively, the reference information is encoded into a device operation vector, and the operation purpose information associated with the device operation vector matching the device operation vector in the vector library is determined as the operation intention of the device control operation;

[0354] Different device operation vectors in the vector library are associated with different operation purpose information.

[0355] In some implementations of the present application, the processor 1110 uses the second largest language model to predict the operation intention of the device control operation according to the reference information, including:

[0356] Generate operation intention prediction prompt words based on reference information;

[0357] Inputting the operation intention prediction prompt word into a second language model, and outputting the operation intention of the device control operation through the second language model;

[0358] The operation intention prediction prompt word includes at least one of the following: the reference information, the first indication information, and the second indication information;

[0359] The first indication information is used to indicate that the task performed by the second large language model is to predict the operation intention based on the reference information;

[0360] The second indication information is used to indicate the prediction task requirements of the second largest language model, wherein the prediction task requirements include at least one of the following: the operation intention predicted by the second largest language model matches the user portrait; the device operating environment information is used as a variable for the second largest language model to predict the operation intention; the device operation information is used as a variable for the second largest language model to predict the operation intention; the prediction result output by the second largest language model is a task-type short sentence, and the task-type short sentence includes an operation type and an operation object.

[0361] It can be understood that the implementation process of each implementation method mentioned in this embodiment can refer to the relevant description of the aforementioned operation guidance method embodiment, and achieve the same or corresponding technical effect. To avoid repetition, it will not be repeated here.

[0362] It should be understood that in the embodiment of the present application, the input unit 1104 may include a graphics processor (Graphics Processing Unit, GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0363] The memory 1109 can be used to store software programs and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1109 may include a volatile memory or a non-volatile memory, or the memory 1109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0364] The processor 1110 may include one or more processing units; optionally, the processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0365] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the aforementioned operation guidance method embodiment of the present application are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0366] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0367] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the aforementioned operation guidance method embodiment of the present application, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0368] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0369] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the operation guidance method embodiment as described above in the present application, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0370] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0371] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0372] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An operation guidance method, It is characterized in that The method comprises: When receiving a device control operation, predicting the operation intention of the device control operation according to reference information; the reference information includes device operating environment information, device operation information and user portrait; Outputting operation guidance information corresponding to the operation intention, the operation guidance information including device response processing information for prompting whether to execute the device response processing corresponding to the operation intention; Among them, the device operating environment information includes at least one of the following: device status information, device time information, foreground application information, background application information, notification information, and media playback status information; the device operation information includes operation feature information of the device control operation.

2. The method according to claim 1, It is characterized in that The method further comprises: Obtain system log information and system status information of the device; The system log information and the system status information are input into a first large language model, and the system log information and the system status information are structured through the first large language model to output device operating environment information and device operation information in a natural language format.

3. The method according to claim 1, It is characterized in that The device operation information also includes copied or selected information entities; The predicting, based on the reference information, the operation intention of the device control operation includes: An operation intention of the device control operation matching the information entity is predicted according to the reference information.

4. The method according to claim 1, It is characterized in that The user portrait includes historical operation information; The historical operation information includes an operation type of at least one historical device control operation within a historical time period and an operation execution count and a weight coefficient of the historical device control operation of each operation type.

5. The method according to claim 1, It is characterized in that Before predicting the operation intention of the device control operation according to the reference information, the method further includes: In the detection time period, when a device control operation or a device response process is detected, device operation information is collected, wherein the device operation information includes at least one of the following: device operation data, application jump data, application usage data, copied or selected information entities; Performing dimensionality reduction processing on the device operation information to obtain operation purpose information corresponding to the device control operation or device response processing, wherein the operation purpose information includes an operation type and an operation object; Statistics are collected on all operation purpose information collected during the detection time period to obtain historical operation information, and the user portrait includes the historical operation information.

6. The method according to claim 1, It is characterized in that After outputting the operation guidance information corresponding to the operation intention, the method further includes: receiving a feedback operation on the operation guidance information; The device response process is executed according to the device response process information.

7. The method according to claim 6, It is characterized in that In the case where the user portrait includes historical operation information, after the receiving of feedback on the operation guidance information, the method further includes: Feedback correction is performed on the historical operation information according to the operation type of the feedback operation.

8. The method according to claim 7, It is characterized in that The step of performing feedback correction on the historical operation information according to the operation type of the feedback operation includes: In the case where the operation type of the feedback operation is a positive feedback operation for controlling the device to perform a device response process corresponding to the operation intention, increasing the operation execution count and weight coefficient of the historical device control operation of the operation type consistent with the operation intention; When the operation type of the feedback operation is a negative feedback operation for controlling the device not to perform a device response process corresponding to the operation intention, the operation execution count and weight coefficient of historical device control operations of the operation type consistent with the operation intention are reduced.

9. The method according to claim 1, It is characterized in that In a case where the operation intention of the device control operation includes N items of intention information, the operation guidance information includes N operation guidance options corresponding one-to-one to the N items of intention information; An operation guidance option is used to indicate whether a prompt executes device response processing information corresponding to an intention information, where N is an integer greater than 1; After outputting the operation guidance information corresponding to the operation intention, the method further includes: When a feedback operation is received for any one of the N operation guidance options, a device response process is performed according to the device response process information indicated by the operation guidance option selected by the feedback operation.

10. The method according to claim 1, It is characterized in that The predicting, based on the reference information, the operation intention of the device control operation includes: Using the second language model, based on the reference information, predict the operation intention of the device control operation; Alternatively, the reference information is encoded into a device operation vector, and the operation purpose information associated with the device operation vector matching the device operation vector in the vector library is determined as the operation intention of the device control operation; Different device operation vectors in the vector library are associated with different operation purpose information.

11. The method according to claim 10, It is characterized in that The using of the second largest language model to predict the operation intention of the device control operation according to the reference information includes: Generate operation intention prediction prompt words based on reference information; Inputting the operation intention prediction prompt word into a second language model, and outputting the operation intention of the device control operation through the second language model; The operation intention prediction prompt word includes at least one of the following: the reference information, the first indication information, and the second indication information; The first indication information is used to indicate that the task performed by the second large language model is to predict the operation intention based on the reference information; The second indication information is used to indicate the prediction task requirements of the second largest language model, wherein the prediction task requirements include at least one of the following: the operation intention predicted by the second largest language model matches the user portrait; the device operating environment information is used as a variable for the second largest language model to predict the operation intention; the device operation information is used as a variable for the second largest language model to predict the operation intention; the prediction result output by the second largest language model is a task-type short sentence, and the task-type short sentence includes an operation type and an operation object.

12. An operation guiding device, It is characterized in that The device comprises: A prediction module, configured to predict the operation intention of the device control operation based on reference information when receiving the device control operation; the reference information includes device operating environment information, device operation information and user portrait; an output module, configured to output operation guidance information corresponding to the operation intention, wherein the operation guidance information includes device response processing information for prompting whether to execute the device response processing corresponding to the operation intention; Among them, the device operating environment information includes at least one of the following: device status information, device time information, foreground application information, background application information, notification information, and media playback status information; the device operation information includes operation feature information of the device control operation.

13. The device according to claim 12, It is characterized in that The device also includes: The acquisition module is used to obtain the system log information and system status information of the device; The structured processing module is used to input the system log information and the system status information into a first large language model, perform structured processing on the system log information and the system status information through the first large language model, and output the device operating environment information and device operation information in a natural language format.

14. The device according to claim 12, It is characterized in that The device operation information also includes copied or selected information entities; The prediction module is specifically used for: An operation intention of the device control operation matching the information entity is predicted according to the reference information.

15. The device according to claim 12, It is characterized in that The user portrait includes historical operation information; The historical operation information includes an operation type of at least one historical device control operation within a historical time period and an operation execution count and a weight coefficient of the historical device control operation of each operation type.

16. The device according to claim 12, It is characterized in that The device also includes: A collection module, used to collect device operation information when a device control operation or device response processing is detected within a detection time period, wherein the device operation information includes at least one of the following: device operation data, application jump data, application usage data, copied or selected information entities; An information processing module, used for performing dimensionality reduction processing on the device operation information to obtain operation purpose information corresponding to the device control operation or device response processing, wherein the operation purpose information includes an operation type and an operation object; The statistical module is used to collect statistics on all operation purpose information collected during the detection time period to obtain historical operation information, and the user portrait includes the historical operation information.

17. The device according to claim 12, It is characterized in that The device also includes: An operation receiving module, used for receiving a feedback operation on the operation guidance information; The response module is used to perform device response processing according to the device response processing information.

18. The device according to claim 17, It is characterized in that In the case where the user portrait includes historical operation information, the device further includes: A feedback correction module is used to perform feedback correction on the historical operation information according to the operation type of the feedback operation.

19. The device according to claim 18, It is characterized in that The feedback correction module is specifically used for: In the case where the operation type of the feedback operation is a positive feedback operation for controlling the device to perform a device response process corresponding to the operation intention, increasing the operation execution count and weight coefficient of the historical device control operation of the operation type consistent with the operation intention; When the operation type of the feedback operation is a negative feedback operation for controlling the device not to perform a device response process corresponding to the operation intention, the operation execution count and weight coefficient of historical device control operations of the operation type consistent with the operation intention are reduced.

20. The device according to claim 12, It is characterized in that In a case where the operation intention of the device control operation includes N items of intention information, the operation guidance information includes N operation guidance options corresponding one-to-one to the N items of intention information; An operation guidance option is used to indicate whether a prompt executes device response processing information corresponding to an intention information, where N is an integer greater than 1; The device also includes: The response module is used to perform device response processing according to the device response processing information indicated by the operation guidance option selected by the feedback operation when receiving a feedback operation on any one of the N operation guidance options.

21. The device according to claim 12, It is characterized in that The prediction module is specifically used for: Using the second language model, based on the reference information, predict the operation intention of the device control operation; Alternatively, the reference information is encoded into a device operation vector, and the operation purpose information associated with the device operation vector matching the device operation vector in the vector library is determined as the operation intention of the device control operation; Different device operation vectors in the vector library are associated with different operation purpose information.

22. The device according to claim 21, It is characterized in that The prediction module is specifically used for: Generate operation intention prediction prompt words based on reference information; Inputting the operation intention prediction prompt word into a second language model, and outputting the operation intention of the device control operation through the second language model; The operation intention prediction prompt word includes at least one of the following: the reference information, the first indication information, and the second indication information; The first indication information is used to indicate that the task performed by the second large language model is to predict the operation intention based on the reference information; The second indication information is used to indicate the prediction task requirements of the second largest language model, wherein the prediction task requirements include at least one of the following: the operation intention predicted by the second largest language model matches the user portrait; the device operating environment information is used as a variable for the second largest language model to predict the operation intention; the device operation information is used as a variable for the second largest language model to predict the operation intention; the prediction result output by the second largest language model is a task-type short sentence, and the task-type short sentence includes an operation type and an operation object.

23. An electronic device, include: A memory, a processor, and a computer program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps in the operation guidance method as described in any one of claims 1 to 11.

Images