Behavior learning-oriented intelligent operation model construction method, device and system

Abstract

The invention provides a behavior learning-oriented intelligent operation model construction method, device and system, and relates to the technical field of data processing, and the method comprises the steps: generating a sample user operation information sequence used for achieving a sample task in target equipment based on an operation image and an operation log; wherein sample user operation information in the sample user operation information sequence comprises an operation instruction determined based on the operation log and an operation target determined based on the operation image; inputting sample natural language instruction information corresponding to the sample task into a first language model to obtain a first operation information sequence output by the first language model; and based on a loss value between a first operation information sequence corresponding to the sample task and a sample user operation information sequence of the sample task, adjusting model parameters of the first language model to obtain an intelligent operation model. The efficiency of obtaining task implementation steps and the efficiency of executing a target task by a user through target equipment can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a method, device and system for constructing an intelligent operation model for behavior learning. Background Art

[0002] With the continuous development of science and technology, more and more devices are beginning to be used in people's lives. In order to improve production efficiency, how to enable these devices to automatically perform specific tasks has become an urgent problem to be solved.

[0003] In related technologies, technicians usually study the execution methods of tasks that need to be automated, write task execution steps based on experience, break down the tasks into multiple steps and write corresponding execution codes to obtain task execution programs, and automatically execute these tasks by running the task execution programs on the device.

[0004] However, the above method of decomposing the task execution steps is inefficient, resulting in a large number of tasks that cannot be automatically processed on the device, resulting in poor efficiency in users executing tasks through the device. Summary of the invention

[0005] The embodiments of the present invention provide a method, device and system for constructing an intelligent operation model oriented to behavior learning, so as to solve the problem of poor efficiency of users performing tasks through devices in related technologies.

[0006] In a first aspect, an embodiment of the present invention provides a method for constructing an intelligent operation model oriented to behavior learning, the method comprising:

[0007] Acquire an operation image generated when a sample user operates a target device to implement a sample task, and an operation log generated when the target device performs the sample task;

[0008] Based on the operation image and the operation log, a sample user operation information sequence for implementing the sample task on the target device is generated; wherein the sample user operation information in the sample user operation information sequence includes an operation instruction determined based on the operation log and an operation target determined based on the operation image;

[0009] Inputting sample natural language instruction information corresponding to the sample task into a first language model to obtain a first operation information sequence output by the first language model;

[0010] Based on the loss value between the first operation information sequence corresponding to the sample task and the sample user operation information sequence of the sample task, the model parameters of the first language model are adjusted to obtain an intelligent operation model; wherein the intelligent operation model is used to output a corresponding target operation information sequence according to the target natural language instruction information, and the target operation information sequence is used to be applied on a target device so that the target device can achieve the target task corresponding to the target natural language instruction information.

[0011] Optionally, generating a sample user operation information sequence for implementing the sample task on the target device based on the operation image and the operation log includes:

[0012] Based on the operation image, extracting the user interaction objects of the sample user and the interaction time corresponding to each user interaction object;

[0013] Based on the operation log, extract the operation instructions of the sample user and the operation time corresponding to each operation instruction;

[0014] Determine the user interaction object whose interaction time overlaps with the operation time of the operation instruction as the operation object corresponding to the operation instruction;

[0015] Based on the operation instructions and the operation objects corresponding to the operation instructions, sample user operation information corresponding to the operation instructions is constructed, and the sample user operation information corresponding to each operation instruction is arranged according to the operation time to obtain the sample user operation information sequence.

[0016] Optionally, the operation image includes an operation scene image and an operation screen image, and extracting the user interaction object of the sample user based on the operation image includes:

[0017] In a case where the operation image includes the operation scene image, determining a first interaction object of the target device in the scene based on the operation scene image, and using the first interaction object as the user interaction object;

[0018] In the case where the operation image includes the operation screen image, a second interaction object is determined from the operation screen image based on an input position of a user input of the sample user in the operation screen image, and the second interaction object is used as the user interaction object.

[0019] Optionally, determining a first interactive object of the target device in a scene based on the operation scene image includes:

[0020] Inputting the operation scene image including the target device into a target detection model, and obtaining a first target detection result of an interactive target object corresponding to the target device and a second target detection result corresponding to the target device output by the target detection model;

[0021] Inputting the first target detection result, the second target detection result, and the sample task into an intention recognition model to obtain interaction probabilities corresponding to each of the interactive target objects output by the intention recognition model;

[0022] The interactive target object with the highest interaction probability is determined as the first interactive object.

[0023] Optionally, obtaining an operation image generated when a sample user operates a target device to implement a sample task includes:

[0024] Detecting the operation type of the operation instruction of the sample user on the target device;

[0025] When the operation type of the operation instruction is an actuator operation, photographing the operation scene image by an external camera of the target device;

[0026] When the operation type of the operation instruction is an interface operation, an operation screen image corresponding to the operation instruction is acquired.

[0027] Optionally, inputting the sample natural language instruction information corresponding to the sample task into a first language model to obtain a first operation information sequence output by the first language model includes:

[0028] Acquire sample environment information of the target device during the period when the sample user operates the target device to implement the sample task;

[0029] The sample natural language instruction information and the sample environment information are input into the first language model to obtain the first operation information sequence output by the first language model.

[0030] Optionally, the method further comprises:

[0031] Obtaining question feedback information for the target operation information sequence;

[0032] Generate model adjustment prompt word information based on the question feedback information, the target operation information sequence and the target natural language instruction information corresponding to the target operation information sequence;

[0033] The model adjustment prompt word information is input into the intelligent operation model.

[0034] In a second aspect, an embodiment of the present invention provides a device for constructing an intelligent operation model for behavior learning, the device comprising:

[0035] An acquisition module, used to acquire an operation image generated when a sample user operates a target device to implement a sample task, and an operation log generated when the target device performs the sample task;

[0036] A sequence module, configured to generate a sample user operation information sequence for implementing the sample task on the target device based on the operation image and the operation log; wherein the sample user operation information in the sample user operation information sequence includes an operation instruction determined based on the operation log and an operation target determined based on the operation image;

[0037] An input module, configured to input the sample natural language instruction information corresponding to the sample task into a first language model to obtain a first operation information sequence output by the first language model;

[0038] A training module is used to adjust the model parameters of the first language model based on the loss value between the first operation information sequence corresponding to the sample task and the sample user operation information sequence of the sample task, so as to obtain an intelligent operation model; wherein the intelligent operation model is used to output a corresponding target operation information sequence according to the target natural language instruction information, and the target operation information sequence is used to be applied in a target device so that the target device can achieve the target task corresponding to the target natural language instruction information.

[0039] Optionally, the sequence module includes:

[0040] A first extraction submodule, configured to extract the user interaction objects of the sample user and the interaction time corresponding to each user interaction object based on the operation image;

[0041] A second extraction submodule, configured to extract the operation instructions of the sample user and the operation time corresponding to each operation instruction based on the operation log;

[0042] An operation object submodule, configured to determine a user interaction object whose interaction time overlaps with an operation time of the operation instruction as an operation object corresponding to the operation instruction;

[0043] The sequence construction submodule is used to construct sample user operation information corresponding to the operation instruction based on the operation instruction and the operation object corresponding to the operation instruction, and arrange the sample user operation information corresponding to each operation instruction according to the operation time to obtain the sample user operation information sequence.

[0044] Optionally, the operation image includes an operation scene image and an operation screen image, and the first extraction submodule includes:

[0045] A first interaction object extraction unit, configured to determine, when the operation image includes the operation scene image, a first interaction object of the target device in the scene based on the operation scene image, and use the first interaction object as the user interaction object;

[0046] A second interaction object extraction unit is used to determine a second interaction object from the operation screen image based on an input position of a user input of the sample user in the operation screen image when the operation image includes the operation screen image, and use the second interaction object as the user interaction object.

[0047] Optionally, the first interactive object extraction unit includes:

[0048] A detection result subunit, configured to input the operation scene image including the target device into a target detection model, and obtain a first target detection result of an interactive target object corresponding to the target device output by the target detection model, and a second target detection result corresponding to the target device;

[0049] An interaction probability subunit, configured to input the first target detection result, the second target detection result, and the sample task into an intention recognition model to obtain interaction probabilities corresponding to each of the interactive target objects output by the intention recognition model;

[0050] The first interactive object determining subunit is configured to determine the interactive target object with the highest interaction probability as the first interactive object.

[0051] Optionally, the acquisition module includes:

[0052] An operation type submodule, used to detect the operation type of the operation instruction of the sample user on the target device;

[0053] An operation scene image submodule, used for capturing the operation scene image by an external camera of the target device when the operation type of the operation instruction is an actuator operation;

[0054] The operation screen image submodule is used to obtain the operation screen image corresponding to the operation instruction when the operation type of the operation instruction is an interface operation.

[0055] Optionally, the input module includes:

[0056] A sample environment information submodule, used to obtain sample environment information of the target device during the period when the sample user operates the target device to implement the sample task;

[0057] The environment information input submodule is used to input the sample natural language instruction information and the sample environment information into the first language model to obtain the first operation information sequence output by the first language model.

[0058] Optionally, the device further comprises:

[0059] A question feedback information acquisition module, used to acquire question feedback information for a target operation information sequence;

[0060] An adjustment prompt word information module is used to generate model adjustment prompt word information based on the question feedback information, the target operation information sequence and the target natural language instruction information corresponding to the target operation information sequence;

[0061] The adjustment prompt word information input module is used to input the model adjustment prompt word information into the intelligent operation model.

[0062] In a third aspect, an embodiment of the present invention provides an intelligent operating system, the intelligent operating system comprising an instruction acquisition unit, an inference unit and an execution unit;

[0063] The instruction acquisition unit is used to acquire target natural language instruction information input by a target user;

[0064] The reasoning unit is used to input the target natural language instruction information into the intelligent operation model so that the intelligent operation model outputs a target operation information sequence; wherein the intelligent operation model is generated based on the intelligent operation model construction method for behavior learning as described in claims 1 to 7;

[0065] The execution unit is used to control the target device to achieve the target task corresponding to the target natural language instruction information based on the target operation information sequence.

[0066] Optionally, the intelligent operating system further includes an environment information acquisition unit;

[0067] The environment information acquisition unit is used to acquire target environment information where the target device is located;

[0068] The reasoning unit is further used to input the target natural language instruction information and the target environment information into the intelligent operation model so that the intelligent operation model outputs a target operation information sequence.

[0069] In a fourth aspect, an embodiment of the present invention further provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method for constructing an intelligent operation model for behavior learning as in the first aspect.

[0070] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the method for constructing an intelligent operation model for behavior learning as in the first aspect.

[0071] In an embodiment of the present invention, the method includes: obtaining an operation image generated when a sample user operates a target device to implement a sample task, and an operation log generated when the target device executes the sample task; generating a sample user operation information sequence for implementing the sample task on the target device based on the operation image and the operation log; wherein the sample user operation information in the sample user operation information sequence includes an operation instruction determined based on the operation log and an operation target determined based on the operation image; inputting sample natural language instruction information corresponding to the sample task into a first language model to obtain a first operation information sequence output by the first language model; adjusting a model parameter of the first language model based on a loss value between the first operation information sequence corresponding to the sample task and the sample user operation information sequence of the sample task to obtain an intelligent operation model; wherein the intelligent operation model is used to output a corresponding target operation information sequence according to the target natural language instruction information, and the target operation information sequence is used to be applied on the target device so that the target device implements the target task corresponding to the target natural language instruction information. It can extract the user's operation instructions and operation goals in the process of executing the task, generate sample user operation information that describes the user's behavior, and train the language model through the sample user operation information to obtain an intelligent operation model that can output a target operation information sequence according to the user's natural language instructions. The target operation information sequence can enable the target device to automatically achieve the target task that the user needs to complete, which not only improves the efficiency of obtaining the task implementation steps, helps to improve the efficiency of automatically processing the target task on the target device, but also enables the user to realize the automatic processing of the target task through natural language, thereby improving the efficiency of the user performing the target task through the target device.

[0072] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0074] Figure 1 It is a flowchart of a method for constructing an intelligent operation model oriented to behavior learning provided by an embodiment of the present invention;

[0075] Figure 2 is a flowchart of another method for constructing an intelligent operation model oriented to behavior learning provided by an embodiment of the present invention;

[0076] Figure 3 This is a structural diagram of an intelligent operating system oriented to behavior learning provided by an embodiment of the present invention;

[0077] Figure 4 This is a functional schematic diagram of an intelligent operating system provided by an embodiment of the present invention;

[0078] Figure 5 It is a structural diagram of an intelligent operation model building device for behavior learning provided by an embodiment of the present invention;

[0079] Figure 6 is a logic block diagram of an electronic device according to an embodiment of the present invention;

[0080] Figure 7 It is a logic block diagram of an electronic device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0081] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0082] Figure 1 is a flowchart of a method for constructing an intelligent operation model for behavior learning provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0083] Step 101: Acquire an operation image generated when a sample user operates a target device to implement a sample task, and an operation log generated when the target device executes the sample task.

[0084] In an embodiment of the present invention, the target device may include any device that can receive user input and generate corresponding feedback to complete a specific task, and may include but is not limited to mobile terminals, computers, vehicles, construction machinery, etc., which are not specifically limited in the embodiment of the present invention. Depending on the target device, the sample tasks may also be diverse. For example, when the target device is a computer, the sample tasks may include but are not limited to mapping tasks, software installation tasks, system cleanup tasks, development environment configuration tasks, and other tasks that can be achieved by operating the computer; when the target device is a construction machinery, the sample tasks may include but are not limited to vehicle movement tasks, lifting tasks, excavation tasks, transportation tasks, etc., which are not specifically limited in the embodiment of the present invention.

[0085] In the embodiment of the present invention, the starting feature and the ending feature can be set for the sample task, so that the operating system can identify the implementation process of the sample task through the starting feature and the ending feature of the sample task, and collect the operation image generated during the sample user's operation of the target device to implement the sample task, and the operation log generated during the target device's execution of the sample task. Exemplarily, the starting feature may include but is not limited to one or more of the starting time, the starting operation instruction, the starting operation image feature, the starting user password, etc., and the ending feature may include but is not limited to one or more of the ending time, the ending operation instruction, the ending operation image feature, the ending user password, etc., which are not specifically limited in the embodiment of the present invention.

[0086] The operation image may include an image of the operation target operated by the sample user through the operation instruction. For example, when the target device is a mobile terminal, the operation image may include a screen image of the mobile terminal, or an external image including the mobile terminal body; when the target device is an engineering machine, the operation image may include an image of an object (such as stone, hoisted object, etc.) or an interactive target (such as a target position to be moved, a target position for placing a hoisted object, etc.) operated by the sample user through the target device.

[0087] The operation log refers to a log containing the operation instructions input by the sample user during the implementation of the sample task by the target device, and its content may include the correspondence between the operation instructions and the operation time. The operation instructions may include any instructions input through the input device of the target device, such as the left mouse button being pressed for 0.1 seconds, the right mouse button being pressed for 1 second, the keyboard X button being pressed for 0.2 seconds, the operation stick A moving from position 1 to position 2, etc., which are not specifically limited in the embodiment of the present invention.

[0088] Step 102, based on the operation image and the operation log, generate a sample user operation information sequence for implementing the sample task on the target device; wherein the sample user operation information in the sample user operation information sequence includes an operation instruction determined based on the operation log and an operation target determined based on the operation image.

[0089] In an embodiment of the present invention, the operation instructions generated by the sample user during the execution of the sample task, as well as the order of these operation instructions, can be extracted from the operation log. Then, according to the operation time corresponding to each operation instruction, the operation target corresponding to the operation instruction is identified and extracted from the operation image of the corresponding time, and the sample user operation information consisting of the operation instruction and the corresponding operation target is obtained. These sample user operation information are arranged according to the order of occurrence of the operation instruction, and the sample user operation information sequence is obtained. Among them, the sample user operation information can specifically include text information describing the operation instruction and the operation target, such as "the confirmation button of the pop-up window in the center of the screen-short press the left mouse button", "the search bar in the current interface-keyboard input search content A", "the container to be hoisted in front-control the boom to lift 5 meters", etc., which are not specifically limited in the embodiment of the present invention.

[0090] For example, this solution can be an intelligent operating system applied on a computer, and the sample task can be a contrast adjustment task in a graphic design process. When a sample user uses a graphic design software to complete a sample task, an operation log of the sample user for the graphic design software and an operation image of the sample user for the graphic design software can be generated. Based on the operation log and operation image, operation instructions and operation targets are extracted to generate a sample user operation information sequence in the process of the sample user completing the graphic design task.

[0091] Step 103: input the sample natural language instruction information corresponding to the sample task into a first language model to obtain a first operation information sequence output by the first language model.

[0092] In an embodiment of the present invention, the first language model may include a language model such as a Transformer that has not been pre-trained, and may also include a pre-trained large language model (Pre-trained Large Language Models), and the pre-trained large language model may include but is not limited to BERT (Bidirectional Encoder Representations from Transformers), LLaMA (Large Language Model Meta AI), Qwen, etc., which are not specifically limited in the embodiment of the present invention. In an embodiment of the present invention, the sample natural language instruction information corresponding to the sample task can be input into the first language model to obtain a first operation information sequence output by the first language model.

[0093] The sample natural language instruction information can be directly input by the user, or obtained by text extraction of the collected sample voice instructions. Specifically, when implementing the sample task, the sample user can express the natural language instructions of the current sample task through text content or voice content, collect these natural language instructions, and generate corresponding sample natural language instruction information to establish a corresponding relationship between them and the sample task. After collecting the sample user operation information sequence corresponding to the sample task, the sample natural language instruction information corresponding to the sample task can be input into the first language model to obtain the first operation information sequence output by the first language model. The first operation information sequence can have the same format as the sample user operation information sequence.

[0094] Continuing with the above example, the sample user can start the contrast adjustment task through voice instructions before adjusting the contrast. The system can respond to the instruction, determine that the sample task is to adjust the contrast, and determine the trigger task start instruction, and start recording the operation log and operation image. After the sample user completes the task, the sample user can indicate through voice that the contrast adjustment task is completed. The system can respond to the instruction, determine the trigger task end instruction, and generate a sample user operation information sequence for implementing the sample task on the target device based on the operation image and operation log, and generate sample natural language instruction information corresponding to the sample task according to the sample user's voice instructions. Then, in the model training stage, the sample natural language instruction information can be input into the first language model to obtain the first operation information sequence corresponding to the contrast adjustment task output by the first language model.

[0095] Step 104, based on the loss value between the first operation information sequence corresponding to the sample task and the sample user operation information sequence of the sample task, adjust the model parameters of the first language model to obtain an intelligent operation model; wherein the intelligent operation model is used to output a corresponding target operation information sequence according to the target natural language instruction information, and the target operation information sequence is used to be applied in a target device so that the target device can achieve the target task corresponding to the target natural language instruction information.

[0096] In an embodiment of the present invention, the loss value between the first operation information sequence corresponding to the sample task and the sample user operation information sequence corresponding to the sample task can be calculated, and the model parameters of the first language model can be adjusted using the loss value. The above training process is repeated until the model performance reaches a preset performance indicator or the training rounds reach a preset round, and then the training is stopped to obtain an intelligent operation model. The intelligent operation model is used to output a corresponding target operation information sequence according to the target natural language instruction information, and the target operation information sequence is used to be applied in the target device so that the target device can achieve the target task corresponding to the target natural language instruction information. It should be noted that the target device for collecting training data and the target device for applying the model can be the same device or different devices of the same type.

[0097] Exemplarily, before training the first language model, the training data including the sample user operation information sequence of the sample task and the sample natural language instruction information of the sample task are collected in the manner disclosed in the above embodiment to obtain a training set obtained by multiple training data. During training, the natural language instruction information of the training data can be input into the first language model to obtain the first operation information sequence output by the first language model, and then the loss value between the sample user operation information sequence and the first operation information sequence corresponding to the natural language instruction information is calculated by the loss function, and then the gradient of the loss value relative to the model parameter is calculated by the back propagation algorithm, and the model parameters are updated using the optimization algorithm to minimize the loss function. Among them, the above loss function may include but is not limited to cross-entropy loss (Cross-Entropy Loss), contrast loss (Contrastive Loss), etc., and the above optimization algorithm may include but is not limited to adaptive moment estimation optimizer (AdaptiveMoment Estimation, Adam), stochastic gradient descent (Stochastic Gradient Descent, SGD), etc., which are not specifically limited in the embodiments of the present invention.

[0098] It should be noted that, when the first language model is a pre-trained large language model, the first language model can also be trained in an unsupervised manner. Specifically, the sample natural language instruction information corresponding to the sample task and the sample user operation information sequence corresponding to the sample task can be input into the first language model as a data pair, so that the first language model can autonomously learn the association between the natural language instruction information and the user operation information sequence.

[0099] An embodiment of the present invention provides a method for constructing an intelligent operation model for behavior learning, including obtaining an operation image generated when a sample user operates a target device to implement a sample task, and an operation log generated when the target device executes the sample task; based on the operation image and the operation log, generating a sample user operation information sequence for implementing the sample task on the target device; wherein the sample user operation information in the sample user operation information sequence includes an operation instruction determined based on the operation log and an operation target determined based on the operation image; inputting sample natural language instruction information corresponding to the sample task into a first language model to obtain a first operation information sequence output by the first language model; adjusting the model parameters of the first language model based on the loss value between the first operation information sequence corresponding to the sample task and the sample user operation information sequence of the sample task to obtain an intelligent operation model; wherein the intelligent operation model is used to output a corresponding target operation information sequence according to the target natural language instruction information, and the target operation information sequence is used to be applied to the target device so that the target device implements the target task corresponding to the target natural language instruction information. It can extract the user's operation instructions and operation goals in the process of executing the task, generate sample user operation information that describes the user's behavior, and train the language model through the sample user operation information to obtain an intelligent operation model that can output a target operation information sequence according to the user's natural language instructions. The target operation information sequence can enable the target device to automatically achieve the target task that the user needs to complete, which not only improves the efficiency of obtaining the task implementation steps, helps to improve the efficiency of automatically processing the target task on the target device, but also enables the user to realize the automatic processing of the target task through natural language, thereby improving the efficiency of the user performing the target task through the target device.

[0100] Reference Figure 2 , Figure 2 is a flowchart of another method for constructing an intelligent operation model for behavior learning provided by an embodiment of the present invention. Figure 2 As shown, the method includes:

[0101] Step 201 : obtaining an operation image generated when a sample user operates a target device to implement a sample task, and an operation log generated when the target device executes the sample task.

[0102] This step can refer to step 101, and will not be described in detail in this embodiment of the present invention.

[0103] Optionally, step 201 may include:

[0104] Sub-step 2011: detecting the operation type of the operation instruction of the sample user on the target device.

[0105] In the embodiment of the present invention, since the target device may include an internal operation interface (such as a touch screen of an internal system) and an actuator for an external object (such as a boom, a bucket, etc.), it is possible to respond to the start instruction of the sample task and then start detecting the sample user's operation instructions for the target device, and determine whether the sample user is operating the internal operation interface or the external operation component according to the operation type of the operation instruction. The operation type of the operation instruction may include but is not limited to actuator operation, interface operation, etc., and the embodiment of the present invention does not specifically limit it. Specifically, the correspondence between various operation instructions and operation types can be established in advance, and the operation type of the operation instruction can be determined by the above correspondence.

[0106] Sub-step 2012, when the operation type of the operation instruction is actuator operation, photographing the operation scene image through the external camera of the target device.

[0107] If the operation type of the operation instruction is an actuator operation, it means that the sample user is interacting with an external target through the actuator. At this time, the current operation scene image can be captured by the external camera of the target device to determine the operation target corresponding to the operation instruction through the operation scene image. Among them, the external camera of the target device can be mounted on the target device or can be independent of the target device, which is not specifically limited in the embodiment of the present invention.

[0108] Exemplarily, the target device may include a forklift, and the operation instruction of the sample user may include controlling the lifting mechanism of the forklift to lift 0.5 meters. After receiving the operation instruction, it is determined that the operation type of the operation instruction is an actuator operation, and an external camera set up in the working scene of the current target device can be used to capture an operating scene image including the target device and the surrounding environment.

[0109] Sub-step 2013, when the operation type of the operation instruction is an interface operation, obtaining the operation screen image corresponding to the operation instruction.

[0110] When the operation type of the operation instruction is interface operation, it means that the sample user is operating the system of the target device through the screen, such as setting the navigation destination, setting the device operating parameters, etc. The operation screen image can be obtained by capturing the screen corresponding to the operation instruction, or taking a picture of the screen image corresponding to the operation instruction, and then the operation target of the operation instruction can be determined through the operation screen image.

[0111] By detecting the operation type of the sample user's operation instruction for the target device, when the operation type of the operation instruction is an actuator operation, the operation scene image is captured by the external camera of the target device, and when the operation type of the operation instruction is an interface operation, the operation screen image corresponding to the operation instruction is obtained. Not only can only the operation image corresponding to the operation instruction be obtained without continuously obtaining the operation image, which reduces the amount of operation image data and helps to improve the model training efficiency, but also different types of operation images can be obtained for different operation types, so that the present invention can obtain operation images containing their operation targets for multiple types of operation instructions, which helps to improve the accuracy of determining the operation target corresponding to the operation instruction based on the operation image.

[0112] Step 202 : extracting the user interaction objects of the sample user and the interaction time corresponding to each user interaction object based on the operation image.

[0113] In an embodiment of the present invention, user interaction objects of sample users and the interaction time corresponding to each user interaction object can be extracted based on the operation image. Among them, user interaction objects include objects that sample users are interacting with, such as options in a touch screen, text boxes in the screen, etc.; user interaction objects may also include objects that sample users are operating through target devices. For example, when the target device is an engineering machine, user interaction objects may include building materials, earthwork, stones, etc. The interaction time may include a time period. For example, the interaction time between the sample user and the text box in the screen is from 12 hours, 2 minutes, 23 seconds to 12 hours, 2 minutes, 58 seconds. The interaction time can be determined based on the shooting or reception time of the operation image.

[0114] Optionally, in the embodiment of the present invention, since the operation image may include the operation scene image and the operation screen image, step 202 may include:

[0115] Sub-step 2021, when the operation image includes the operation scene image, determine the first interaction object of the target device in the scene based on the operation scene image, and use the first interaction object as the user interaction object.

[0116] In an embodiment of the present invention, if the operation image includes an operation scene, the first interactive object of the target device in the scene can be determined based on the operation scene image, and the first interactive object can be used as the user interaction object of the sample user. It should be noted that the operation image can be set according to the task requirements of the sample task. For a certain sample task, it can be set to only obtain the operation scene image, or it can be set to only obtain the operation screen image. For some sample tasks that require the operation of the target device system and the interaction between the target device and the environment, the operation scene image and the operation screen image can be recorded or photographed at the same time; wherein, the operation scene image and the operation screen image can be continuous images during the execution of the sample task, or they can be images corresponding to the operation instructions obtained by using the above-mentioned sub-steps 2011 to 2013, and the embodiment of the present invention does not make specific limitations.

[0117] Specifically, in one embodiment, the first interactive object can be identified by a first interactive object recognition model trained based on a neural network model, the operation scene image can be input into the first interactive object recognition model, and the first interactive object recognition model outputs the first interactive object that the target device is interacting with in the operation scene image. In another embodiment, during the sample user's sample task, the object being operated can be specified in the scene through voice instructions, laser markings, and other prompts, so that the system can determine the first interactive object from the operation scene image according to the above prompts.

[0118] Optionally, sub-step 2021 may include:

[0119] Sub-step 20211, inputs the operation scene image including the target device into a target detection model to obtain a first target detection result of the interactive target object corresponding to the target device output by the target detection model, and a second target detection result corresponding to the target device.

[0120] In an embodiment of the present invention, in order to improve the accuracy of determining the first interactive object, an operating scene image including a target device may be obtained, and the operating scene image including the target device may be input into a target detection model to obtain an interactive target object corresponding to the target device output by the target detection model, and a target device detection result corresponding to the target device. Among them, the above-mentioned target detection model may be obtained by training based on an initial target detection model, and the initial target detection model may include but is not limited to Faster R-CNN (Region-based Convolutional Neural Networks), YOLO (You Only Look Once), etc., which are not specifically limited in the embodiment of the present invention. The target detection model can identify the target device and other objects in the scene from the operating scene image, and determine the interactive target object corresponding to the target device from other objects according to the device type of the target device, and output the first target detection result of the interactive target object, and at the same time output the second target detection result of the target device.

[0121] The interactive target object refers to an object that the target device can interact with. Different target devices can correspond to interactive target objects of different object types. The object type of the interactive target object corresponding to the target object can be set in advance by the technician, and a corresponding relationship between the device type of the target device and the object type of the interactive target object can be established. For example, if the target device is an excavator, the corresponding interactive target object can include but is not limited to earthwork, stone, etc.

[0122] Sub-step 20212, inputting the first target detection result, the second target detection result and the sample task into an intention recognition model, and obtaining the interaction probabilities corresponding to each of the interactive target objects output by the intention recognition model.

[0123] In an embodiment of the present invention, the first target detection result, the second target detection result and the sample task can be input into the intention recognition model to obtain the interaction probability corresponding to each interactive target object output by the intention recognition model. The interaction probability is used to indicate the probability that the target device is operating the interactive target object.

[0124] Sub-step 20213: determining the interactive target object with the highest interaction probability as the first interactive object.

[0125] In the embodiment of the present invention, the interactive target object with the highest interaction probability in the operation scene image may be used as the first interactive object.

[0126] By inputting the operation scene image containing the target device into the target detection model, the first target detection result of the target device corresponding to the interactive target object and the second target detection result corresponding to the target device output by the target detection model are obtained; the first target detection result, the second target detection result and the sample task are input into the intention recognition model to obtain the interaction probability corresponding to each interactive target object output by the intention recognition model; the interactive target object with the largest interaction probability is determined as the first interactive object. The first interactive object can be determined completely automatically, which helps to improve the efficiency and accuracy of determining the first interactive object.

[0127] Sub-step 2022, when the operation image includes the operation screen image, based on the input position of the user input of the sample user in the operation screen image, determine a second interaction object from the operation screen image, and use the second interaction object as the user interaction object.

[0128] In the embodiment of the present invention, if the operation image is an operation screen image, the second interaction object can be determined from the operation screen image according to the input position of the user input in the operation screen image, and the second interaction object can be used as the user interaction object.

[0129] Exemplarily, if the user input is a touch operation, the input position of the input instruction in the operation screen image can be determined based on the contact position of the user's finger with the touch screen, and the second interactive object (such as a button, text box, etc.) corresponding to the input position can be extracted, and the second interactive object can be used as the user interactive object.

[0130] When the operation image includes the operation scene image, the first interactive object of the target device in the scene is determined based on the operation scene image, and the first interactive object is used as the user interactive object. When the operation image includes the operation screen image, the second interactive object is determined from the operation screen image based on the input position of the user input in the operation screen image, and the second interactive object is used as the user interactive object. Thus, the corresponding user interactive object can be determined for different types of input instructions, which helps to improve the accuracy of determining the user interactive object.

[0131] Step 203: extracting the operation instructions of the sample user and the operation time corresponding to each operation instruction based on the operation log.

[0132] In the embodiment of the present invention, the operation instructions of the sample users and the operation time corresponding to each operation instruction can be extracted based on the operation log, wherein the operation time indicates the occurrence time of the corresponding operation instruction.

[0133] Step 204: Determine the user interaction object whose interaction time overlaps with the operation time of the operation instruction as the operation object corresponding to the operation instruction.

[0134] In the embodiment of the present invention, the interaction time indicates the interaction time between the sample user and the user interaction object, and the operation time of the operation instruction indicates the occurrence time of the operation instruction. The two can be matched with each other, and the user interaction object whose interaction time overlaps with the operation time of the operation instruction is determined as the operation object corresponding to the operation instruction. In other words, if an operation instruction S1 occurs within the interaction time corresponding to the user interaction object S2, the user interaction object S2 is determined as the operation object corresponding to the operation instruction S1.

[0135] Step 205: construct sample user operation information corresponding to the operation instruction based on the operation instruction and the operation object corresponding to the operation instruction, and arrange the sample user operation information corresponding to each operation instruction according to the operation time to obtain the sample user operation information sequence.

[0136] In the embodiment of the present invention, the user interaction objects of the sample users and the interaction time corresponding to each user interaction object can be extracted based on the operation image; the operation instructions of the sample users and the operation time corresponding to each operation instruction can be extracted based on the operation log; the user interaction objects whose interaction time overlaps with the operation time of the operation instruction are determined as the operation objects corresponding to the operation instruction; based on the operation instruction and the operation object corresponding to the operation instruction, the sample user operation information corresponding to the operation instruction is constructed, and the sample user operation information corresponding to each operation instruction is arranged according to the operation time to obtain the sample user operation information sequence. The sample user operation information sequence can be determined automatically, which helps to improve the efficiency of the system in recording the sample user operation information.

[0137] Step 206: Acquire sample environment information of the target device during the period when the sample user operates the target device to implement the sample task.

[0138] In the embodiment of the present invention, when the target device performs the sample task in different operating environments, the sample user may have different processes for implementing the sample task through the target device. For example, in hot weather, the sample user may take slower actions when operating the construction machinery to complete the sample task, or may pause for a longer time between instructions to avoid overheating of the construction machinery; for another example, in rainy and snowy weather, the sample user may take slower movement when operating the construction machinery to complete the sample task to avoid slipping; for another example, in cold weather, the sample user may wait for a while after starting the device to warm up the device.

[0139] Therefore, in the embodiment of the present invention, the sample environment information of the target device during the period when the sample user operates the target device to achieve the sample task can be obtained, and the sample environment information can also be used as the learning content of the model. The sample environment information may include but is not limited to the environment temperature, environment weather, environment brightness, environment terrain, road surface material, etc., which is not specifically limited in the embodiment of the present invention.

[0140] Step 207: input the sample natural language instruction information and the sample environment information into the first language model to obtain the first operation information sequence output by the first language model.

[0141] In an embodiment of the present invention, sample natural language instruction information and sample environment information may be input into a first language model to obtain a first operation information sequence output by the first language model, wherein both the sample environment information and the sample natural language instruction information may be text information.

[0142] Step 208, based on the loss value between the first operation information sequence corresponding to the sample task and the sample user operation information sequence of the sample task, adjust the model parameters of the first language model to obtain an intelligent operation model; wherein the intelligent operation model is used to output a corresponding target operation information sequence according to the target natural language instruction information, and the target operation information sequence is used to be applied in a target device so that the target device can achieve the target task corresponding to the target natural language instruction information.

[0143] This step can be referred to as step 104, and will not be described in detail in this embodiment of the present invention.

[0144] Optionally, after obtaining the intelligent operation model, the intelligent operation model may be continuously improved according to user feedback during the use of the intelligent operation model. For details, see steps A1 to A3:

[0145] Step A1, obtaining question feedback information for the target operation information sequence.

[0146] In an embodiment of the present invention, for each target operation information sequence output by the intelligent operation model in the application, user problem feedback information can be received. Among them, the problem feedback information may include operation steps indicating that there is a problem in the target operation information sequence. For example, the order of steps 1 and 2 in the sequence is wrong, step 4 in the sequence should be XXX, step 6 in the sequence is redundant, and so on. Specifically, the user can describe the problem feedback information by voice or text, and feedback to the system that implements the above-mentioned behavior learning-oriented intelligent operation model construction method. The system can be a cloud-based model training system or a local intelligent operating system of the target device, which is not specifically limited by the embodiment of the present invention.

[0147] Step A3: Generate model adjustment prompt word information based on the question feedback information, the target operation information sequence and the target natural language instruction information corresponding to the target operation information sequence.

[0148] In an embodiment of the present invention, the question feedback information corresponding to the above-mentioned target operation information sequence, the target operation information sequence and the target natural language instruction information corresponding to the target operation information sequence can be spliced ​​to obtain model adjustment prompt word information; the question feedback information corresponding to the above-mentioned target operation information sequence, the target operation information sequence and the target natural language instruction information corresponding to the target operation information sequence can also be connected through fixed phrases to obtain model adjustment prompt word information; the question feedback information corresponding to the above-mentioned target operation information sequence, the target operation information sequence and the target natural language instruction information corresponding to the target operation information sequence can also be input into a specially trained model adjustment prompt word information generation model to obtain model adjustment prompt word information output by the model adjustment prompt word information generation model; technical personnel can select an appropriate model adjustment prompt word information generation method according to actual business needs, and the embodiment of the present invention does not make specific limitations.

[0149] Step A3: input the model adjustment prompt word information into the intelligent operation model.

[0150] In an embodiment of the present invention, the model adjustment prompt word information can be used as the prompt word of the intelligent operation model and input into the intelligent operation model so that the intelligent operation model can update and learn autonomously. In the subsequent target operation information sequence output process, the model adjustment prompt word information can be referred to to avoid similar problems and achieve more accurate target operation information sequence output.

[0151] Another method for constructing an intelligent operation model for behavior learning provided by an embodiment of the present invention can extract the operation instructions and operation goals of the user in the process of executing a task, generate sample user operation information that describes the user behavior, and train a language model through the sample user operation information to obtain an intelligent operation model that can output a target operation information sequence according to the user's natural language instructions. The target operation information sequence can enable the target device to automatically achieve the target task that the user needs to complete, which not only improves the efficiency of obtaining the task implementation steps, helps to improve the efficiency of automatically processing the target task on the target device, but also enables the user to realize the automatic processing of the target task through natural language, thereby improving the efficiency of the user performing the target task through the target device.

[0152] Corresponding to the method provided in the embodiment of the method for constructing an intelligent operation model for behavior learning of the present invention, see Figure 3 The present invention also provides a structure diagram of an intelligent operating system, such as Figure 3As shown, an intelligent operating system 30 includes an instruction acquisition unit 31, an inference unit 32 and an execution unit 33:

[0153] The instruction acquisition unit 31 is used to acquire target natural language instruction information input by a target user.

[0154] The reasoning unit 32 is used to input the target natural language instruction information into the intelligent operation model so that the intelligent operation model outputs a target operation information sequence; wherein the intelligent operation model is generated based on the above-mentioned behavior learning-oriented intelligent operation model construction method.

[0155] In the embodiment of the present invention, the target operation information sequence may include each target operation information for controlling the target device to achieve the target task, and each target operation information may include a target operation instruction and an operation target corresponding to the target operation instruction.

[0156] Exemplarily, when the target device is a computer, the target task may include forwarding the latest update posted by contact XXX. Accordingly, a target operation information corresponding to the target task may include: entering XXX in the contact search box on the current interface; the next target operation information may include: clicking the search button on the current interface.

[0157] When the target equipment is a lifting equipment, the target task may include lifting the container onto a transport vehicle. Accordingly, a target operation information corresponding to the target task may include: moving the clamping mechanism to the anchor point position of the container to be lifted; the next target operation information may include: driving the clamping mechanism to clamp the anchor point of the container to be lifted, and so on.

[0158] The execution unit 33 is used to control the target device to achieve the target task corresponding to the target natural language instruction information based on the target operation information sequence.

[0159] In an embodiment of the present invention, the execution unit may receive a target operation information sequence, and control the target device to directly execute the target operation instruction and the corresponding operation target indicated by each target operation information. The target operation information may also be parsed and analyzed, converted into a series of machine instructions that can be directly executed by the target device, and then these machine instructions are applied to the target device.

[0160] Optionally, the intelligent operating system further includes an environment information acquisition unit;

[0161] The environment information acquisition unit is used to acquire target environment information where the target device is located;

[0162] The inference unit is further used to input the target natural language instruction information and the target environment information into an intelligent operation model so that the intelligent operation model outputs a target operation information sequence;

[0163] It should be noted that, in the intelligent operating system provided in the embodiment of the present invention, a behavior-learning-oriented intelligent operation model construction module may also be integrated to generate the above-mentioned intelligent operation model through a behavior-learning-oriented intelligent operation model method.

[0164] This system has the ability to learn users' specialized operating behaviors and can simulate complex human operating processes to achieve target tasks; this system enables users to interact with the operating system in a more natural way (such as text or voice); this system can automatically execute target tasks indicated by users through natural language instruction information, and can improve the level of work automation; this system can also add a problem feedback mechanism, so that the system can self-learn and optimize, and continuously improve the accuracy of the output target operation information sequence as it is used; this system can be deployed on a cloud server and provide powerful model reasoning and computing capabilities through cloud computing.

[0165] Reference Figure 4 , Figure 4 FIG. 4 shows a functional schematic diagram of an intelligent operating system provided by an embodiment of the present invention. Figure 4As shown in the figure, the intelligent operating system has multimodal perception function, can integrate multiple sensors, and collect multiple types of data (such as operation images, voice commands, etc.); behavior learning function, learns user operation behavior through an integrated behavior learning engine; intelligent resource management function, realizes dynamic allocation and management of system resources, and intelligently adjusts the use of CPU, memory, storage and network resources according to the needs of the current task and the system status; automated task execution function, can provide automated execution of target tasks through intelligent operation models, reduce repetitive work, and improve user efficiency; natural language user interface, can allow users to interact with the system through natural language processing technology (such as voice or text), realize more natural and intuitive control methods, and improve user operation efficiency; feedback optimization function, can automatically adjust and optimize the intelligent operation model in the system according to user feedback , to improve the accuracy of automated tasks and user satisfaction; data security and privacy protection functions, which can encrypt user voice operation instructions, training data, etc., and transmit and process them after encryption, and can also provide secure access control to ensure the security of user data; cross-platform compatibility function, this system can support multiple hardware platforms and different target devices, ensuring that the operating system can provide a consistent user experience on different target devices; cloud collaborative processing function, which can realize cloud computing resource integration, deploy intelligent operation models on the cloud platform, and use cloud computing resources to realize data storage, processing and analysis, etc., to improve system performance and scalability; API interface function, so that third-party application and service development can provide open application program interface (API) and software development kit (SDK) to enable third-party developers to create applications and services compatible with this system. It should be noted that the intelligent operating system can be the operating system of the target device. For example, when the target device is a personal computer, the intelligent operating system can be a computer operating system.

[0166] An intelligent operating system for behavior learning provided by an embodiment of the present invention can extract the operation instructions and operation goals of the user in the process of executing a task, generate sample user operation information describing the user behavior, and train a language model through the sample user operation information to obtain an intelligent operation model that can output a target operation information sequence according to the user's natural language instructions. The target operation information sequence can enable the target device to automatically achieve the target task that the user needs to complete, which not only improves the efficiency of obtaining the task implementation steps, helps to improve the efficiency of automatically processing the target task on the target device, but also enables the user to realize the automatic processing of the target task through natural language, thereby improving the efficiency of the user performing the target task through the target device.

[0167] Corresponding to the method provided in the embodiment of the method for constructing an intelligent operation model for behavior learning of the present invention, see Figure 5The present invention also provides a structure diagram of an intelligent operation model construction device for behavior learning, such as Figure 5 As shown, in this embodiment, the behavior learning-oriented intelligent operation model building device 40 includes:

[0168] An acquisition module 41 is used to acquire an operation image generated when a sample user operates a target device to implement a sample task, and an operation log generated when the target device performs the sample task;

[0169] A sequence module 42, configured to generate a sample user operation information sequence for implementing the sample task on the target device based on the operation image and the operation log; wherein the sample user operation information in the sample user operation information sequence includes an operation instruction determined based on the operation log and an operation target determined based on the operation image;

[0170] An input module 43, configured to input the sample natural language instruction information corresponding to the sample task into a first language model, and obtain a first operation information sequence output by the first language model;

[0171] The training module 44 is used to adjust the model parameters of the first language model based on the loss value between the first operation information sequence corresponding to the sample task and the sample user operation information sequence of the sample task, so as to obtain an intelligent operation model; wherein the intelligent operation model is used to output a corresponding target operation information sequence according to the target natural language instruction information, and the target operation information sequence is used to be applied in a target device so that the target device can achieve the target task corresponding to the target natural language instruction information.

[0172] Optionally, the sequence module includes:

[0173] A first extraction submodule, configured to extract the user interaction objects of the sample user and the interaction time corresponding to each user interaction object based on the operation image;

[0174] A second extraction submodule, configured to extract the operation instructions of the sample user and the operation time corresponding to each operation instruction based on the operation log;

[0175] An operation object submodule, configured to determine a user interaction object whose interaction time overlaps with an operation time of the operation instruction as an operation object corresponding to the operation instruction;

[0176] The sequence construction submodule is used to construct sample user operation information corresponding to the operation instruction based on the operation instruction and the operation object corresponding to the operation instruction, and arrange the sample user operation information corresponding to each operation instruction according to the operation time to obtain the sample user operation information sequence.

[0177] Optionally, the operation image includes an operation scene image and an operation screen image, and the first extraction submodule includes:

[0178] A first interaction object extraction unit, configured to determine, when the operation image includes the operation scene image, a first interaction object of the target device in the scene based on the operation scene image, and use the first interaction object as the user interaction object;

[0179] A second interaction object extraction unit is used to determine a second interaction object from the operation screen image based on an input position of a user input of the sample user in the operation screen image when the operation image includes the operation screen image, and use the second interaction object as the user interaction object.

[0180] Optionally, the first interactive object extraction unit includes:

[0181] A detection result subunit, configured to input the operation scene image including the target device into a target detection model, and obtain a first target detection result of an interactive target object corresponding to the target device output by the target detection model, and a second target detection result corresponding to the target device;

[0182] An interaction probability subunit, configured to input the first target detection result, the second target detection result, and the sample task into an intention recognition model to obtain interaction probabilities corresponding to each of the interactive target objects output by the intention recognition model;

[0183] The first interactive object determining subunit is configured to determine the interactive target object with the highest interaction probability as the first interactive object.

[0184] Optionally, the acquisition module includes:

[0185] An operation type submodule, used to detect the operation type of the operation instruction of the sample user on the target device;

[0186] An operation scene image submodule, used for capturing the operation scene image by an external camera of the target device when the operation type of the operation instruction is an actuator operation;

[0187] The operation screen image submodule is used to obtain the operation screen image corresponding to the operation instruction when the operation type of the operation instruction is an interface operation.

[0188] Optionally, the input module includes:

[0189] A sample environment information submodule, used to obtain sample environment information of the target device during the period when the sample user operates the target device to implement the sample task;

[0190] The environment information input submodule is used to input the sample natural language instruction information and the sample environment information into the first language model to obtain the first operation information sequence output by the first language model.

[0191] Optionally, the device further comprises:

[0192] A question feedback information acquisition module, used to acquire question feedback information for a target operation information sequence;

[0193] An adjustment prompt word information module is used to generate model adjustment prompt word information based on the question feedback information, the target operation information sequence and the target natural language instruction information corresponding to the target operation information sequence;

[0194] The adjustment prompt word information input module is used to input the model adjustment prompt word information into the intelligent operation model.

[0195] An embodiment of the present invention provides an intelligent operation model construction device for behavior learning, which can extract the operation instructions and operation goals of the user in the process of executing a task, generate sample user operation information describing the user behavior, and train a language model through the sample user operation information to obtain an intelligent operation model that can output a target operation information sequence according to the user's natural language instructions. The target operation information sequence can enable the target device to automatically achieve the target task that the user needs to complete, which not only improves the efficiency of obtaining the task implementation steps, helps to improve the efficiency of automatically processing the target task on the target device, but also enables the user to realize the automatic processing of the target task through natural language, thereby improving the efficiency of the user performing the target task through the target device.

[0196] Figure 6 6 is a logical block diagram of an electronic device 600 according to an embodiment of the present invention. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0197] Reference Figure 6 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output interface 612 , a sensor component 614 , and a communication component 616 .

[0198] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0199] The memory 604 is used to store various types of data to support the operation of the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0200] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0201] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0202] The audio component 610 is used to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0203] The input / output interface 612 provides an interface between the processing component 602 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0204] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of the components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of contact between the user and the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0205] The communication component 616 is used to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0206] In an exemplary embodiment, the electronic device 600 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to implement a method for constructing an intelligent operation model for behavioral learning provided in an embodiment of the present invention.

[0207] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of an electronic device 600 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0208] Figure 7 700 is a logic block diagram of an electronic device 700 according to an embodiment of the present invention. For example, the electronic device 700 may be provided as a server. Figure 7 The electronic device 700 includes a processing component 722, which further includes one or more processors, and a memory resource represented by a memory 732 for storing instructions executable by the processing component 722, such as an application. The application stored in the memory 732 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 722 is configured to execute instructions to perform a method for constructing an intelligent operation model for behavior learning provided by an embodiment of the present invention.

[0209] The electronic device 700 may further include a power supply component 726 configured to perform power management of the electronic device 700, a wired or wireless network interface 750 configured to connect the electronic device 700 to a network, and an input-output interface 758. The electronic device 700 may operate based on an operating system stored in the memory 732, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0210] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the method for constructing an intelligent operation model oriented to behavior learning when executed by a processor.

[0211] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the applications disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0212] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for constructing an intelligent operation model for behavior learning, characterized in that: The method for constructing an intelligent operation model oriented to behavior learning includes: Acquire an operation image generated when a sample user operates a target device to implement a sample task, and an operation log generated when the target device performs the sample task; Based on the operation image and the operation log, a sample user operation information sequence for implementing the sample task on the target device is generated; wherein the sample user operation information in the sample user operation information sequence includes an operation instruction determined based on the operation log and an operation target determined based on the operation image; Inputting sample natural language instruction information corresponding to the sample task into a first language model to obtain a first operation information sequence output by the first language model; Based on the loss value between the first operation information sequence corresponding to the sample task and the sample user operation information sequence of the sample task, the model parameters of the first language model are adjusted to obtain an intelligent operation model; wherein the intelligent operation model is used to output a corresponding target operation information sequence according to the target natural language instruction information, and the target operation information sequence is used to be applied on a target device so that the target device can achieve the target task corresponding to the target natural language instruction information.

2. The method for constructing an intelligent operation model for behavior learning according to claim 1, characterized in that: The generating, based on the operation image and the operation log, a sample user operation information sequence for implementing the sample task on the target device comprises: Based on the operation image, extracting the user interaction objects of the sample user and the interaction time corresponding to each user interaction object; Based on the operation log, extract the operation instructions of the sample user and the operation time corresponding to each operation instruction; Determine the user interaction object whose interaction time overlaps with the operation time of the operation instruction as the operation object corresponding to the operation instruction; Based on the operation instructions and the operation objects corresponding to the operation instructions, sample user operation information corresponding to the operation instructions is constructed, and the sample user operation information corresponding to each operation instruction is arranged according to the operation time to obtain the sample user operation information sequence.

3. The method for constructing an intelligent operation model for behavior learning according to claim 2, characterized in that: The operation image includes an operation scene image and an operation screen image, and extracting the user interaction object of the sample user based on the operation image includes: In a case where the operation image includes the operation scene image, determining a first interaction object of the target device in the scene based on the operation scene image, and using the first interaction object as the user interaction object; In the case where the operation image includes the operation screen image, a second interaction object is determined from the operation screen image based on an input position of a user input of the sample user in the operation screen image, and the second interaction object is used as the user interaction object.

4. The method for constructing an intelligent operation model for behavior learning according to claim 3, characterized in that: The determining, based on the operation scene image, a first interactive object of the target device in the scene includes: Inputting the operation scene image including the target device into a target detection model, and obtaining a first target detection result of an interactive target object corresponding to the target device and a second target detection result corresponding to the target device output by the target detection model; Inputting the first target detection result, the second target detection result, and the sample task into an intention recognition model to obtain interaction probabilities corresponding to each of the interactive target objects output by the intention recognition model; The interactive target object with the highest interaction probability is determined as the first interactive object.

5. The method for constructing an intelligent operation model for behavior learning according to claim 3, characterized in that: The step of obtaining the operation image generated when the sample user operates the target device to implement the sample task includes: Detecting the operation type of the operation instruction of the sample user on the target device; When the operation type of the operation instruction is an actuator operation, photographing the operation scene image by an external camera of the target device; When the operation type of the operation instruction is an interface operation, an operation screen image corresponding to the operation instruction is acquired.

6. The method for constructing an intelligent operation model for behavior learning according to claim 1, characterized in that: The step of inputting the sample natural language instruction information corresponding to the sample task into a first language model to obtain a first operation information sequence output by the first language model includes: Acquire sample environment information of the target device during the period when the sample user operates the target device to implement the sample task; The sample natural language instruction information and the sample environment information are input into the first language model to obtain the first operation information sequence output by the first language model.

7. The method for constructing an intelligent operation model for behavior learning according to claim 1, characterized in that: The method further comprises: Obtaining question feedback information for the target operation information sequence; Generate model adjustment prompt word information based on the question feedback information, the target operation information sequence and the target natural language instruction information corresponding to the target operation information sequence; The model adjustment prompt word information is input into the intelligent operation model.

8. An intelligent operation model construction device for behavior learning, characterized in that: The intelligent operation model building device for behavior learning includes: An acquisition module, used to acquire an operation image generated when a sample user operates a target device to implement a sample task, and an operation log generated when the target device performs the sample task; A sequence module, configured to generate a sample user operation information sequence for implementing the sample task on the target device based on the operation image and the operation log; wherein the sample user operation information in the sample user operation information sequence includes an operation instruction determined based on the operation log and an operation target determined based on the operation image; An input module, configured to input the sample natural language instruction information corresponding to the sample task into a first language model to obtain a first operation information sequence output by the first language model; A training module is used to adjust the model parameters of the first language model based on the loss value between the first operation information sequence corresponding to the sample task and the sample user operation information sequence of the sample task, so as to obtain an intelligent operation model; wherein the intelligent operation model is used to output a corresponding target operation information sequence according to the target natural language instruction information, and the target operation information sequence is used to be applied in a target device so that the target device can achieve the target task corresponding to the target natural language instruction information.

9. An intelligent operating system, characterized in that: The intelligent operating system includes an instruction acquisition unit, an inference unit and an execution unit; The instruction acquisition unit is used to acquire target natural language instruction information input by a target user; The reasoning unit is used to input the target natural language instruction information into the intelligent operation model so that the intelligent operation model outputs a target operation information sequence; wherein the intelligent operation model is generated based on the intelligent operation model construction method for behavior learning as described in claims 1 to 7; The execution unit is used to control the target device to achieve the target task corresponding to the target natural language instruction information based on the target operation information sequence.

10. The intelligent operating system according to claim 9, characterized in that: The intelligent operating system also includes an environmental information acquisition unit; The environment information acquisition unit is used to acquire target environment information where the target device is located; The reasoning unit is further used to input the target natural language instruction information and the target environment information into the intelligent operation model so that the intelligent operation model outputs a target operation information sequence.

Images